Would a recommendation based on objective rankings be possible in the ASR equipment tests?

[pma]

Would a recommendation based on objective rankings be possible in the ASR equipment tests?

I am thinking if it would be possible and if it would make a sense to give a component (at least electronic components like amplifiers and DACs) recommendations based on a set of objective measurements. The main ASR reviews are supported by a wide set of measured data, however the final verdict “recommended” or “not recommended” is based on the verdict of @amirm , and this verdict as such is subjective rather than objective. Is it OK? Maybe yes, maybe no. My doubts are based on several review recommendations, I would name two of them, when

Crown 4|300N Amplifier Review

This is a review and detailed measurements of the Crown DCI 4 | 300N four channel DSP and networked professional amplifier. It was kindly purchased (used) and sent to me for testing and costs US $3,511. There is only a power on/off on the front. I wish there was a processing bypass as well. The...

www.audiosciencereview.com

is a recommended component, but

Review and Measurements of Musical Fidelity M2si Amp

This is a review and detailed measurements of the Musical Fidelity M2si integrated stereo amplifier. It was kindly sent in by a member. I think the list price is US $999. However I see it selling for just US $599 from the outfit the owner purchased it. Other places have it as high as US...

www.audiosciencereview.com

is not recommended with explanation: “I can't recommend it on pure performance”.

I think that it might have been nice to have an automated system that would evaluate a full set of measurements (frequency response, noise, dynamic range, THD and THD+N, THD+N vs. frequency sweep, IMD, IMD+N vs. frequency sweep …..) in a similar way as it is done in RightMark Audio Analyzer software (RMAA). The resulting ranking would be, in my opinion, much more objective than the subjective opinion of one person. Based on a fact we are on Audioscience Review forum, it would probably fit better with the forum goal – to bring to the readers independent objective information.

For those who are not familiar with RMAA, this is how the RMAA chart looks like. An example of a full report may be seen here:

RightMark Audio Analyzer test : [ASIO] ULTIM2 96

and this is the header only (from another test):



The test conditions may be chosen and displayed in the RMAA main window:

I especially like the IMD swept tones test, which performs a twin-tone sweep of equidistantly spaced (1kHz) two tone over the measuring band.



I would appreciate your thoughts on equipment recommendation based on objective component test ranking.

 

[Blumlein 88]

I don't know if it is doable, but I like the idea.

I remember reading his review of a Motu which he was lukewarm about yet he praised a couple of other interfaces with measurements 2-3 db worse in nearly every parameter, and having fewer I/O options as well as costing much more. Objectively I don't think it made sense. I don't think Amir is bad about this, the reviews were over like 2.5 years and I don't expect anyone to remember every single detail right of the top of their head. Based on reputation, intended use, tactile quality etc you can have different feelings after using it beyond measures. But measures and objective performance are the mantra of ASR. Now beyond any of this all the measures are there for us to make use of and that is the big value of what Amir does. Plus you can go into the review index and filter for a certain type of device and then rank it by SINAD by clicking on the headers, same for price etc.

The big issue doing more is intended use and additional features. Or like in the two products you listed, one is much, much more powerful than the other. It also has some DSP, but is not really aimed for home stereo use. In some ways you end up with apples vs oranges.

 

[FINFET]

Can train an “Amir classifier” by feeding all the scores for each parameter as vector data as well as whether it is recommended to a linear classifier or neural network/MLP, which you can then use to automatically generate the output based on previous training data.

 

[pma]

I don't know if it is doable, but I like the idea.

Thank you. Would it make any sense to think deeper about it, or is it just wasting time. I am afraid the later is more probable, regarding this idea attracts almost no attention. I have a feeling that maybe 2-3 years earlier it might have fallen on a more fertile soil here.

 

[DonH56]

The idea seems sound; implementation may be challenging. Will RMAA (etc.) integrate with @amirm's AP analyzer for a seamless workflow, or add more overhead? Currently I think the main ranking mechanism is the SINAD chart. Recommendations may depend upon other factors more subjective than just raw performance, like the interface and so forth, and sometimes a component with poorer performance may be recommended based upon price and/or application. I agree there is subjectiveness, but it's sort of the nature of the beast.

How does RMAA objectively rank results? What differentiates "excellent" from "very good", for example? If we build a giant table of results, can it automatically rerank everything if the weighting or definition changes? As a gross example, 1% THD used to be "hi-fi", but now the number is probably more like 0.01% for an amp and 0.001% for a DAC. We'd need a way to recompile the results to keep them relevant over time, and likely need to adjust the weightings (ranking) depending upon the product (e.g. DAC vs. preamp vs. amp vs. speakers).

As for RMAA or a similar scheme, the one thing many have commented upon is the use of SINAD as the ranking criteria. Folk like to see a simple "tell-all" ranking, and SINAD provides that, but leads to apples-to-oranges comparisons in some cases and masks a lot of detail. For myself I tend to look in the top tier or two of SINAD rankings then dig into the review for the details. A table of results similar to what you posted from RMAA would be interesting. In my day job test results can run to many pages and organizing them into a concise summary is always challenging. Some of the test equipment generates automated reports in HTML format (web pages) that expand sections as requested (click on an arrow or whatever). That would be neat but probably difficult and time-consuming to implement. One idea would be to generate tables of results and make a "live" SINAD chart such that you could click on a component to see the table on the fly. Probably many hours of work to actually implement...

Rambling thoughts, basically three paragraphs to say "I don't know"....

 

[sarumbear]

Would a recommendation based on objective rankings be possible in the ASR equipment tests?

No

I am thinking if it would be possible and if it would make a sense to give a component (at least electronic components like amplifiers and DACs) recommendations based on a set of objective measurements. The main ASR reviews are supported by a wide set of measured data, however the final verdict “recommended” or “not recommended” is based on the verdict of @amirm , and this verdict as such is subjective rather than objective.
[...]
I would appreciate your thoughts on equipment recommendation based on objective component test ranking.

I don't like recommendations. They are always subjective. I want to make up my own mind from the data presented to me.

 

[abdo123]

Has someone tried ChatGPT for this?

 

[Ken Tajalli]

No


I don't like recommendations. They are always subjective. I want to make up my own mind from the data presented to me.

I don't get that, on one breath you say recommendation can not be done on objective test results, and then on the second, you like to make your own mind using the same objective test results?
What am I missing here?

 

[sarumbear]

I don't get that, on one breath you say recommendation can not be done on objective test results, and then on the second, you like to make your own mind using the same objective test results?
What am I missing here?

My own decision is not the same as a recommendation by someone else. Objective doesn't mean there is no criteria. Almost always we will have different criteria to the person who tests the device. Not only that but the tests will be incomplete and not every aspect of the device is tested and hence, we don't have enough data to make an objective decision.

Case in point is the Soundcraft mixer recently tested. Only the DAC & ADC of the device is tested because @amirm treated it as an audio interface. However, the device is marketed as Small-format Analog Mixing Console with USB I/O. I may be interested in the mixing console part of the unit and may not care the USB I/O specs that I do not need. How are we going to make an objective recommendation then as there is no data for the device's main section.

 

[pma]

How does RMAA objectively rank results? What differentiates "excellent" from "very good", for example?

It depends on results in individual disciplines and in a full set of disciplines. It reminds me a bit to car testing (acceleration, manoeuvrability, fuel consumption, space, safety NCAP, you name it). It still needs a qualified operator to know at which level (V, W, ...) is the individual discipline performed. However, I like that it is not based on a single test, like 1kHz/5W SINAD chart.

 

[Stinius]

It depends on results in individual disciplines and in a full set of disciplines. It reminds me a bit to car testing (acceleration, manoeuvrability, fuel consumption, space, safety NCAP, you name it). It still needs a qualified operator to know at which level (V, W, ...) is the individual discipline performed. However, I like that it is not based on a single test, like 1kHz/5W SINAD chart.

I agree, the 1kHz/5W SINAD chart is as meaningless as it can be when it comes to how an amp behaves and how to sort the amps.
I like your idea Pavel, but I doubt that it will be approved by the people that focus on the SINAD thing.
BTW.: How would you include squarewave testing, clipping behaviour and testing the amp with comlex loads? In my opinion it tells a lot more about the amp than the SINAD.

Stein

 

[DonH56]

It depends on results in individual disciplines and in a full set of disciplines. It reminds me a bit to car testing (acceleration, manoeuvrability, fuel consumption, space, safety NCAP, you name it). It still needs a qualified operator to know at which level (V, W, ...) is the individual discipline performed. However, I like that it is not based on a single test, like 1kHz/5W SINAD chart.

Thanks Pavel. So do they define the weighting, is it programmable by the user, or an internal thing?

I agree, the 1kHz/5W SINAD chart is as meaningless as it can be when it comes to how an amp behaves and how to sort the amps.
I like your idea Pavel, but I doubt that it will be approved by the people that focus on the SINAD thing.
BTW.: How would you include squarewave testing, clipping behaviour and testing the amp with comlex loads? In my opinion it tells a lot more about the amp than the SINAD.

Stein

I do not think it is "meaningless" since the chart shows a very wide range of SINAD performance. But, it is only the beginning... Back when I was doing R&D stuff several customers defined weighted metrics that combined several parameters into a single number, again hiding a little too much...

If I were to include square waves and clipping, something I suspect is very time-consuming and a little scary since these are customers' devices and not manufacturer loaners, I would note any ringing/overshoot (%) in the square waves and clipping would be a simple pass/fail based upon how well the amp (whatever) recovers after clipping. Which would have to be defined, natch.

 

[restorer-john]

If I were to include square waves and clipping, something I suspect is very time-consuming and a little scary since these are customers' devices and not manufacturer loaners, I would note any ringing/overshoot (%) in the square waves and clipping would be a simple pass/fail based upon how well the amp (whatever) recovers after clipping. Which would have to be defined, natch.

Remember this discussion? :

does it make sense to test how amps behave when they clip?

Yes just in case that the probe 1:10 was not calibrated and trimmed with 1kHz calibrator square. I think it was the case and you probably hit the nail. Before ANY measurement with 1:10 probe switch position, calibration is necessary. We may have more than one scope ....... with the same probe...

www.audiosciencereview.com

 

[kemmler3D]

I don't think a go/no-go recommendation on amps or DACs is particularly useful, unless you literally don't understand any of the measurements including SINAD.

SINAD already boils it down to 1 metric, which is too simplistic for most purchase decisions, but it gives a good starting point.

Beyond that you need an idea of connectivity, compatible loads, and so on, to make a decision. I guess you could filter recommendations by those criteria, but I don't think there is a great way to make the decision FOR someone.

 

[DonH56]

Remember this discussion? :

does it make sense to test how amps behave when they clip?

Yes just in case that the probe 1:10 was not calibrated and trimmed with 1kHz calibrator square. I think it was the case and you probably hit the nail. Before ANY measurement with 1:10 probe switch position, calibration is necessary. We may have more than one scope ....... with the same probe...

www.audiosciencereview.com

I did not remember that exact thread, but that is exactly what I had in mind, and shows why it is not all that simple a thing to put into a simple metric. Thus my pass/fail cop-out...

I am uncomfortable with the assumption that all amps today have "nice" clipping behavior. Amir tested some AVRs (I think) a few years ago and showed some pretty nasty clipping responses.

Trust you to find that one, thanks John!

 

[rwortman]

I read the reviews for information. Recommended/not recommended is just one man’s opinion. I form my own.