Master Thread: Are Measurements Everything or Nothing?

[pma]

What if? What if NAD and PS Audio engineers deliberately added some odd and even harmonics, with specific magnitude ratios, so the precepted sound to the human ear will be optimized to have these audiophile characteristics, such as “warmth” and “depth”, which cannot be captured by the Audio precision analyzer? Maybe they did some research and came up with this secret recipe for an harmonic contamination pattern that does magic to the sound perception, but results in a cr@ppy SINAD?


What if we are all wrong here? Is it time to shut down ASR and lock Amir up?

Did anyone research this? Is there an AES article which correlates harmonic contaminations to sound perception?

Sorry, this is similar to reading fake news. Distortion does not bring any "optimization" to human perception of reproduced music. Audibility of different distortion spectrum profiles has been investigated and it is well known that lower order harmonics are less audible than higher order harmonics. And masking curves show it with respect to frequency and level.
And yes, there have been research studies on audibility of distortion profiles

https://www.researchgate.net/public...tortion_Profiles_in_Relation_to_Audio_Quality

 

[Mathias]

What if we are all wrong here?

A good measurement does not mean it sounds good. It means it sounds right. That's what high fidelity is all about.

 

[sergeauckland]

I agree with your conclusion but I don’t agree with your noise threshold. I think it should be a bit lower still.

One can argue about this, but a noise level of -60dB below programme levels is inaudible. Even -40dB is inaudible. The issue is more that when the quiet bits come along, even -40dB needs to be a lot less relative to peak levels, so -80dB is pretty much there. -90dB better still but unless one has a very quiet room, I suggest that -80dB is normally quite enough. I'm not referring to headphones of near-field listening, as I don't do either to any extent, almost all my listening is in-room, several metres from the loudspeakers.

S

 

[EB1000]

The NAD is a preamp/streamer isn't it? What is the amp you're comparing with the 4700?

Both were using the same external power amp - Hypex NC400 DIY kit (monoblocks). I've listened to the C658 with and without Dirac Live. Even my daughter taught the NAD sounded better. SPL was the same (tested with my phone).

I can understand that anything below -60dB is inaudible, but I can't explain why the NAD, which scored below the X4700H, sounds better?

 

[Colonel7]

Both were using the same external power amp - Hypex NC400 DIY kit (monoblocks). I've listened to the C658 with and without Dirac Live. Even my daughter taught the NAD sounded better. SPL was the same (tested with my phone).

Its possible that your phone db measurements are off by 0.5 db or more which could be the difference. Pretty sure the technically savvy here would tell you to measure across the terminals to get 0.1-0.2 db margin of error.

 

[Roland]

I think we could all be wrong about the assumed correlation between measurements and sound. I had a Denon 3700x and didn’t like the sound coming from my Revel f206s; the Yamaha RX-A 3080 sounded better. It shouldn’t have done based on the measurements, but it did. My Naim Supernait 2 sounds better than either, but it shouldn’t according to the measurements.

 

[EB1000]

I think we could all be wrong about the assumed correlation between measurements and sound. I had a Denon 3700x and didn’t like the sound coming from my Revel f206s; the Yamaha RX-A 3080 sounded better. It shouldn’t have done based on the measurements, but it did. My Naim Supernait 2 sounds better than either, but it shouldn’t according to the measurements.

It's exactly my point. If measurements don't correlate to sound quality, what does then? Are Naim/ PS Audio/NAD engineers design their products based on trial and error, or do they know something that we don't?

Another think I've noticed, I could listen to music for hours on the C658, but I get fatigued after 30 minutes listening with the Denon AVR, even when the same power amp is used...

So what's the point in measurements if they don't matter?

 

[Jim Matthews]

Did anyone research this? Is there an AES article which correlates harmonic contaminations to sound perception?

This is worse looking into...

You said it, brother.

As mentioned above; SINAD is a verifiable, repeatable, measurable proxy for implementation of a good circuit design and proper power supplies.

******

Regarding PSAudio, why buy a more expensive product that has demonstrably lesser performance ?

For me, and my money it's about finding value - typically that's revealed in SINAD evaluations. I would like to see a plot of performance vs. price.

 

[threni]

It's exactly my point. If measurements don't correlate to sound quality, what does then? Are Naim/ PS Audio/NAD engineers design their products based on trial and error, or do they know something that we don't?

Another think I've noticed, I could listen to music for hours on the C658, but I get fatigued after 30 minutes listening with the Denon AVR, even when the same power amp is used...

So what's the point in measurements if they don't matter?

Measurements DO correlate to sound quality, though, because sound quality = high fidelity = faithfulness to the source. If you put in a certain tone into an amp and get something different (other than a different amplitude) out but which you personally prefer then it's simply incorrect to say it has a higher sound quality. It is objectively worse than a better performing amp. It would be more accurate if you say "this amp is good but I prefer the amps which introduce distortion". If this site consisted purely of Amir typing what he thought of this or that amp today, after a few glasses of wine, it would be...well, like reading pretty much any other audiophile website!

 

[MakeMineVinyl]

So what's the point in measurements if they don't matter?

Measurements do matter, but its important not to fixate on a single one like SINAD and look at all of them to try to get an objective picture of a component's performance. But the problem is that these are objective in nature and can't possibly take into account things like 'what is the distortion signature' - what is the spectra, and thus audibility, of the noise - is it above the audio band? etc. With speakers for example, if you don't like the sound of a particular speaker, all the 'great' measurements in the world don't matter squat.

With electronics, my system would absolutely flunk this forum's objective measurements on such things as SINAD. I don't go out of my way to build/buy gear which has 'poor' performance, but I balance many parameters of performance and utility to work to my satisfaction as a whole system. As one example, my (vacuum tube) amplifiers have typical or somewhat lower 'distortion' associated with tubes - they would thus fail a SINAD test. But the other component - the noise - is incredibly low since I use highly efficient full range horns, and any noise, even that associated with good solid state amplifiers would be excessive and audible.

So I have extremely low noise which would probably pass the 'noise' part, but these components would flunk the 'distortion' part. See my point here? A single measurement such as SINAD can be utterly useless if taken as a single figure of merit without considering other aspects.

 

[abdo123]

Measurements do matter, but its important not to fixate on a single one like SINAD and look at all of them to try to get an objective picture of a component's performance. But the problem is that these are objective in nature and can't possibly take into account things like 'what is the distortion signature' - what is the spectra, and thus audibility, of the noise - is it above the audio band? etc. With speakers for example, if you don't like the sound of a particular speaker, all the 'great' measurements in the world don't matter squat.

With electronics, my system would absolutely flunk this forum's objective measurements on such things as SINAD. I don't go out of my way to build/buy gear which has 'poor' performance, but I balance many parameters of performance and utility to work to my satisfaction as a whole system. As one example, my (vacuum tube) amplifiers have typical or somewhat lower 'distortion' associated with tubes - they would thus fail a SINAD test. But the other component - the noise - is incredibly low since I use highly efficient full range horns, and any noise, even that associated with good solid state amplifiers would be excessive and audible.

So I have extremely low noise which would probably pass the 'noise' part, but these components would flunk the 'distortion' part. See my point here? A single measurement such as SINAD can be utterly useless if taken as a single figure of merit without considering other aspects.

can you share pics and details about your horn?

 

[antcollinet]

Especially with amplifiers, there is much more to how it sounds than SINAD. Just as one example - it's ability to drive real world speakers with varying impedance through the frequency range. An amp that can deliver high peak current at needed frequencies will do a better job than an amp (with the same SINAD) that can't.

An amp with the noise spread through the spectrum rather than higher peaks at just a few frequencies will sound better (probably) with the same SINAD.

But as Amir has pointed out - it is an indication of the competence of the design team. If they can get SINAD right, you can have a good handle on whether they know what they are doing in other areas.

 

[blueone]

SINAD therefore is a proxy and litmus test for design excellence.

I wonder how many times you have to say this before everyone gets it? Perhaps a short "Why I measure SINAD" article would help. (I didn't see one in the Audio Reference Library forum.)

 

[solderdude]

Why are a lot of folks so hung up on SINAD ?
Where does the belief come from that SINAD is said to be the holy grail and all important.
It is NOT coming from ASR it is what certain people SAY ASR (Amir) stands for.

SINAD is just one of the aspects that are measured and NOT the most important one either nor is it claimed to be.
It is also obvious that technical signal fidelity on a higher number means less alteration of the signal so is technically better.
That doesn't mean some don't prefer something else.

Is there really consensus about the distortion character ?
Is that true for all levels or must it alter at different levels because our hearing is not linear ?
Can this preference be shown to exist in laboratory conditions ?
What's the magic ingredient ?
Is there more than one way to Rome ?
Does lower signal fidelity have a curve and certain character ? I mean you can slowly make technical performance worse and find a sweet spot where it improves in fidelity and then gets worse ?
Suppose we have a DAC doing this and an amp doing this, does it mean it is over the 'maximum' fidelity again and worse ?
Or do we have to 'match' specific devices in order to obtain maximum SQ ?
Audio and audiophile has been around for decades .. it has been researched to death... do findings disappear in drawers from manufacturers and no one can read them (their own coca cola recipe).

For some technically better is preferred as one does not have to worry about noise levels or audible levels of distortion.
There is nothing wrong about testing equipment for technical performance.
This is what ASR is about and for that reason is it time to shut down ASR ?
Locking Amir up with his test equipment will only lead to more reviews so by all means lets do that.

It says something about distortion of a 1kHz sine wave at a specific level and the noise and distortion distance in a specific load.
Nothing more.

 

[brandall10]

SINAD is just one of the aspects that are measured and NOT the most important one either nor is it claimed to be.

I think this probably comes from the fact that it's the first measurement in a review, and it does seem Amir gives it outsized weight. Right off the bat, if it's toward the top of the range, it's proclaimed to be in the top x% of measuring DACs. Time and time again if it's unusually high Amir usually says "Do you see what I'm seeing here?!" Being excellent in other measurements usually doesn't elicit this type of response.

Personally, as a layperson here, I give it more weight than anything else. Of course I'll go through the rest of the measurements to see if nothing else is amiss, and to the final verdict, but that's how interpret reviews - if SINAD is in check, I tend to believe it will be an instrument grade product.

 

[solderdude]

I think this probably comes from the fact that it's the first measurement in a review, and it does seem Amir gives it outsized weight.

Agreed. just when it was the only thing of importance then Amir could just mention the SINAD number and leave it at that but he doesn't for good reason.
I can understand many folks read it this way and the vast majority of readers have no real clue what all the other plots mean.

As a more technical minded person I would like to see even more (specific) tests done as the performed test don't always show enough for me, but it is better than nothing or just someones subjective opinion.

Most folks just want to know what is 'best' and look for a single number (like transducer ratings).
That's the whole issue.
One has to look at all measurements and understand them. A tall order for less technical people for sure. There is no solution for that other than education.
Education is what one gets by reading comments, asking questions and get pointed towards relevant info.

 

[Harmonie]

That's an old debate.
True, accurate doesn't mean nicer or more pleasing.
Other examples: Photography.
How many pictures have been edited to look "nicer".

You can find some examples on youtube in which they use some distortion gear on a harp and make interesting sound.

Depends what your after @EB1000

 

[amirm]

I think this probably comes from the fact that it's the first measurement in a review, and it does seem Amir gives it outsized weight. Right off the bat, if it's toward the top of the range, it's proclaimed to be in the top x% of measuring DACs. Time and time again if it's unusually high Amir usually says "Do you see what I'm seeing here?!" Being excellent in other measurements usually doesn't elicit this type of response.

It is not the first thing in a review. The first thing is a full dashboard with a number of pieces of information. It is all presented at once. Some of it is not interesting like the frequency unless it is. In a recent DAC review, I actually highlighted that instead of the rest of the info: https://www.audiosciencereview.com/...dio-gd-master-7-singularity-review-dac.26460/

That was the first graph because it was very informative yet didn't impact SINAD.

The dashboard also had other comments beyond SINAD:

As you see, I noted with question mark the extra spectrum around FFT. Again that didn't impact SINAD.

With phone dongles, one of the most important factors is output voltage. So I always note that way at the beginning as well. With DACs, I also note it if it is above nominal and what I had to do to bring it down.

When I bring up the dashboard, I scan it all and look for what is informative or interesting about the design. SINAD as a summery figure does that pretty quickly so it is a fixture in my reviews and for good reason. But I never stop at it. Each review has half a dozen to a dozen more measurements beyond what else is in the dashboard. If all I cared about was SINAD, I would not bother with the rest but I do at great expense in the form of time to measure and post.

The reason SINAD has gotten so much "PR" is because it is ranked. I believe strongly in that as otherwise, you have no context of what the measurement means. In US appliances have a range as well:

They could have just mentioned the $67 number but who knows and memorized the full range for such an appliance? No one. Same with SINAD. I put in the table and now people immediately know how good or bad it is. I then cut it down to just quarters to show that precise value (except way on top) is of little importance.

If a product lands in anything but blue category, then they could have done better. Maybe for the price it is OK to be lower. Maybe not. I like to see companies do the best possible job they can to reduce noise and distortion especially if it cost next to nothing. If they don't care, then they don't care.

 

[amirm]

To be clear I have huge admiration for companies that peg the SINAD to max. I now what effort it takes to scrutinize every bit of detail in design and manufacturing to get there. I like to highlight such dedication to excellence. When Toyota brought out its Lexus branded cars to US, they used to run commercials like this:

No one drives a car with glasses on the hood so the application to real life is nil. But it does show great attention to design and removal of vibration.

By the same token, I show my appreciation when this level of accuracy is achieved in an audio product.

At the other extreme, a poor SINAD is like someone spitting in our face saying, "I didn't have to design anything good." To that I say fine, here are the results and see where you land in this chart.

Ultimately this is the philosophy of what I do. After testing nearly 1000 audio product, I analyze a product and give you an opinion about it.

 

[brandall10]

Agreed. just when it was the only thing of importance then Amir could just mention the SINAD number and leave it at that but he doesn't for good reason.

Just quoting the comment at the end of my post again: " Of course I'll go through the rest of the measurements to see if nothing else is amiss, and to the final verdict, but that's how interpret reviews - if SINAD is in check, I tend to believe it will be an instrument grade product."

To me, it's the main gut check. Reading the rest of what you wrote and what Amir just wrote, nothing changes my opinion on that. It's the canary in the coal mine of it being an excellent measuring product - if there's a short fall elsewhere, it's usually relatively minor. Part of this is likely due to manufacturers knowing this is in fact how ranking is done, and also the fact that it shows they put some care into measurements themselves.