Master Thread: Are measurements Everything or Nothing?

[Killingbeans]

can’t we just write a short paragraph mentioning measurements and blind testing are important in the forum’s landing page to keep parrots from thread shitting over and over again?

If only it was that easy. We've already done a thousand iterations of that, and none of them seem to work.

The newcommers are usually very insistent on ignoring them, and just start presenting their impressions as a solid foundation for facts.

Call it "parroting" if you will, but the most sensible thing is to keep making the implausibility of things as clear as possible.

 

[Zensō]

a great audio system is a work of art

Wha? Sorry, but I have to strongly disagree.

if it was purely based on coupling the “best”measuring components a deaf monkey could do it.

Pretty much. It’s not rocket science… or art.

 

[Cote Dazur]

i appreciate this forum. measurements are a critical element in narrowing focus on what to audition and are a fundamental aspect of any review worth its salt. i also admire the steadfast belief in double blind testing to ensure psychology is not coloring impressions of a new “key” element introduced in to a chain.

If you appreciate then you understand why it is needed to be repeated every time someone comes with a subjective opinion on anything that has zero value for anyone, not even for the one making the subjective comment.
The tone and content of the balance of your post clearly shows you are not really as appreciative as you write in your first paragraph (of your first post on ASR).
Hopefully, with time, you will truly understand the immense value that ASR brings to the people interested in pursuing high fidelity sound and be well prepared to detect snake oil peddlers.

 

[thor79]

Measurements are essential for
checking whether or not an appliance is working electrically as intended.

With regard to the purpose for which we use hi-fi, I must say that data is almost as valuable as the indication of the alcohol content of a bottle of wine.

Why am I here, because I am an engineer and interested in how data for hi-fi can achieve the same value and meaning as data for a measuring instrument, which paradoxically is the only way we have to measure hi-fi besides the ear and the brain.

 

[solderdude]

When you are an engineer I would say that data like only a single THD number in a spec sheet is as valuable as the indication of the alcohol content of a bottle of wine.

When you are an engineer that knows about audio gear and look at a more elaborate set of measurements then you should already have far more valuable information at hand.

When you are an engineer that knows about audio gear and perception and a full measurement suite (so more than the average review on ASR) that would be all you need to know.

The correlation between measurements and perception is something that will not be there for most people.
The reasons for that could be:
Lack of proper measurements
Lack of understanding measurements
Lack of faith in measurements

When you are interested in measurement data and listening you could best look into perception and audible limits before venturing in understanding what measurements can show what and when higher fidelity does not lead to improved sound quality any more.

 

[Michael Larkin]

o

OTOH, there's quite a lot of evidence that placebos in their proper medical context actually do work, and sometimes permanently (as with fake knee operations):

Yes, the Placebo Effect Is Real, Not a Trick

The fact that you may start to get better if you believe you are receiving treatment is one of the best-attested facts in medicine. Despite that, far from being accepted, this “placebo effect” is seen…

mindmatters.ai

8 Placebo Effect Examples in Real Life – StudiousGuy

studiousguy.com

also this:

In respect of audio gear, I think the point some make is that the "placebo effect" is responsible for subjectivists thinking that some gear is better than others. But my point is that this could apply both for subjectivists and objectivists.

How so for objectivists? Well, if one believes that measurements are best in indicating effectiveness of DACs, amps and speakers, one is more likely to consult a site like ASR, and to choose from the best-measuring gear that is within one's budget. And when one purchases it, one is more likely than otherwise to think it sounds good because one believes in measurements.

But does a piece of gear that measures well necessarily sound good? That's a $64,000 question. As is the obverse: if it sounds good based on a personal, subjective evaluation, does it necessarily measure well?

I genuinely believe that Amir is somewhat unusual in that he has trained himself to pick up on subjective differences in sounds that correlate with certain objective measurements. But has he also inadvertently trained himself (because of his objectivist leanings), to think this means his measurements are reliable indicators of sound quality -- judgements that more subjectively-oriented people would be silly to ignore?

I idly wonder whether subjectivists and objectivists might be two sides of the same coin. The main thing that differentiates them is that one side believes passionately in measurements and the linkage of those with sound perception. As for the other side, the belief is that good sound depends more on subjective impressions.

The common element is that there's belief on both sides. But how is to one to know that a degree of correlation of sound quality with objective measurements is the whole story? I mean, why does Amir's table of SINADs for DACs go from "best" (highest) to "worst" (lowest)? If anyone can objectively show gradual degradations in the perception of sound as one goes from highest to lowest, then please supply the relevant data, because I'm not confident in my own mind that that would prove to be the case. The 18th DAC down in the list might or might not sound better than the 65th to at least some listeners who were unware of Amir's findings.

Why do we prefer some kinds of sound perception to others? Why do I rate the four DACs I own, from best to worst, in the order [prices at the time of purchase]: Aune X8 viii + sparkos op amp + Aune XP2 power supply (total £470) -- Earmen Tradutto (£699) -- Ifi Zen DAC V2 (£139) -- Topping E30 (£110)? Buggered if I know, but it sure as heck has little to do with measurements or online subjective reviews (even though the latter did to some extent influence my purchases). Others might rate them differently, and if so, that's fine, but where would they stand according to SINAD measurements? I think Amir places the E30 above the Tradutto, but to me, it's the other way round. I'm not sure he's reviewed the other two.

At the moment, I try to avoid both excessive objectivism and subjectivism. But I can't avoid the latter completely because in the final analysis, given acceptable equivalence of features and convenience, the deciding factor is how a piece of kit sounds subjectively to me. Amir's measurements may say something ranks objectively very highly, and yet to me it might sound worse than something that ranks lower. And vice-versa for that matter. I guess I'm suggesting that proponents on both sides might chill out a bit and accept that there could be a degree of merit on both sides of the objectivist/subjectivist divide.

 

[pkane]

OTOH, there's quite a lot of evidence that placebos in their proper medical context actually do work, and sometimes permanently (as with fake knee operations):

So that's why every proper medical study has a group of test subjects who receive placebo? To figure out how to effectively fool patients into being healthy? Or is it to eliminate the possibility of imaginary effects from skewing the (objective) results of the study? Hmm....

 

[fpitas]

Measurements are essential for
checking whether or not an appliance is working electrically as intended.

With regard to the purpose for which we use hi-fi, I must say that data is almost as valuable as the indication of the alcohol content of a bottle of wine.

Why am I here, because I am an engineer and interested in how data for hi-fi can achieve the same value and meaning as data for a measuring instrument, which paradoxically is the only way we have to measure hi-fi besides the ear and the brain.

No offense intended, but you have a lot to learn about audio.

 

[VQR]

In respect of audio gear, I think the point some make is that the "placebo effect" is responsible for subjectivists thinking that some gear is better than others. But my point is that this could apply both for subjectivists and objectivists.

Two opposing sides means nothing toward validity. One asks for proof, the other refuses and makes excuses. When pointed toward evidence, what invariably happens is:

Objectivists: I think you fell for placebo.
Subjectivists: No u

It's a childish example, but when you scrape past word fluff you often find that to be the case. Sometimes they invoke lies that Amir and others say a DAC with worse SINAD, say, 100 dB, is going to sound worse than one with top tier SINAD which is currently 123 dB SINAD. There's a good thread that gives a starting point on when our hearing won't notice what the analyzer can.

Amir's gone on record several times saying higher SINAD is not a reason to dump your transparent and good DAC for the latest and greatest. Often, SINAD is just a way to tease out if there's any immediate flaws with a DAC, such as PSU noise and harmonics.

There are 32 tone tests than can tease out IMD or distortion across the audio band. We get linearity tests to see how consistent the signal is across all frequencies. It's not uncommon for some boutique HPA or speaker amps to fail at lower frequencies.

For amplifiers especially, it's tested how they react at different resistance loads, what power you get, and at what points the power sags. We see what the noise differences are between gain settings. This is all good information to know!

The lie often found on Head-fi and SBAF that ASR only cares about SINAD and doesn't listen to music. Any headphone, speaker, and HPA review proves that false upon just reading.

 

[Killingbeans]

I idly wonder whether subjectivists and objectivists might be two sides of the same coin. The main thing that differentiates them is that one side believes passionately in measurements and the linkage of those with sound perception. As for the other side, the belief is that good sound depends more on subjective impressions.

There's no doubt that both "sides of the fence" are equally susceptible to bias.

It's been said before (a lot) but the whole "subjectivist" vs. "objectivist" deal is completely pointless. We are all subjectivists.

I think the thing that makes an "objectivist" like me, is a focus on probability rather than possibility.

I prefer "That's not very probable, so lets spend our time and energy on something else" over "Think of the possibilities! Lets just roll with it".

If anyone can objectively show gradual degradations in the perception of sound as one goes from highest to lowest, then please supply the relevant data, because I'm not confident in my own mind that that would prove to be the case.

I would instead make the radical guess that nearly all DACs on the list would sound practically indistinguishable when tested blind and level matched.

You'd have a hard time finding users in here who believe that 'better measurements == better sound'. Most of us will make a judgement on the "sound" of a piece of gear based on our best current understanding of the thresholds of human hearing and the different masking effects being at play.

 

[IPunchCholla]

Measurements are essential for
checking whether or not an appliance is working electrically as intended.

With regard to the purpose for which we use hi-fi, I must say that data is almost as valuable as the indication of the alcohol content of a bottle of wine.

Why am I here, because I am an engineer and interested in how data for hi-fi can achieve the same value and meaning as data for a measuring instrument, which paradoxically is the only way we have to measure hi-fi besides the ear and the brain.

You should read Cork Dork. The author becomes a sommelier and explains the process of how you do it. You can get a pretty good idea of what a bottle of wine should taste like by knowing four things: the amount of alcohol, how sweet/dry it is, how full-bodied it is, and its (legitimate*) tasting notes. The first three can be measured by instruments. Perhaps the 4th as well. So give the number of elements measure for electronics on this site, it would be more like printing the alcohol, sugar, and specific gravity on a bottle of wine, along with the vintage (Though the vintage alone will tell you those things to a large amount if you are really in the know). Sommeliers use the first three and a ton of experience in tasting to identify down to the vintage an unknown wine.

*interestingly, like hearing, we often overestimate our ability to discriminate by taste. Also like hearing, taste is very open to suggestion and bias. In Cork Dork, the author points out that we can only process two or three-ish tastes at any given time, so those long lists of tasting notes are pretty useless as the more minor ones (if they exist) will be overwhelmed by the more prominent ones. Masking effects apply to taste as well. Wine could easily have its own science site (maybe it does, I’m not into wine).

 

[Cote Dazur]

"placebo effect" is responsible for subjectivists thinking that some gear is better than others. But my point is that this could apply both for subjectivists and objectivists.

Of course it does apply to both, we are subject to bias. Still there is a lot of differences between both approach. As already mentioned, we are all subjectivist, no one is immune. Where we differ is how we proceed. Objectivist know they are subject to bias, in order to help themselves get a better grasp, they will try to gather knowledge to help understand what is objectively happening to make better decision. Subjectivist usually know little about anything and believe their ears can be trusted above all else.
We are all fumbling in the dark, subjectivist are totally blind, objectivist have a little bit of light, not enough to understand everything about their surrounding but enough to avoid falling off the cliff.

 

[antcollinet]

You should read Cork Dork. The author becomes a sommelier and explains the process of how you do it. You can get a pretty good idea of what a bottle of wine should taste like by knowing four things: the amount of alcohol, how sweet/dry it is, how full-bodied it is, and its (legitimate*) tasting notes. The first three can be measured by instruments. Perhaps the 4th as well. So give the number of elements measure for electronics on this site, it would be more like printing the alcohol, sugar, and specific gravity on a bottle of wine, along with the vintage (Though the vintage alone will tell you those things to a large amount if you are really in the know). Sommeliers use the first three and a ton of experience in tasting to identify down to the vintage an unknown wine.

*interestingly, like hearing, we often overestimate our ability to discriminate by taste. Also like hearing, taste is very open to suggestion and bias. In Cork Dork, the author points out that we can only process two or three-ish tastes at any given time, so those long lists of tasting notes are pretty useless as the more minor ones (if they exist) will be overwhelmed by the more prominent ones. Masking effects apply to taste as well. Wine could easily have its own science site (maybe it does, I’m not into wine).

The wine analogy is always applied (to audio) incorrectly.

The wine is the music. Not the gear. The analogy to the gear is the bottle and glass. If they are dirty they contaminate the wine.

You don't need to measure wine to find out how it is going to taste - you just enjoy it, like you just enjoy music. You do need QA to check the cleanliness of the bottles, and you need to make sure you are using a clean glass so as not to spoil the wine - just as you need to make sure your audio reproduction gear doesn't apply distortion and noise, and spoil the music.

 

[IPunchCholla]

The wine analogy is always applied wrongly.

The wine is the music. Not the gear. The analogy to the gear is the bottle and glass. If they are dirty they contaminate the wine.

You don't need to measure wine to find out how it is going to taste - you just enjoy it, like you just enjoy music. You do need QA to check the cleanliness of the bottles, and you need to make sure you are using a clean glass so as not to spoil the wine - just as you need to make sure your audio reproduction gear doesn't apply distortion and noise, and spoil the music.

True, it is a bad analogy. Also wine, like music, (and paintings, dance, etc) can be measured and described. It’s just that the value can not be assigned, as that is largely at the discretion of the experiencer.

Art isn’t indescribable.

 

[Xulonn]

True, it is a bad analogy. Also wine, like music, (and paintings, dance, etc) can be measured and described. It’s just that the value can not be assigned, as that is largely at the discretion of the experiencer.

Art isn’t indescribable.

I enjoyed many inexpensive fine wines in the day based on blind tasting sessions performed by groups of friends. The hosts were wealthy and could easily afford any wine they wanted, but they preferred value to marketing hype and "vinophile" reputation.

As with audio, "accuracy" in sighted wine tasting is limited by confirmation bias. Great cheap wines were extremely rare, but there were many gems in the mid-price range. And of course, there are superb wines in the expensive classes, but also some brands and releases that were vastly over-rated.

Ab out 25 years ago, I brought a bottle of Napa Valley wine and 3 years later, paid the $10 corkage fee to drink it with friends at a nice little obscure Sausalito restaurant. It was for a birthday party for the wife of the host of those blind tasting gatherings. The wine was a Grgich Hills Cabernet Sauvignon from a young Napa vineyard that I had purchased for $25 from a local wine shop in El Cerrito, California, across the bay from San Francisco. We agreed thst it was one of the best Cabs we ever had, and when the birthday lady went to buy some, the price had gone up to $60 - in 3 years. So she didn't buy it, and continued her quest for low-priced bargains.

Typically, unlike wine, audio component prices don't go up sharply after a good review and increased popularity - but if anyone has stories about such price increases, I'd enjoy reading them.

 

[Michael Larkin]

Of course it does apply to both, we are subject to bias. Still there is a lot of differences between both approach. As already mentioned, we are all subjectivist, no one is immune. Where we differ is how we proceed. Objectivist know they are subject to bias, in order to help themselves get a better grasp, they will try to gather knowledge to help understand what is objectively happening to make better decision. Subjectivist usually know little about anything and believe their ears can be trusted above all else.
We are all fumbling in the dark, subjectivist are totally blind, objectivist have a little bit of light, not enough to understand everything about their surrounding but enough to avoid falling off the cliff.

Reminds me of the Mulla Nasrudin story about the man who looked for his lost keys under a lamppost rather than in the dark where they were actually lost. To what extent might shining a light on things like SINAD parallel this? Could one be looking in areas that don't have conclusive significance when it comes to sound perception?

BTW, I just realised that I inadvertently posted this in a thread I didn't mean to -- hence my inclusion of the video from gavagai, posted elsewhere. But no matter, luckily I don't seem to have posted it in a totally irrelevant thread.

 

[Cote Dazur]

Could one be looking in areas that don't have conclusive significance when it comes to sound perception?

Of course, once you have objective information, you still have to decide how relevant it is to your situation.
On the other hand no knowing any information does not allow for any decision, just fumbling in the dark. Leaving our self open to any snake oil argument from any peddlers so happy to deal with a group of ignorant, superstitious people.
Objective information piles up, their is more that SINAD to objective information. Once you are educated you can help yourself and help other. You are very new here, take you time, learn as much as you can, it gets easier the more you know.

 

[Killingbeans]

Could one be looking in areas that don't have conclusive significance when it comes to sound perception?

Not when you stop to think about what an electrical signal actually is. A lot of people seem to swear to some variation or another of the idea that a time varying voltage can somehow hide crucial information in plain sight. I seriously can't fathom the logic in that. It can't happen without resorting to magic.

Acoustics on the other hand... Ohh boy. Stereo playback with conventional speakers is parlour trick. A very convincing one (for what it is) with a relatively high WAF, but still just a parlour trick. There aren't any big mysteries about audio as signals to be solved, IMO. There's only further progression to be had in making the acoustic illusion more practical and convincing/non-offensive.

 

[DonR]

Typically, unlike wine, audio component prices don't go up sharply after a good review and increased popularity - but if anyone has stories about such price increases, I'd enjoy reading them.

Yes, usually it's the other way around. A high price leads to good reviews.

 

[thor79]

No offense intended, but you have a lot to learn about audio.

Unfortunately no, I have a lot of knowledge and not least a lot of listening experience with hi-fi equipment that I have built from scratch.
Have worked a lot with measuring instruments , there is a commonality between this and audio but also big differences, the biggest in my opinion is that not everything can be measured/simulated/theoretically proven and it is therefore left to the ear and brain to "measure"

There are different perceptions of what hifi is combined with different experience levels this can make a big difference, in discussions and results

The goal for ASR as I understand it is for the sound to reach the listener with as little measurable impact on the signal as possible.

Others don't care about this, hifi is what they like, and after the rebirth of EQ it is now easy to meet this requirement

Myself and others see hifi as a representation/illusion of a current musical event where there is a reference, a reference one should have a good knowledge of. Recordings must be found, or one must make them self.

The ASR approach more or less closes itself around what can be measured and the threshold one sets for human hearing.
If no difference can be measured, the conclusion is no audible difference, or maybe the difference may be so small that it is below the human hearing threshold.

The problem with this approach is that many of the sound differences that millions of hifi enthusiasts have heard, also in blind testing, including myself disappears, the evolution one experiences by modifying the same designs over time disappears. Sound differences on cables, components, resistors, capacitors, power cables, opamps , devices whose measurements give values below the threshold of what a human can hear. DAC , amplifiers partial power amplifiers, all these sound differences disappear and apparently only loudspeakers and acoustics make a difference,

I could cite countless examples of this approach not being consistent with reality.
But will just mention one where three guys test class D the new Hypex Purifi and ICEpower

Live Test - 3 Times Next Gen Class D - Hypex - Purifi - ICEpower

YES! The first live stream of this year! And we're going to make it another great one! This time: 3 x NEXT GEN class D. The latest Hypex Nilai, the latest ge...

www.youtube.com

The Hypex and Purifi measure very similarly, and many of the measurements are below audible, yet the three hear crucial differences in sound, depending on music selection among other things.
Yes, no blind tests etc, there are actually people who can profitably judge hifi without blind tests and achieve progress and correct ratings. Believe it or not
I would also claim that I can change the sound significantly on all three amps in a short time without the measurements changing, very small changes can occur but would not explain the massive perceived sound difference.

Therefore the ASR approach to hifi is not appropriate in my opinion, but it is the only thing we have at the moment besides the ear and the brain, good measurements are nice to have ,agree , strive for it myself, ,but it is no guarantee for a good illusion/reproduction but not a hindrance either , the direct opposite , bad measurements and low Sinad on the other hand can curiously give a sound experience in the absolute top class , thinking here of tubes and gramophone.
It would also be interesting to investigate how much bias belief in measurements affects what is heard , does Amir measure before he listens or what .