Master Thread: Are Measurements Everything or Nothing?

[amirm]

I have asked that question, sometimes to EEs who design amplifier circuits. The answers:

1) Capacitors and resistors have sonic characteristics that cannot be measured
2) The way in which feedback is implemented affects sound quality in a way that cannot be measured (in some aspects).

So how do they design their amps? Randomly fiddling with parts 'till it sounds good to their ears covered by gray hairs???

Sadly, that is the answer for some. They don't perform any controlled listening tests. They hypothesize something, listen to it, or even perform an AB test, and declare it a winner. That then becomes the horse they ride into town. "Feedback is bad so we don't use it."

Even more sad is that they get their ideas for such things from lay audiophiles who don't really know what a resistor is, or what feedback means! It is a viscous circle of confusion with the two antidotes being measurements and controlled listening tests. Both of which they abhor as your post indicates.

Best to run such claims by someone who is not designing such amps. Here is the most well-known electrical engineer on the Internet, Dave Jones:

The whole thing is a fun watch but I queued it up to the relevant section.

 

[Chagall]

Despite your experience and controls you admit you can't be trusted to make an objective decision?
I'd love to see the reaction of a judge to being told he can't be relied on to make objective decisions from the bench.

Do you include AI in your "infallible machines" list, or are there caveats there, too?

Just asking questions...

No, there’s a difference between admitting humans have bias and saying they can’t be trusted. We can be trusted because we acknowledge limits and build systems to mitigate them.

Judges don’t make decisions by their gut. They’re constrained by laws, precedents, evidence rules, juries, appeals courts, even recusal for conflicts of interest. The whole justice system exists precisely because no one assumes judges are perfectly objective.

Protocols aren’t insults to expertise - they are what make it reliable.

 

[Wesseling Audio Lab]

When the Derek Chauvin trial was broadcasted live I watched the entire trial including daily pre and post analysis. It was very interesting and engaging. The prosecution lawyers were brilliant.

Some time after the judge handed down the sentencing he declared that the sentence was "not" a "subjective" decision and said that it was “technical” and went on to say that it was to involved to explain at that time and gave copies of a 12 page document to the lawyers so they could read his technical decision for the sentence.

Interesting Stuff!

 

[ahofer]

Despite your experience and controls you admit you can't be trusted to make an objective decision?
I'd love to see the reaction of a judge to being told he can't be relied on to make objective decisions from the bench.

Do you include AI in your "infallible machines" list, or are there caveats there, too?

Just asking questions...

It strikes me as incredibly naïve to think that humans can be fully objective. There are lots of controls in place to mitigate judicial decision bias, but nobody is averring that they have achieved perfect objectivity (the ‘fair witnesses’ of Heinlein come to mind, along with Kant’s “crooked timber”). How could you have a successful appeal, or rather, an objective reversal-on-appeal?

To answer a snarky question seriously: LLMs have serious problems with factual and (pseudo) judgement accuracy. Machine learning/gradient descent can also take sub-optimal or risky paths. Deterministic methods are only as good as the rules and programming on which they are built. So, yes, there are many caveats.

If you really, seriously, think humans can achieve perfect objectivity, I’d suggest some reading on cognitive bias. My absolute favorite in the field is the foundational “money pump” experiment by Lichtenstein and Slovak. There’s a wonderful interview where she puts the subject’s irrationality right in their face, they admit it is irrational and harming them, but they keep doing it.

Just asking questions..

This type of ‘just asking questions’ is rightly called “JAQing off”.

 

[Speedskater]

It's easy for humans to be objective, if you ask objective questions.

Objective question:
Does "A" sound the same or different than "B" ?

Subjective question:
Does "A" sound better or worse than "B" ?

 

[Ken1951]

It strikes me as incredibly naïve to think that humans can be fully objective. There are lots of controls in place to mitigate judicial decision bias, but nobody is averring that they have achieved perfect objectivity (the ‘fair witnesses’ of Heinlein come to mind, along with Kant’s “crooked timber”). How could you have a successful appeal, or rather, an objective reversal-on-appeal?

To answer a snarky question seriously: LLMs have serious problems with factual and (pseudo) judgement accuracy. Machine learning/gradient descent can also take sub-optimal or risky paths. Deterministic methods are only as good as the rules and programming on which they are built. So, yes, there are many caveats.

If you really, seriously, think humans can achieve perfect objectivity, I’d suggest some reading on cognitive bias. My absolute favorite in the field is the foundational “money pump” experiment by Lichtenstein and Slovak. There’s a wonderful interview where she puts the subject’s irrationality right in their face, they admit it is irrational and harming them, but they keep doing it.



This type of ‘just asking questions’ is rightly called “JAQing off”.

Have always liked Heinlein's "Fair Witness" idea. My wife and I refer to it often. "Astrid", of "Astrid et Raphaelle" would make a superb one.

 

[Chagall]

It's easy for humans to be objective, if you ask objective questions.

Objective question:
Does "A" sound the same or different than "B" ?

Subjective question:
Does "A" sound better or worse than "B" ?

This is a neat distinction, but without controls and protocols, even objective questions are up for debate.

 

[goryu]

It's easy for humans to be objective, if you ask objective questions.

Objective question:
Does "A" sound the same or different than "B" ?

Subjective question:
Does "A" sound better or worse than "B" ?

Both questions appear subjective- one person's "same" or "different" may not be another's- how something "sounds" is subjective. Only a question that can be answered objectively is an objective question- is A louder than B? - loudness can be quantified and verified, with a sound level meter for example.

 

[vddobrev]

Both questions appear subjective- one person's "same" or "different" may not be another's- how something "sounds" is subjective. Only a question that can be answered objectively is an objective question- is A louder than B? - loudness can be quantified and verified, with a sound level meter for example.

I disagree, even "does A sound different than B" can be verified by a wave analysis, obviously the sound of A and B is different.

 

[vddobrev]

It's easy for humans to be objective, if you ask objective questions.

Objective question:
Does "A" sound the same or different than "B" ?

Subjective question:
Does "A" sound better or worse than "B" ?

Excellent example! The first question implies distinction (that can be measured), while the second question implies preference (that cannot be measured)!

 

[Speedskater]

Both questions appear subjective- one person's "same" or "different" may not be another's- how something "sounds" is subjective. Only a question that can be answered objectively is an objective question- is A louder than B? - loudness can be quantified and verified, with a sound level meter for example.

It takes a rather large difference, to identify just what that difference is.

 

[goryu]

The question asks for a subjective reply- does it sound different? What does "different" mean? To answer one must interpret. My wife says "sounds the same to me" no matter what I do to my audio system, even though I can indeed prove the output signal is different in any number of objective ways. And it isn't only my wife- I have received this same response from others as well. I think the question itself needs to be specific enough to eliminate subjective interpretation- "different" allows too much interpretation (the mother of subjectivity) and not enough specificity.

 

[vddobrev]

The question asks for a subjective reply- does it sound different? What does "different" mean? To answer one must interpret. My wife says "sounds the same to me" no matter what I do to my audio system, even though I can indeed prove the output signal is different in any number of objective ways. And it isn't only my wife- I have received this same response from others as well. I think the question itself needs to be specific enough to eliminate subjective interpretation- "different" allows too much interpretation (the mother of subjectivity) and not enough specificity.

There cannot be interpretation of letters A to Z, or numbers 0 to 9. Written or pronounced, distinct and different.

 

[Speedskater]

"different" to a skilled AB/X listener, is a very, very small difference. In a 5 to 10 second test using selected tracks, the only only descriptive word is different.
Nothing like the differences in audiophile reviews.

 

[jools]

It's easy for humans to be objective, if you ask objective questions.

Objective question:
Does "A" sound the same or different than "B" ?

Subjective question:
Does "A" sound better or worse than "B" ?

But the same sound can be perceived as completely different - for example due to the McGurk effect which there is no way around for a human. We hear our expectations as much as the sound.

 

[goryu]

Again, "different" how? In terms of human perception, it's a judgment not a metric.

 

[Chagall]

Again, "different" how? In terms of human perception, it's a judgment not a metric.

Yes, all ABX or similar double-blind tests rely on judgments - either perception or preference. But, how consistently they make that judgment above chance (e.g., 18 out of 20 correct) becomes objective evidence, metric, or data.

Only a question that can be answered objectively is an objective question- is A louder than B? - loudness can be quantified and verified, with a sound level meter for example.

Even this is open to interpretation - measured SPL or perceived loudness?

 

[guy48065]

No, there’s a difference between admitting humans have bias and saying they can’t be trusted. We can be trusted because we acknowledge limits and build systems to mitigate them.

Judges don’t make decisions by their gut. They’re constrained by laws, precedents, evidence rules, juries, appeals courts, even recusal for conflicts of interest. The whole justice system exists precisely because no one assumes judges are perfectly objective.

Protocols aren’t insults to expertise - they are what make it reliable.

At the end of the day it's still humans making rational decisions. Guided by protocols, logic, rules, laws, etc. We've all seen this objectivity crumble, and biases revealed in the daily news but the system remains dependant on human experience and honesty to prevail. Can machines do better? Be more reliable and trusted?

Many of us use AI and have seen what nonsense it can create to answer a simple question. AI has nearly limitless access to world knowledge yet it sometimes fabricates BS responses that any human can tell is a so-called "hallucination".

 

[solderdude]

There are at least 2 subjective reviewers that own/bought an ABX device.
Both made posts about owning these devices.
This was a few years ago already.
Neither of them seems to be using that but very likely have been playing with it.
Neither of them has ever made a post about being able to tell certain devices apart using that device.
Both of them continued to use their ears and flowery words to describe 'differences in sound'.

When using a proper test method (when the differences are small) the ability to tell electronic devices apart by sound signature seems to disappear.
The simplest way around that is to blame the test method and continue to do what the followers/readers expect ... flowery words and magic.

 

[Pearljam5000]

If measurements were everything
Then everyone would buy speakers without even hearing them at all and would just rely on the measurements...
Which is not the case
So measurements... Aren't everything.