Why is audio objectivism so frequently focused on all the wrong things?

[Island_Kenny]

The variation would range from non-existent to minuscule in well engineered gear, through to small to wide variation in less consistently manufactured gear.

Take for instance a single board D/A converter. The variations between functioning products will be absolutely minimal.

Take a hand constructed point to point tube amplifier or a multi PCB amplifier product with considerable inter board wiring and routing issues and the variation would be greater, especially in the THD and noise results. Consider at the extreme low levels of THD in say the Benchmark AHB amplifier, simply cleaning a speaker terminal socket could result in lower THD.

Totally agree on your comments. Unfortunately, a lot of high-end audio toys claim to be "handcrafted".

 

[Blumlein 88]

Do you have a link so I can read up on this?

You might try watching this very helpful video . I'd suggest it as a first step as it covers the basics and how they broke through the circle of confusion.

Sean Olive in various blog posts here covers much of what Harman does.
https://seanolive.blogspot.com/2012/11/behind-harmans-testing-lab.html


If this info is interesting you really need to read Toole's book.
https://www.amazon.com/Sound-Reprod..._1?keywords=floyd+toole&qid=1572231617&sr=8-1

And as someone mentioned, Dr. Toole is a member and posts here. You could search for some of his posts on these topics.

 

[Dogen]

This is my fundamental disagreement. I think that the blind test is the thing that matters, because the ultimate goal of an audio playback chain isn't to make an analyzer happy, it's to make humans happy. The analyzer is only a proxy for what humans like, and it only has value if its measurements can be correlated in a meaningful way against human preferences.

If you're not doing that human-perception check on it, you can't actually know whether you're measuring the right thing. Look at the example of TIM in the early solid-state days, as I linked above, for instance: By using gallons of negative feedback, manufacturers were able to create amps that measured amazingly, but sounded bad; later on, they discovered that a thing they hadn't been measuring was correlated to that heavy use of negative feedback, and that's what was causing it to sound bad.

If you didn't take seriously the human-perception check there, you'd just go forward, making things that "measured great" via measurements that didn't capture all the relevant factors, but that were actually pretty bad. And if you were to castigate "subjectivists" for not liking your "great measuring" stuff, you'd be the one who ultimately had egg on your face, not them.

A blind test can never conclude that A sounds better than B (makes humans more happy) unless it also proves that test subjects can reliably tell the difference between A and B. Testing the first hypothesis rarely produces positive results with adequately measuring equipment, so there’s little hope of proving the second hypothesis.

Let’s first test the first hypothesis in double blind trials, then we can move on to B

 

[Guermantes]

@Mikey, is it possible that your own premise is focused on "the wrong things". As I always believed, the goal of hi-fi was the best possible reproduction of the recording, i.e. the highest fidelity. To me the "best" means reduction of distortions and other non-linearities, attention to the dynamic response and bandwidth of the system in order to give me what I really want to focus on: the music (or spoken word or whatever has been captured). These are all measurable engineering goals. Perhaps your existential angst would be better directed at the IEEE or AES for promoting those goals, not Amir or this site.

In essence we focus on the equipment so that in use it disappears -- the best sounding gear should have no "sound" at all. However, if you focus on preference for the sound of the equipment, then I think you have shifted the goalposts. But that is always the paradox of the audiophile: We come to drink the wine but end up obsessing about the glass.

As for audibility, this is something that can be very variable depending on context. My day job is in forensic audio and constantly involves trying to dig voices out of the noise floors of recordings -- to make the in-audible audible. The limitations of consumer (and professional) recording devices and codecs/formats is a constant concern for me. What may be fit-for-purpose with respect to thresholds of audibility in one context may be inadequate in an other. I admit that my situation is well outside the normal use-case for these devices but it makes me appreciate the advantages of good engineering when I see it.

 

[strangeskies]

Wait. I'm still hung-up on the analogy in the OP. What are the "lost keys"?

Sure seems like the keys are a physical manifestation of a false dilemma fallacy instead of "truth". Is there not value in objective measure of signal fidelity? AND potential value in statistically valid results from well-designed tests for "euphony" (the rigorous quantification of "what makes the most humans happy"?).

Of course we could just give the humans a wonderful suite of signal processing tools and let them (even the weird taste-outliers) dial-in their happiness.

 

[Island_Kenny]

@MikeyAs I always believed, the goal of hi-fi was the best possible reproduction of the recording, i.e. the highest fidelity.

Here lies our fundamental disagreement. To me, as stated by Toole in the youtube video posted above, the goal of Hi-Fi is to reproduce the experience of a musical performance. By looking at a small set of electronic measurements as the gold standard, the big picture is lost.

 

[beefkabob]

Here lies our fundamental disagreement. To me, as stated by Toole in the youtube video posted above, the goal of Hi-Fi is to reproduce the experience of a musical performance. By looking at a small set of electronic measurements as the gold standard, the big picture is lost.

Yeah, and well-measuring equipment does a good job of this. And poorly measuring equipment usually does not. And most people are bad at discerning what they get.

 

[amirm]

To me, as stated by Toole in the youtube video posted above, the goal of Hi-Fi is to reproduce the experience of a musical performance.

That is a goal that cannot be achieved with recorded music. All you can do is to be faithful to the recording. Dr. Toole says that right at the start:

See, it says reconstructing a captured performance. It says nothing about what came before that, i.e the musical performance as you say.

 

[beefkabob]

That is a goal that cannot be achieved with recorded music. All you can do is to be faithful to the recording. Dr. Toole says that right at the start:

See, it says reconstructing a captured performance. It says nothing about what came before that, i.e the musical performance as you say.

Whatever Toole says, this is splitting hairs. Sure, it's gonna take some animatronic animals playing real and identical instruments to recreate the exact experience of a live performance, but a recording absolutely does attempt to "reproduce the experience of a musical performance". Even if that performance is a studio album, it's still a performance. Even though playing on a stereo is not an exact replica of each sound in whatever room and whatever location the mike is in, the goal of a recording is to give people a way to experience music.

Now the goal of a component designer should be, I'd hope, to be faithful to the recording. To do otherwise is to betray the buyer.

 

[solderdude]

1. There are audible differences between (non-broken) DACs and amps, such that you can tell the difference between a Modi Multibit and a Topping DX70 by listening. If that's the case, these differences aren't found in the numbers this site is collecting and publishing, so wouldn't it make more sense to go back to blind testing to collect more data about where listener preferences genuinely lie, and then try to identify a new hypothesis about which numbers do matter for that?

The bolded parts is where grievances of subjectivists that keep popping up here after one of their favorite devices isn't recommended by Amir based on technical grounds. The immediate response is always... well the truth (what they hear) differs from the measurements ergo measurements mean nothing.
The thing is that of course people hear differences between DACs and amplifiers. That's how the brain works. The thing is... when these same people test this under blind and level matched conditions their abilities seems to have disappeared.
Those like me who actually did these kind of tests know this all too well. It is frustrating as I am human and sometimes hear (or should I say think I hear) differences which when tested with proper controls do not seem to be there.

Now one can conclude 2 things from this. The conclusion one draws depends on the individuals 'religion'.
1) Obviously subjective differences don't exist when the playing field is equal so fooled myself ... (again)
2) Blind tests are flawed because of (many reasons are brought on the table)

THIS is where the differences between the 2 camps originate. Those that actually test with proper controls all experience this. And when differences are found (which is equally possible) there are 2 possibilities yet again
1) the test was done properly and differences are there. In this case usually quite measurable differences are found.
2) the test was only thought to be done properly but wasn't. One or more controls were 'off'.

Those in the objective camp will probably look at the controls and tests and use the measurements and call it a day.
Those in the subjective camp will just say ... there I told you I heard it correctly and call it a day.

There is one of the dangers of blind tests.

2. There are no audible differences between (non-broken) DACs and amps, such that you could buy basically anything above a certain quality level, and it'd be audibly indistinguishable from anything else. In that case, these measurements should just be pass/fail, with no reason to give any more detailed breakdown (as there's no benefit to over-engineering inaudible "improvements"), and recommendations should be based on price, build quality, and ergonomics. But rather than assuming this, it seems like you'd want to prove it first, by actually doing those blind preference tests, like Toole did with speakers.

The bolded part is where the fallacy is. You see there is no clear border here.
What is a pass for person A may be a fail to person B.
So that's why there are tiers and different important aspects are measured and looked at as a whole. One can rank them on aspects like distortion (departure from accuracy), frequency response, practical noise levels.

Here is the tricky part again. Not every one perceives certain distortions the same way. Older folks don't hear 20kHz any more. Someone listening to music with insensitive headphones/speakers won't hear self-noise of amps/DACs. (They can here noise in recordings).
Folks listening with highly sensitive transducers might hear self-noise or hum where it is impossible with low efficiency gear.

So... one O.K./not O.K. value does not exist. Period.
What one can do is publish the various numbers and give some importance to it based on certainty some device is likely to perform well under all conditions right down to worst performance.

For distortion SINAD is a small indicator but not the only one as most equipment performs optimally at 1kHz and near maximum levels.
FR plots can be posted... no need to rate them.
Noise levels can also be ranked. Pointless to rank them from say max. output level. Better to do this from a reference... say 1V.
Except with most headphones one does not listen at 1V so one can rank from a more (but not perfect) reference of say 50mV.

This way, those interested in measurements can draw their own conclusions based on measurements of DACs, amplifiers made under equal (but not always realistic) circumstances. Testing all circumstances and including exhaustive listening tests and comparisons correctly would be waaaayyyy to time consuming.

Those that not care about numbers and prefer to just listen will laugh at those looking at the numbers. Reason is they don't see the value of numbers.
They rather trust an instrument that is unreliable as test device (ears+brain) and get annoyed when someone (Amir or others) post their beloved device is rated poorly. Infuriates some even and they go on about Amir (and others) salivating over numbers and the SINAD chart in particular and that listening is not included.

There you have it.. different P.O.V. and long held beliefs where one wants to cling to. On both sides.
It is very rare that a subjectivist starts testing the right way and admits the errors in his ways when he seems to have lost his abilities.
It is also very rare that someone that picked gear on numbers (just to be save) won't like what they bought.

Subjectivsist that bought something that measures well and find it not to their liking (for whatever reason) is likely to happen.
Objectivistic people that bought gear, where happy until measurements came out saying it isn't as good as they thought will most likely stop using it and replace it.
That's how the business and humans operate.

Pick your poison, think contrary now and then, be inquisitive, test for yourself but do it the right way, ask how if one doesn't know, step out of the comfort zone if one really wants to learn.
Seeking confirmation between like minded people rarely leads to better insight.

 

[Cosmik]

But in fact, Toole and Olive's research shows that time coherence doesn't matter. It doesn't predict listener preference, and the human ear appears to be totally insensitive to it....
Which again is the very core of my argument: Measurement without a theory behind it can mislead as easily as it can illuminate. If you don't have a coherent theory of which measurements matter, at what levels, and the blind test data to back that up, you can't really present a meaningful set of measurements except by coincidence. Even if you're going for "accuracy."

Listener preference is being misused here, I think. The idea behind testing for preference is not to find what the listener 'likes', but to imply that it is a proxy for 'accurate' - without going so far as to state that the listener knows what 'accurate' sounds like. It would be decidedly unscientific to state that your listeners could discern 'accurate' from recordings that are put together in a variety of ways, and who have biases - such as they know they are listening to an audio system even in a double-blind test, and may drop into 'audio system listening mode' rather than 'concert mode'. The way to bypass all these problems and more, is to imply (without stating) that preference is a proxy for 'good'.

A speaker that doesn't time-align its drivers is not a speaker that has the ambition of meeting the definition of a speaker, but is instead doing something arbitrary. Suppose all the blocks in the system were allowed to have truly arbitrary functions defined by feedback from listener preference. Which system would give maximum preference scores in a pure listening test? Answer: the one that replaced whatever was fed into it with the listener's favourite recording of music. Our real systems are not so extreme, but they are on the same continuum. Design by preference is a road to nowhere, and only by attempting to adhere to rigid definitions and specifications can a system be called a serious audio system that will reveal to all listeners things they never expected to hear.

Listeners' preferences should be catered for in the choice of recordings not arbitrary variations in 'audio systems'.

 

[Wombat]

Whatever Toole says, this is splitting hairs. Sure, it's gonna take some animatronic animals playing real and identical instruments to recreate the exact experience of a live performance, but a recording absolutely does attempt to "reproduce the experience of a musical performance". Even if that performance is a studio album, it's still a performance. Even though playing on a stereo is not an exact replica of each sound in whatever room and whatever location the mike is in, the goal of a recording is to give people a way to experience music.

Now the goal of a component designer should be, I'd hope, to be faithful to the recording. To do otherwise is to betray the buyer.

And this is done by using system components with summed distortion figures below audibility. Speakers are progressively getting better in this regard.

 

[Guermantes]

I would say the recording is fundamentally a type of fiction that may or may not be based on real events.

This is nothing unusual -- we are quite happy to fill the structure of our consciousness with fictions. I would say it is a defining human trait.

We may even call some of our fictions "truths" . . .

 

[Julf]

I would say the recording is fundamentally a type of fiction that may or may not be based on real events.

Just like, say, a painting. You still don't want to wear soft focus or rose-colored glasses to a Rembrandt exhibition.

We may even call some our fictions "truths" . . .

Some of us require a bit of objective evidence for that...

 

[Wombat]

I would say the recording is fundamentally a type of fiction that may or may not be based on real events.

This is nothing unusual -- we are quite happy to fill the structure of our consciousness with fictions. I would say it is a defining human trait.

We may even call some our fictions "truths" . . .

The deluded do fiction is truth, unfortunately.

 

[Wombat]

This is going to seem a little trolly, because it is a basically existential criticism of this site. But I'm actually sympathetic to the goals of audio objectivism, and this post arises from my frustration with how it's so often carried out. In particular, one of the most perplexing things to me about the world of audio objectivism, as embodied on this site especially, is that it doesn't seem to even be trying to achieve its stated goals.

Because the problem at hand is that sighted listening impressions are -- famously, notoriously -- unreliable. It's provably impossible to remove expectation bias and to prevent non-auditory cues from strongly coloring audible impressions. Combine that with the weakness of long-term auditory memory, and it's clear that most subjective impressions/reviews aren't worth the photons they're displayed with. We need better!

And so we know how to do better: You do blind comparisons, and try to find what product is preferred when people don't know what it is. This is how Floyd Toole and Sean Olive compared speakers at the NRCC and Harman; this is how Wine Spectator compares and scores wines. (As an aside, how embarrassing is it that wine magazines have vastly better methodological rigor than audiophile magazines!)

If you want to take it a step further from that, you can try to identify patterns based on what you find in the blind comparisons (as Toole and Olive did with the "spinorama" measurements), and then do further experiments to see how strongly correlated those measurements are with blind preference -- keeping in mind that it's the blind preference that's the ultimate arbiter, and the measurements that are the hypothesis being tested.

But that's not what this site's flavor of objectivism does. This site doesn't do any blind listening comparisons at all. It appears to just take as a given, for no obvious reason, that the set of measurements Amir performs are the set of measurements that would correlate to blind listening preference. This seems like a strange assumption, particularly because those in the subjectivist camp already know about these measurements, and believe that they are not correlated with audible performance at the levels generally measured here, that there are other factors in play that matter.

What's worse is, even if you believe that this set of measurements is what's important, it's not clear that any of the measurements (except for the most broken units) rise to any level of significance whatsoever. Take the "SINAD" tests for DACs, for instance. Amir is savage about the Schiit Modi Multibit, because it measures worse than other products. Due to a second harmonic at -80dB, and higher order harmonics and other noise at -100dB, its SINAD measurement gets a "red" rating, as obviously inferior to other, better-measuring products.

But why? The AES has done actual listening tests, and they've experimentally established limits to the audibility of THD, and they're much, much higher than we're dealing with here, particularly for 2nd order harmonic distortions. If your hypothesis is that measurements like SINAD tell the full story of listening quality, then based on what we know about perceptibility of distortion, your conclusion has to be that any product with measurements like the Modi Multibit is audibly perfect, and all this green/yellow/red stuff is marketing fluff that conveys no useful information.

So it seems to me that there are two possible scenarios in play:

1. There are audible differences between (non-broken) DACs and amps, such that you can tell the difference between a Modi Multibit and a Topping DX70 by listening. If that's the case, these differences aren't found in the numbers this site is collecting and publishing, so wouldn't it make more sense to go back to blind testing to collect more data about where listener preferences genuinely lie, and then try to identify a new hypothesis about which numbers do matter for that?

2. There are no audible differences between (non-broken) DACs and amps, such that you could buy basically anything above a certain quality level, and it'd be audibly indistinguishable from anything else. In that case, these measurements should just be pass/fail, with no reason to give any more detailed breakdown (as there's no benefit to over-engineering inaudible "improvements"), and recommendations should be based on price, build quality, and ergonomics. But rather than assuming this, it seems like you'd want to prove it first, by actually doing those blind preference tests, like Toole did with speakers.

Either way, it's hard for me to see the value in taking well-understood measurements, and then grading equipment based on how well it performs at solidly inaudible levels on those metrics. That has the appearance of science, but not the substance. It's not even rewarding good engineering, because good engineers don't gold-plate irrelevant metrics, they focus on what actually matters.

You've probably heard the old saw about the man looking for his keys in the parking lot. A good samaritan comes over to help him, and after some fruitless minutes, asks whether the man is sure that he lost the keys here. "Oh no," the man says, "I lost them in the bushes, but I'm looking here because that's where the light is."

Audio objectivists have spent too long looking where the light is; It's time to start looking in the bushes.

Try to be more objective in your views.

 

[Guermantes]

Just like, say, a painting. You still don't want to wear soft focus or rose-colored glasses to a Rembrandt exhibition



Some of us require a bit of objective evidence for that...

What!? Don't you understand yet that the phenomenological subject takes precedence over veracity? Back to Plato's Cave with you!

 

[Island_Kenny]

That is a goal that cannot be achieved with recorded music. All you can do is to be faithful to the recording. Dr. Toole says that right at the start:

View attachment 37137

See, it says reconstructing a captured performance. It says nothing about what came before that, i.e the musical performance as you say.

"For the benefit of listeners"... This is how I read it. You cannot take the listeners out of the equation

In another thread I mentioned the Steinways Spirio system which reproduce the music/performance from a totally different approach(limited and predates Thomas Edison). The recordings are the mechanic movements of the hammers, or they can recreate from sound recording back to the movement of the hammers, thus achieve a more realistic listening experience through a piano instead of loud speakers. For one thing, you can walk around the piano and or in any corner of a room, and don't have to sit in a sweet spot.

you can compare Spirio or similar systems to any electronic components record/reproduction through blind listening tests, but not by any analyzers. I never said these numbers are not important, but they should not be gold standard for the quality of "sound reproduction".

 

[pkane]

"For the benefit of listeners"... This is how I read it. You cannot take the listeners out of the equation

In another thread I mentioned the Steinways Spirio system which reproduce the music/performance from a totally different path, the recording is the mechanic movement of the hammers, or they recreate from sound recording back to the movement of the hammers, thus achieve a more realistic listening experience through a piano instead of loud speakers.

you can compare Spirio or similar system to any electronic components record/reproduction through blind listening tests, but not by any analyzers.

You can certainly compare performances with a computer. An AP Analyzer is a specialized device for certain kind of measurements. It doesn't measure performances but that doesn't mean they cannot be measured.

Do tell me, do they also have a 'Spirio' system for an orchestra? I imagine it'd look something like this for all the performers :

 

[Island_Kenny]

You can certainly compare performances with a computer. An AP Analyzer is a specialized device for certain kind of measurements. It doesn't measure performances but that doesn't mean they cannot be measured.

I am open to any measurements as long as they are established to be associated with listener's experience, even very weak associations.