Stereophile doubles down on the snake oil!

[Arindal]

Why after decades of this, there is not a plethora subjective advocates showing blind test result, confirming subjective impressions are at least useful? why the lack of confirmation?

Why should they? No-one has to prove any claim, that is applicable to both sides. Most of claims by objectivists also never underwent blind testing, or are results of pretty questionable or outdated test results. Blind testing as a method is simply a statistical method to measure discrimination of audible events, so to extract a useful conclusion from such a test exceeding a simple ´listeners could not distinguish differences´ or ´listeners preferred loudspeaker A subjectively over loudspeaker B´ is pretty difficult.

Blind testing might be useful in many cases and I encourage everyone to conduct such as a self-test, but it is also requires a huge investment of time and effort. Not everyone is ready to do that, and that is totally fine. Even the most objectivistic and no-frills developers in pro audio are not using blind testing on a regular base, so why should the consumer?

In the early days of high end and snake oil discussions, calls for blind testing and reliable evidence might have been useful to make some subjectivists rethink their bold claims. I do not see this anymore. When it comes to sound quality, subjective methods of assessment have anyways won and are accepted by a majority of buyers.

Even mechanical watches don't need or benefit from tourbillons any more.

I would not go into the discussion about multi-axle tourbillons or Gyrotourbillon (which can actually enable a wristwatch to show better performance), but you might have missed my point that some people do not buy these for performance but for snobbism reasons. I was not referring to the true believers, but to those who simply find it cool to have something extraordinarily expensive to show off.

But in the audio world, there are deep enthusiasts who believe they hear a difference when they install a boutique power cable. Even when they have enough technical education to know better, they believe it. They believe it because the "experts" have been saying it over and over again for decades.

I agree about true believers existing (who are not the same people with the aforementioned tourbillon snobs), but I think this explanation falls short. The number of ´experts´ claiming that power cables make a huge difference, and the number of people publicly disputing that, offering reasonable explanation why it is impossible and calling for blind tests, is pretty much of a draw game in recent 20 years. The moment you google for expensive products falling into the snake oil category, you are most likely to be confronted with both opinions, if not a majority of people disputing it.

So why do high end people nevertheless believe in such phenomena? Why do they believe subjectivistic writers and dealers making nonsensical claims about huge increase in sound quality?

My explanation would be: Because they themselves feel that they had experienced the differences numerous times, and that these claims are matching their personal experience. They might be totally deluded, but you cannot argue against personal experience just with calls for blind testing and rational arguments. Try to argue with people who believe in snake oil for health care - you cannot reach such people.

It turns out that those measurements back in the magazines of the 70's and 80's weren't bad but they weren't actually complete

Agreed, but even more, a majority of them was not relevant. What people experienced back then with their own ears, and what measurements/specs told them should sound right or wrong, was such a contradiction that they just ditched the idea of measurements being relevant. Magazines and their apodictic way of telling people what is right or wrong, were part of the problem 40 years ago, but maybe the industry and their tech-spec-marketing thing was even worse, eventually laying the ground for a totally subjectivistic approach.

To win back the trust of hi-fi people, in my understanding, we have to have an open discussion about relevance from today's point of view, with the least of dogmatism and the most of tolerance, meeting people´s expectation and subjective goals regarding sound quality with objective methods.

 

[rdenney]

...

(1) I would not go into the discussion about multi-axle tourbillons or Gyrotourbillon (which can actually enable a wristwatch to show better performance), but you might have missed my point that some people do not buy these for performance but for snobbism reasons. I was not referring to the true believers, but to those who simply find it cool to have something extraordinarily expensive to show off.



I agree about true believers existing (who are not the same people with the aforementioned tourbillon snobs), but I think this explanation falls short. The number of ´experts´ claiming that power cables make a huge difference, and the number of people publicly disputing that, offering reasonable explanation why it is impossible and calling for blind tests, is pretty much of a draw game in recent 20 years. The moment you google for expensive products falling into the snake oil category, you are most likely to be confronted with both opinions, if not a majority of people disputing it.

(2) So why do high end people nevertheless believe in such phenomena? Why do they believe subjectivistic writers and dealers making nonsensical claims about huge increase in sound quality?

(3) My explanation would be: Because they themselves feel that they had experienced the differences numerous times, and that these claims are matching their personal experience. They might be totally deluded, but you cannot argue against personal experience just with calls for blind testing and rational arguments. Try to argue with people who believe in snake oil for health care - you cannot reach such people.

(4) Agreed, but even more, a majority of them was not relevant. What people experienced back then with their own ears, and what measurements/specs told them should sound right or wrong, was such a contradiction that they just ditched the idea of measurements being relevant. Magazines and their apodictic way of telling people what is right or wrong, were part of the problem 40 years ago, but maybe the industry and their tech-spec-marketing thing was even worse, eventually laying the ground for a totally subjectivistic approach.

To win back the trust of hi-fi people, in my understanding, we have to have an open discussion about relevance from today's point of view, with the least of dogmatism and the most of tolerance, meeting people´s expectation and subjective goals regarding sound quality with objective methods.

Numbered for convenience.

1. You cannot know the motives of a person, however much you assume so. This is a legal principle. You can derive a person's intentions from their actions, but you can't know their motives. So, a person buying a tourbillon may be a "true believer" as you put it, but even that description is making assumptions favorable to your point but not to observed behavior. Rather, I would say that enthusiasts, as distinct from luxury buyers, may have different intentions. They are more likely to think of collecting as a matter of curation. But even the wealthy may buy a tourbillon just because it is cool and not because they wish to look down their noses at the less fortunate. Your characterization is uncharitable at the least, and in fact derides the credulous buyers of things like boutique power cords rather than explaining their intentions.

I completely reject the notion that buyers of four-figure power cords are relying on experiences in any sort of repeatable way. They may persuade themselves--with the help of the usual suspects--that they hear a difference, but they never validate that perception with data. The magazines could do that for them, but alas they usually do not. There is a deep divide between those who have been telling that story and those who hang out at ASR. If I said "there is no audible difference in modern DACs" or "any competent amp operated in its linear range will sound indistinguishable from any other competent amp in its linear range, if levels are matched" on even relatively balanced forums like Audiokarma, it will attract vigorous rejection. Most other audio forums are worse, and indeed ASR exists because its founder was bottled up and disinvited from the forum on which he had been active. These are enthusiasts, not merely rich collectors, and they believe they hear a difference. But when you ask for validating data, you are accused of being deaf.

2. I'm not sure they actually believe it, but they may just want to have a system that falls within the canon of approved high-end goods, as a means of ensuring they are not subjecting themselves to compromise unwittingly. Again, they do not test the outcomes, but if that's their intention and they have the money, so be it. The press should not manipulate this desire, in my view, and then call themselves journalists.

3. When someone says they hear a difference that cannot be explained by a technical understanding of the design and the psychoacoustic capabilities of listeners, my response is show me the data from properly controlled testing. In my professional world, about which I remain discreet, non-technical people at the policy level make assumptions about motives all the time, and often this leads to disappointing or even counterproductive outcomes. That's why experts in my line of work are expected to know statistics--they use statistical testing to determine how people actually react, not how policy-makers expect them to react, and not how users predict that they will react based on mere descriptions and assumptions. As with audio, everyone who merely uses the things designed by those in my line of work think of themselves as experts just because they are experienced users. Data often shows otherwise. The media almost never gets this right, but they should.

4. People in general think of engineers as hopeless geeks. Even audio enthusiasts, who are hopeless geeks already. They don't understand the technical descriptions and thus eschew them. This happens in every area of human endeavor, it seems to me. The media should provide the data that helps people past their assumptions and biases, rather than reinforcing them to support the objectives of those who cater to those biases with their marketing. Yes, there are tribal beliefs on both sides of this debate, and often people prefer to be in one tribe just because they like the people of that tribe better, not because they understand the issues at stake. Those on the "feel" side of this debate tend to be well-spoken, artistic, and literary--all traits desirable for writers in a lifestyle magazine. Those on the "data" side of this debate tend to be engineers carrying the Dilbert baggage of social ineptitude and artistic illiteracy. If I've had any success as an engineer, it's been largely based on my ability to transcend that stereotype, and yet I am bottled up by those biases in it in all aspects of my life by those around me. So, here I am, making assumptions about people based on my own experience and without real data, so I'll do what the media often does not: I'm just guessing and may have it all wrong.

Rick "data speaks with its own voice" Denney

 

[Madlop26]

Why should they? No-one has to prove any claim, that is applicable to both sides. Most of claims by objectivists also never underwent blind testing,
Even the most objectivistic and no-frills developers in pro audio are not using blind testing on a regular base, so why should the consumer?

Why?? because of burden of proof of course, objectivists are not the ones claiming hearing differences.

When it comes to sound quality, subjective methods of assessment have anyways won and are accepted by a majority of buyers.

Acceptance does not equal reality.

They might be totally deluded, but you cannot argue against personal experience just with calls for blind testing and rational arguments. Try to argue with people who believe in snake oil for health care - you cannot reach such people.

Scientific rational arguments are the best we have, could be futile for some, but not for all.

To win back the trust of hi-fi people, in my understanding, we have to have an open discussion about relevance from today's point of view, with the least of dogmatism and the most of tolerance, meeting people´s expectation and subjective goals regarding sound quality with objective methods.

Diplomacy has no role here, this is a simple scientific fact that can answered with relatively small research, but let's face it, there is no economic interests for this to happen, but the contrary.

 

[Sal1950]

Diplomacy has no role here, this is a simple scientific fact that can answered with relatively small research, but let's face it, there is no economic interests for this to happen, but the contrary.

Amen, perfect !!!

The problem is that even the experts now believe they hear stuff and are unwilling to test their ability to detect those attributes they describe using controlled testing. I'd like to see carefully controlled comparisons during the subjective review, which would apply science without making it any less subjective. Do that in addition to the objective performance measurements. But that runs the risk of bringing the whole house of cards down.

Not sure who those "experts" are but yes, that's the whole situation in a nutshell.
MHO is that those experts mostly know full well that if they brought in objective controls the vast majority of what they hear would prove to be an illusion and disappear in a puff of magic smoke.

To win back the trust of hi-fi people, in my understanding, we have to have an open discussion about relevance from today's point of view, with the least of dogmatism and the most of tolerance, meeting people´s expectation and subjective goals regarding sound quality with objective methods.

That's a very fancy way of saying what?
The High-End's whole world is built on a shaky house of cards with it's foundation being the big money made from snake-oil lies and BS products.
They'll never risk that, the only thing that could be done is to burn it down.

Thank's to folks like Amir, Delasala, Archimago, etc; we've made great inroads in showing off the emperor's new clothes.
But there are some willing to learn, and others whose brain is locked into the religion of audios magic smoke that eat that crap up.
We can only keep on offering them some Steak & Lobster system paths, then let them continue to eat the Crap if that's their choice.

"Audio as a hobby is dying, largely by its own hand. As far as the real world is concerned, high-end audio lost its credibility during the 1980s, when it flatly refused to submit to the kind of basic honesty controls (double-blind testing, for example) that had legitimized every other serious scientific endeavor since Pascal." Gordon Holt

Maybe it just deserves to roll over like a dead cockroach and die.
Who really cares any more.

 

[krabapple]

Mat, I like you, I think you are a nice guy, but still...we are in the same endless discussion.... Does sighted impressions has any relevance at all? This is still the most fundamental question in the audiophile community. To answer the question we need data, no opinions. Why after decades of this, there is not a plethora subjective advocates showing blind test result, confirming subjective impressions are at least useful? why the lack of confirmation? if you believe you loudspeakers assessment are good, why you do not take the time for blind testing, show us the data and shut up all the people that roll their eyes when confronted to any type of subjective opinion. why not become a subjective legend?

With very limited exceptions, what Joe Audiophile reports about speaker sound from his sighted audition means absolutely nothing to me. It's noise.

Same if they argue for the value -- 'real', sentimental, or just 'vibes' -- of sighted evaluations of the sound of audio gear: Noise.

It's not even entertaining , except for occasional florid examples from Stereophile et al., held up for giggles.

Be a good ASR neighbor: keep your noise to yourself.

(And strive to be less verbose/more concise generally, here.)

 

[ta240]

Mat, I like you, I think you are a nice guy, but still...we are in the same endless discussion.... Does sighted impressions has any relevance at all?

And yet I still don't see anyone questioning the reviews here that included sighted listening after measuring. Are we limited to just one person on the planet at a time that is able to do that without any bias?

 

[kyuu]

And yet I still don't see anyone questioning the reviews here that included sighted listening after measuring. Are we limited to just one person on the planet at a time that is able to do that without any bias?

There's been plenty of questioning about that since this site started, based on my rather extensive perusal of the speaker reviews during my time here. But what's your point? The subjective listening portion of the reviews is there to give context to the measurement data. It's also useful in that Amir is a trained listener and a known quantity. I don't think anyone suggests that he is without bias.

Many people completely ignore the subjective portion of the reviews. I think they are missing out on important information when they do this, as it's not always clear how the data relates to how a speaker ultimately performs even when you are proficient at reading the data (and many people have little or no idea how to read the data and would be completely lost without the subjective portion), but of course everyone can decide for themselves what to do with the information presented.

If you're trying to somehow pretend that there's little difference between Stereophile"s reviews and Amir's, you're way off base (even ignoring that we get much better measurement data than what is in Stereophile's rather antiquated presentation).

 

[Madlop26]

And yet I still don't see anyone questioning the reviews here that included sighted listening after measuring. Are we limited to just one person on the planet at a time that is able to do that without any bias?

I do not know anyone here that claims Amir subjective assessments are without any possible bias

 

[ta240]

There's been plenty of questioning about that since this site started, based on my rather extensive perusal of the speaker reviews during my time here. But what's your point? The subjective listening portion of the reviews is there to give context to the measurement data. It's also useful in that Amir is a trained listener and a known quantity. I don't think anyone suggests that he is without bias.

Many people completely ignore the subjective portion of the reviews. I think they are missing out on important information when they do this, as it's not always clear how the data relates to how a speaker ultimately performs even when you are proficient at reading the data (and many people have little or no idea how to read the data and would be completely lost without the subjective portion), but of course everyone can decide for themselves what to do with the information presented.

If you're trying to somehow pretend that there's little difference between Stereophile"s reviews and Amir's, you're way off base (even ignoring that we get much better measurement data than what is presented in Stereophile's rather antiquated presentation).

There is little, if any, difference between one part of Amir's and the Stereophile reviews. That is the only part of his reviews that I'm talking about.

It could also easily be argued that some Stereophile reviewers are a 'known quantity' and should also qualify as trained listeners.

Everything you said to defend the sighted listening test part of the reviews here could be said about the Stereophile reviews. So, why are Stereophiles reviews dismissed instead of being treated as another data point?

 

[ta240]

I do not know anyone here that claims Amir subjective assessments are without any possible bias

Yet, they are accepted without question.

If anyone else posted a comment on how any component sounded they would be immediately asked if it was a double blind test. If they replied no, it would be dismissed. And if they told us that someone told them how it would sound before they listened they would likely be mocked.

Going with the angle that nobody has said it is without bias so that means we effectively said there is bias, seems like a stretch. Especially on a site that is all about questioning everything and where listening bias is nearly always questioned whenever a subjective preference is stated.

 

[rdenney]

There is little, if any, difference between one part of Amir's and the Stereophile reviews. That is the only part of his reviews that I'm talking about.

It could also easily be argued that some Stereophile reviewers are a 'known quantity' and should also qualify as trained listeners.

Everything you said to defend the sighted listening test part of the reviews here could be said about the Stereophile reviews. So, why are Stereophiles reviews dismissed instead of being treated as another data point?

Amir's subjective reviews are about 100 words and to the point. He's an educated guy, obviously, but I don't think he'd make any claim to being literary and he's certainly not flowery.

Some of Stereophile's reviews are similarly direct, but many write English Major sentences designed to impress rather than inform.

Rick "occasionally guilty of that himself" Denney

 

[Madlop26]

There is little, if any, difference between one part of Amir's and the Stereophile reviews. That is the only part of his reviews that I'm talking about.

If the Stereophile reviewers are Trained listeners, you have a point. Research showed trained listeners are 3 to 20 times better than untrained listener for consistent discriminating assessments; otherwise, they are not at the same level.
Also, there is the financial gain factor to account for, yep, not the same level.

 

[ta240]

I still say it comes down to either all sighted, subjective reviews are useless or not. Either we dismiss them all outright or we accept them until each individual one is proved to be ineffective.

Either a salesman telling someone to listen for something specific is a valid listening experience or it isn't. I find it funny that if one person hears what a salesman tells them they are about to hear they are a fool, buying into the hype. Yet, if another person hears what a measurement tells them they are about to hear they are a brilliant listener, able to pick out the small detail.

And dismissing the validity of the results because you don't like the format it is presented in? How is it bad to present the writing in an entertaining and pleasant way to read? Not everyone is looking to read a technical journal.

There may be financial gain (selling advertisements, even though every product they review doesn't likely advertise in their magazine) on the one side but the other side has a very valid interest in preserving their argument about measurements and so there is likely a real, if unconscious, need for the listening to match the measurements.

 

[kyuu]

I still say it comes down to either all sighted, subjective reviews are useless or not.

That's throwing the baby out with the bathwater. But that's fine, you're allowed to ignore any and all subjective listening if you so choose.

 

[ta240]

That's throwing the baby out with the bathwater. But that's fine, you're allowed to ignore any and all subjective listening if you so choose.

It seems more absurd to say all of them in the entire world, except one are garbage.

I do appreciate the fact that you chose not to quote my line about "accept them until each individual one is proved to be ineffective". Just dismissing the ones that don't agree with a hypotheses does not seem very scientific.

 

[Madlop26]

Again, we do not have research data to know if sighted reviews have any usefulness, but at least we know; that trained listeners without financial gains are the ones who have the best chance to be useful.

 

[krabapple]

And yet I still don't see anyone questioning the reviews here that included sighted listening after measuring. Are we limited to just one person on the planet at a time that is able to do that without any bias?

Really? Have you looked?

 

[rdenney]

From my perspective: Let's say I have a choice between two models of speakers. Both of them show excellent published spinoramas from ASR, Erin, or elsewhere. Both are from worthy brands.

Let's say one of them received praise from a more level-headed reviewer at Stereophile, along with a good measurement report from JA, plus a range of generally favorable subjective reports in various fora. The other one has never had a review written about it in the audio press, but forum opinions are positive.

Which one might I buy? In that case, the favorable subjective review adds value and tips the scale. That value is not definitive or determinative by itself, but it's a validation of the measured performance.

As we know, with speakers, the measurements themselves are not wholly determinative like they would be with, say, a DAC. But the descriptions of the speakers by the reviewer suggested to me that he was favorable for the same reasons I might be, for the most part. He likes the same sort of music that I like, and for similar reasons, for example. Good reviewers don't tell people to buy things--they describe how they are used effectively so that the reader can determine if that fits their use cases. I would say the same thing for movie reviewers, and movies can really only be evaluated subjectively. Reviewers that try to tell me what to buy add no value to my thought processes.

The reviewer did not, however, answer a key requirement from a use case he did not need, and I had to figure that out from other sources and measurements.

That reviewer complete his review before JA measured them, but by the time I read the review I had already seen measurements. Or, maybe I dug around until I found measurements after reading the review, and had found the review on the strength of the brand's technical pedigree. Could have gone either way.

(As you have guessed, this is a true story, not just a rhetorical example. I'm hardly a beginner, but still drew value from that process.)

But if I bought a thousand-dollar power cord just because someone put it on a recommended list at S'phile, then both of us are guilty of ignorance (in my case) and questionable integrity (in the journalist's case). Inexcusable in both cases. There are lines that can be drawn.

Rick "putting his money where his mouth is" Denney

 

[Buckchester]

Why should they? No-one has to prove any claim, that is applicable to both sides. Most of claims by objectivists also never underwent blind testing, or are results of pretty questionable or outdated test results. Blind testing as a method is simply a statistical method to measure discrimination of audible events, so to extract a useful conclusion from such a test exceeding a simple ´listeners could not distinguish differences´ or ´listeners preferred loudspeaker A subjectively over loudspeaker B´ is pretty difficult.

Blind testing might be useful in many cases and I encourage everyone to conduct such as a self-test, but it is also requires a huge investment of time and effort. Not everyone is ready to do that, and that is totally fine. Even the most objectivistic and no-frills developers in pro audio are not using blind testing on a regular base, so why should the consumer?

In the early days of high end and snake oil discussions, calls for blind testing and reliable evidence might have been useful to make some subjectivists rethink their bold claims. I do not see this anymore. When it comes to sound quality, subjective methods of assessment have anyways won and are accepted by a majority of buyers.



I would not go into the discussion about multi-axle tourbillons or Gyrotourbillon (which can actually enable a wristwatch to show better performance), but you might have missed my point that some people do not buy these for performance but for snobbism reasons. I was not referring to the true believers, but to those who simply find it cool to have something extraordinarily expensive to show off.



I agree about true believers existing (who are not the same people with the aforementioned tourbillon snobs), but I think this explanation falls short. The number of ´experts´ claiming that power cables make a huge difference, and the number of people publicly disputing that, offering reasonable explanation why it is impossible and calling for blind tests, is pretty much of a draw game in recent 20 years. The moment you google for expensive products falling into the snake oil category, you are most likely to be confronted with both opinions, if not a majority of people disputing it.

So why do high end people nevertheless believe in such phenomena? Why do they believe subjectivistic writers and dealers making nonsensical claims about huge increase in sound quality?

My explanation would be: Because they themselves feel that they had experienced the differences numerous times, and that these claims are matching their personal experience. They might be totally deluded, but you cannot argue against personal experience just with calls for blind testing and rational arguments. Try to argue with people who believe in snake oil for health care - you cannot reach such people.



Agreed, but even more, a majority of them was not relevant. What people experienced back then with their own ears, and what measurements/specs told them should sound right or wrong, was such a contradiction that they just ditched the idea of measurements being relevant. Magazines and their apodictic way of telling people what is right or wrong, were part of the problem 40 years ago, but maybe the industry and their tech-spec-marketing thing was even worse, eventually laying the ground for a totally subjectivistic approach.

To win back the trust of hi-fi people, in my understanding, we have to have an open discussion about relevance from today's point of view, with the least of dogmatism and the most of tolerance, meeting people´s expectation and subjective goals regarding sound quality with objective methods.

 

[Sal1950]

I do not know anyone here that claims Amir subjective assessments are without any possible bias

Just the opposite, Amirm often states his possible bias including his business associations.