Stereophile doubles down on the snake oil!

[kemmler3D]

Yet, they are accepted without question.

Not really, not even by Amir himself sometimes. He regularly points out that he gives himself sighted bias in certain tests. I also think the way his subjective impressions are presented makes it easy to ignore them... sort of like a post-credits thing after the actual movie.

 

[Sal1950]

I still say it comes down to either all sighted, subjective reviews are useless or not. Either we dismiss them all outright or we accept them until each individual one is proved to be ineffective.

I think you miss the point that in Amirs case his short review accompany a rather lengthy measurement presentation..
Only if the subjective impressions and the measurements seem drastically at odds might a call for DBT be appropriate.

 

[JiiPee]

I think you miss the point that in Amirs case his short review accompany a rather lengthy measurement presentation..
Only if the subjective impressions and the measurements seem drastically at odds might a call for DBT be appropriate.

No. It is You who missed the point. When the sighted subjective impressions match nicely with measurements that were already known, then it is exactly the time to consider if that was due to expectation bias, and suggest DBT.

 

[Sal1950]

No. It is You who missed the point. When the sighted subjective impressions match nicely with measurements that were already known, then it is exactly the time to consider if that was due to expectation bias, and suggest DBT.

DBT against what ?

 

[JiiPee]

DBT against what ?

Whatever speaker You want to use as a reference. Usually I'd say a speaker that is considered to have a sound that You find "good". The point is to review the speaker by sound alone without any other information (like known measurement results) being there to potentially cause a biased results.

Against what did You think to perform DBT in case the sighted listening and measurements were at odds with each other?

 

[ta240]

I think you miss the point that in Amirs case his short review accompany a rather lengthy measurement presentation..
Only if the subjective impressions and the measurements seem drastically at odds might a call for DBT be appropriate.

At that point we are throwing all things science out the window and blindly accepting any data that enforces your hypothesis.

The same thing happens pretty much any time someone posts that they bought equipment that tests well and says it sounds so much better than their old equipment. The replies are vastly different than when the same thing is said about an unapproved piece of equipment.

I'm guessing Audioquest has no issues with telling someone that a set of interconnects will make the music sound sharper and then counting it as proof positive when the customer says "they did make the music sound sharper", and only resorting to a double blind test if the customer says it made the music sound smoother. This place just seemed to call for a higher standard.

 

[ta240]

DBT against what ?

What are other people supposed to DBT against when they are challenged to it here? Use that.

 

[Sal1950]

Whatever speaker You want to use as a reference. Usually I'd say a speaker that is considered to have a sound that You find "good". The point is to review the speaker by sound alone without any other information (like known measurement results) being there to potentially cause a biased results.

Against what did You think to perform DBT in case the sighted listening and measurements were at odds with each other?

It's all really too hypothetical at that point.

What are other people supposed to DBT against when they are challenged to it here? Use that.

Gear, usually they make claims of X sounding so much better than the old Y.
Or cable X sounding better/different than Y
Or power cord made X improvement over , etc etc etc.

 

[Arindal]

Maybe it just deserves to roll over like a dead cockroach and die.
Who really cares any more.

I do care.

Not for the snake oil vendors or overpriced high end products, not at all. Sadly, experience is showing that they won't die thanks to enormous margins and direct influence on their buyers. We have been at this point some 20+ years ago, when hi-fi and the idea of improving sound quality became simply unattractive to most of music lovers. For the record, back then I found rather the snake oil fraction and the arrogant nostalgia movement to blame. Nevertheless it mostly hit a lot of honest companies with affordable products like speaker manufacturers, as well as hi-fi dealerships offering reasonably priced gear. The ultra expensive stuff and snake oil survived.

And - yes - although I have no personal understanding for esoteric products and no sympathy for overpriced stuff, I increasingly feel that the groups should unite and join forces. Without people willing to spend unreasonable amounts of money on hi-fi, a lot of nice things would lose their financial base as well. So to say they are subsidizing my Qobuz account, my roon subscription, a lot of hi-fi companies being willing to invest in software and wonderful streaming engines, a zillion of musicians, record labels and a parts of the music distribution chain.

I would feel sad if all that would disappear.

But there are some willing to learn, and others whose brain is locked into the religion of audios magic smoke that eat that crap up.

As mentioned, we had been at this point 20 years ago. History was showing that the ones ´locked into the religion´, as you put it, are still there and believing in or earning from snake oil. The people who have walked away from hi-fi in masses, were the young, reasonable folks being willing to learn. And they have learned that hi-fi as a whole made the impression of a nostalgic, esoteric circus producing nothing but flamewars and dogmatic beliefs on either side, so they were sticking with their iPods, smartphones, earbuds and $79 bluetooth cans to listen to music. A whole generation is lost for hi-fi, and the second is about to follow.

Again, my point is not to promote or tolerate snake oil. My point is to make audio objectivism attractive to normal people, educate them and enable them to achieve their goals with technology that is really working. And for this you IMHO have to listen to them and sometimes ask yourself why so many stick to non-factual beliefs and snake oil.

 

[JiiPee]

Since this thread is at least supposed to be about Stereophile, I summarize my feelings in three points:
1. I condemn vehemently the snake oil salesmanship prevalent in Stereophile.
2. Outside the pure snake oil category, I find Stereophile subjective reviews mostly silly fictional BS when it comes to sound quality, and they should be clearly labelled as such, but if someone finds that kind of stuff entertaining, who am I to deny it from them...
3. Stereophile Lab reports bring actual value to people interested in this hobby. I have no reason to suspect the quality of execution, or the honesty of reporting of the measurements. This is why I still occasionally visit their web site.

 

[Arindal]

They may persuade themselves--with the help of the usual suspects--that they hear a difference, but they never validate that perception with data.

Surely they don't. For them, it is a hobby, and the subjective feeling that they managed to influence something to the positive, is enough of encouragement to them. Even if it is a complete delusion.

Most other audio forums are worse, and indeed ASR exists because its founder was bottled up and disinvited from the forum on which he had been active.

I somehow sensed there was some Sophoklean tragedy full of offense, rejection and despair behind it ;-)

And yes, being accused of deafness by esoteric believers is something I have experienced as well. I pretty well understand the motivation and support the general mission of ASR. I just do not think that methods of spreading the message chosen, are always in favor of attracting the indecisive crowd of music lovers in between. Rather the opposite, the more dogmatic and intolerant the whole thing appears to external readers, the more likely people will simply walk away, according to my experience.

The other problem with that approach is the friend-or-foe-recognition. If every honest speaker manufacturer is labelled ´snake oil vendor´ for the simple reason of not backing up every single design decision by double blind tests and scientific studies, well, the list of friends of this type of audio objectivism will be inherently limited.

my response is show me the data from properly controlled testing.

You can do that, but it would not really support your point in the eyes of the indecisive third party or make believers rethink their claims. Most of things in audio R&D happen without controlled testing for practical reasons, even the most rational design decisions undoubtedly based on science and measurements.

Calling for blind tests or evidence I found to be surprisingly ineffective. What is more effective in my understanding, is to demo listeners what science and solid engineering can actually do to achieve their personal goals of sound quality. Use DSP, latest speaker technology and science to create irresistible bass that would make people dance!

4. .. The media should provide the data that helps people past their assumptions and biases, rather than reinforcing them to support the objectives of those who cater to those biases with their marketing. Yes, there are tribal beliefs on both sides of this debate, and often people prefer to be in one tribe just because they like the people of that tribe better, not because they understand the issues at stake. Those on the "feel" side of this debate tend to be well-spoken, artistic, and literary--all traits desirable for writers in a lifestyle magazine. Those on the "data" side of this debate tend to be engineers carrying the Dilbert baggage of social ineptitude and artistic illiteracy.

I wholeheartedly agree.

But it is not only the Dilbert stereotype, the point of objectivism gets even more unattractive if paired with dogmatism and the attitude of treating everyone outside the own tribe like a moron.

 

[Mart68]

Again, we do not have research data to know if sighted reviews have any usefulness, but at least we know; that trained listeners without financial gains are the ones who have the best chance to be useful.

We do know that some people changed their preference between blind and sighted listening.

That is to say, the speaker they preferred in sighted listening was not the speaker they preferred when listening blind.

Personally I think sighted reviews of speakers can have some value, but this is diluted further when the review is in a publication funded by advertising. In that situation, even if the reviewer thinks the speaker is a piece of junk he has to be circumspect in what he says.

I recall a speaker review from one magazine that several people brought to my attention suggesting the speaker was worth an audition as the review was so positive. It was a big horn loaded speaker along the lines of Klipsch. The reviewer stated he had spend 6 weeks trying the speaker in different positions in his room, attempting to get it to sound right. A big red flag they had all missed.

 

[rdenney]

What are other people supposed to DBT against when they are challenged to it here? Use that.

We should remember that blind testing is only called for when there is a difference that is not readily explained (and therefore corrected for) from actual measurements. There is nothing to test if no difference can be heard. I think it was J_J that reminded us of that.

If Amir adjusts the equalization such that speakers under test sound like his reference speakers, then there is no longer anything to test. Claims of unexplained effects are what demand testing or further measurement. A typical audio press reviewer might say something like, "but the Magitronic speakers, as one would expect from the higher price, demonstrated a spaciousness that revealed the inner detail of the viola section, as compared to the Rickophone speakers, that allowed me to clearly hear that they were on the third row of the orchestra, behind the second violins, which I know is detectable on this recording." Amir might say, "I adjusted the 8 KHz dip in the response upward using DSP, and then the speakers sounded just like I expected them to." In the former case, the reviewer claims an attribute that may well be imaginary and a comparison that may well be faulty for a variety of reasons. Validating that unexplained effect and distinctiveness demands some form of controlled testing, or at least deeper measurements that reveal an explanation. But my fictional Amir's claim is that once a small correction was made in the response--an adjustment suggested by the measurements--there was no difference. There is nothing to test there.

My fictional Amir might say, "I attempted to correct the on-axis response using DSP, but still the speakers sounded weak in the treble, which I attribute to the beaming that starved the room reflections of those frequencies." (I can't actually hear that sentence in Amir's voice, but go with me.) In this case, his perception is wholly supported by the measurements, and we can conclude that without his subjective observation. Again, there is no claim that requires proof.

The point of controlled testing is to validate unexplainable differences. Differences that are explainable as a result of technical measurements don't need those proofs.

I can think of at least one review from Amir (and I only read the ones that review products of interest to me) where his subjective responses were at odds with the measurements, and that caused him to double down on the measurements again to try to explain the differences, rather than giving the perception equal weight and letting the reader sort it out as would usually be the case in the audio press these days. I even recall him joking about it, with the conclusion that he could not let his perceptions stand without explanation.

Rick "required proofs demand on the claims made" Denney

 

[rdenney]

...
And yes, being accused of deafness by esoteric believers is something I have experienced as well. I pretty well understand the motivation and support the general mission of ASR. I just do not think that methods of spreading the message chosen, are always in favor of attracting the indecisive crowd of music lovers in between. Rather the opposite, the more dogmatic and intolerant the whole thing appears to external readers, the more likely people will simply walk away, according to my experience.
...

The parts I didn't quote I think overstate the case. Our most aggressive anti-subjectivists at ASR pale in comparison to the derision heaped upon ASR, Amir, and the members therein in forums like Audiogon or Head-Fi (or Hoffman or too often even Audiokarma...). Those are just the ones I have occasionally participated in. I used to participate quite a lot at Audiokarma, but dropped off because I was constantly biting my tongue to avoid bitter reactions from those whose belief systems were rooted in myth and lore.

And ASR's job is not necessarily to persuade the unpersuadable. Though the attribution to Mark Twain is unsupported, someone still observed at some point that it is easier to fool people than to convince people that they have been fooled. This aphorism fits my experience completely. Anybody in any field who has to present data-driven results to people untrained in and uncommitted to those methodologies will eventually run into this effect. It takes a powerful rational will to allow data to overcome the biases of myth and lore.

I think of ASR as a place were those who do come to that insight will have a compatible place to go, and a lot of the tribal reaction is just self-congratulation. For me, ASR is a safe space where these topics can (usually at least) be explored without those of us who prefer to be driven by data spending 90% of our time with our backs against the wall. I've never been called "stupid" here the way I was too often called "stupid" on AK.

Back to Stereophile. The hints I have read--and we only get hints--have suggested that there have been significant editorial controversies over these topics there. I am somewhere between Matt Hooper and those who believe that everyone at Stereophile are brigands and liars. I avoid the reviewers who make me roll my eyes (frankly, most of them) but read those whose reviews make sense even though they are unsupported by evidence. Example: When Kal reviews a Buckeye amp that turns out to be faulty as sounding superb, and then John's measurements reveal a fault, there are two possibilities. One is that Kal's bias in favor of the Buckeye caused him to overlook the fault, even if unwittingly so. The other is that the fault, though plain to see in the measurements, did not have any audible effect that could be perceived. Kal doesn't have to prove that he didn't perceive something--he only has to prove that he did, if it can't be properly explained. Because of my experience reading Kal's reviews, I am persuaded that the effect that John measured was simply inaudible, and I think I said as much during that discussion. But I'm sure there are the cynical among the readers that assume that Kal is a shill for Class D amps and would cover up any fault. I don't want to be that cynical myself, though I will be with reviewers who have earned that cynicism (which Kal definitely has not). The problem that John revealed (and we do have to remember that John did reveal the fault) had the salutary effects of 1.) causing significant introspection on the part of the manufacturer, which revealed a production issue that was easily corrected, and 2.) emphasized to us that the specs we talk about are excessive to our needs, and we talk about them because of the good engineering they reveal that we can admire. Both outcomes are valuable, and getting both of those outcomes required measurements and the subjective review.

(In no way, however, I cannot excuse the recommendation of boutique power cords and other snake oil products, about which uncontrolled sonic claims are made that have no rational explanation from engineering or measurement.)

Rick "balanced" Denney

Edit: Did I actually write "can" instead of "cannot" in the last paragraph? Oh, my.

 

[BeeKay]

Karl Valentin (1882-1948)

Everything has already been said, just not by everyone yet.

 

[kemmler3D]

the point of objectivism gets even more unattractive if paired with dogmatism and the attitude of treating everyone outside the own tribe like a moron.

Agree, but the fix for this is not to soften the stance and allow sighted bias a longer leash here, it's to meet people where they are and help them understand what's going on, rather than assault newbies with demands for DBT and such. This is why I've long advocated for a FAQ so we can help people up the learning curve without writing a whole textbook in every thread.

 

[John Atkinson]

Well, if it is not about ranking things, then how come there are published classes of recommended components:
"Class A Best attainable sound for a component of its kind, almost without practical considerations; the least musical compromise.
Class B The next best thing to the very best sound reproduction...
Class C Somewhat lower-fi sound...
Class D Satisfying musical sound..."

And if so, where is Class E, F, etc. of components that are not recommended, which would complete the normal distribution? Nowhere to be found, because this would (a) decrease the flow of new products from manufacturers for future reviews, and (b) increase the threat of legal actions that are costly to address and result in higher business liability insurance premiums.

My apologies for not responding earlier to your posting. It is a not-uncommon mistake to assume that the performance of the products Stereophile reviews follows a Normal distribution.

As I wrote in an essay published in 1996 and have repeated since then, “Even though Stereophile publishes more reviews of more components than any other US audio magazine, we can do no more than scratch the surface of what is available. The essential mechanism for winnowing down the list of potential review candidates is the enthusiasm of our writers. They seek out what excites them, the products that get their creative juices flowing. This automatically means that the products that get ink spilled on them tend to be the ones that hold the most promise of sounding good. Not surprisingly, most of them are recommended—though not always without reservation, as you can read.”

See https://www.stereophile.com/asweseeit/366

This essay also discusses the lack of correlation between product reviews and whether or not the manufacturer advertises in Stereophile.

John Atkinson
Technical Editor, Stereophile

 

[teched58]

<snip>

This essay also discusses the lack of correlation between product reviews and whether or not the manufacturer advertises in Stereophile.

John Atkinson
Technical Editor, Stereophile

I respectfully posit that the lack of correlation is not from lack of trying.

 

[kyuu]

The essential mechanism for winnowing down the list of potential review candidates is the enthusiasm of our writers. They seek out what excites them, the products that get their creative juices flowing. This automatically means that the products that get ink spilled on them tend to be the ones that hold the most promise of sounding good. Not surprisingly, most of them are recommended—though not always without reservation, as you can read.

I respectfully posit that the notion that there would be some kind of inherent correlation between what "excites" hi-fi reviewers and good performance is... dubious, at best. Even sans any explicit payola, it might imply that these writers would be biased toward hearing a positive difference even when we know the product they are reviewing is incapable of making any actual difference to the sound, though. Nevermind those where the measurements reveal that the product is actually audibly defective, yet it receives a glowing review anyway.

 

[Sal1950]

Our most aggressive anti-subjectivists at ASR pale in comparison to the derision heaped upon ASR, Amir, and the members therein in forums like Audiogon or Head-Fi (or Hoffman or too often even Audiokarma...). Those are just the ones I have occasionally participated in. I used to participate quite a lot at Audiokarma, but dropped off because I was constantly biting my tongue to avoid bitter reactions from those whose belief systems were rooted in myth and lore.

That's putting it mildly Rick.

And ASR's job is not necessarily to persuade the unpersuadable. Though the attribution to Mark Twain is unsupported, someone still observed at some point that it is easier to fool people than to convince people that they have been fooled. This aphorism fits my experience completely. Anybody in any field who has to present data-driven results to people untrained in and uncommitted to those methodologies will eventually run into this effect. It takes a powerful rational will to allow data to overcome the biases of myth and lore.

That's a good one. and then there's
You can lead a horse to water but you can't make him drink. LOL

As I wrote in an essay published in 1996 and have repeated since then, “Even though Stereophile publishes more reviews of more components than any other US audio magazine, we can do no more than scratch the surface of what is available. The essential mechanism for winnowing down the list of potential review candidates is the enthusiasm of our writers. They seek out what excites them, the products that get their creative juices flowing. This automatically means that the products that get ink spilled on them tend to be the ones that hold the most promise of sounding good. Not surprisingly, most of them are recommended—though not always without reservation, as you can read.”

That's a good and understandable position John.
What I find unfortunate is the fact that there's little left of the affordable stuff getting reviewed.
It's mainly about the "luxury goods" market and the bragging rights to things like $5 and 6 digit amps the best of which
really sound no better or different than the Buckeye that Kal reviewed a few months back.