[YouTube] The Big Measurement & Listening Mistake Some Hi-Fi Reviewers Make - SoundStage! Real Hi-Fi

[amirm]

BTW, I think the biggest problem with subjectivist reviewers is lack of skill, not because the test is sighted. Lack of skill is clearly documented in research by Harman as I showed earlier:

If these were ratings of doctors, would you select anyone from "Audio Reviewers" category? I assume the answer is no. Why are you making a dispensation for audio?

The last time I was at Harman taking this test it was with a group of top dealers that Harman had brought in to quality for their room optimization technology. Sean Olive ran the test and this group could not get past level 2 or 3 which is what the general public does. In other words, they had no special skills, as the "Audio Retailers" bucket shows. I got up to 5 or 6 at the time (this was 10 years ago) but Sean sailed way past me. This is the power of training and knowing what you are doing. These subjectivist simply don't know what they are doing.

In contrast, sighted vs not has a soft impact on listener preference with speakers due to large difference between them. When testing their own Harman employees with implicit bias for their own brands, the difference was not that large between sighted and blind:

Whether sighted or blind, speakers G and D both got better scores than Speakers S and T. The role reversal only happened for speaker S and there, the difference was between 5.8 and 6.4. Not 1 vs 10. Speaker T maintained its score either way.

So this is not like testing electronics where frequency response is flat and only difference is distortion. There, bias can easily overwhelm the tiny differences between electronics. Here, the bias factor needs to be quite large. Having Harman's own employees in this test provided that to some extent. But nothing like what we find in sighted tests of electronics vs blind.

Net, net, I think the major issue in subjective speaker reviews is that the job is very hard and next to impossible. You are judging a speaker with no reference to what is correct. You will be relying on memory of many other speakers which is faulty especially when combined with inability to reliably detect differences in speakers.

Keep in mind that blind testing of speakers as I have explained is NOT done with one speaker. As Dr. Toole elegantly put it, it will produce nothing useful. They test 3 or more speakers against each other. Only then the results are found to be useful (post statistical analysis). You can see that in place in that graph above.

There is no research to back listening tests of one speaker by itself, blind or otherwise. You all need to internalize this as it is not often understood and attention just put on "blind vs sighted." A multi-way sighted speaker test is far more revealing in my book than any blind single speaker test.

This is such a hairy problem that one needs to give up than attempting with a straight face to say they can solve it. This is why I have resorted to EQ testing. This way I have a comparison. I can do it blind. And I can subsegment the problem and solve it. Please think through this before saying again that you want blind tests of speakers. And that testing of a single speaker by itself done by joe reviewer is good if he doesn't look at measurements. There is nothing of the sort in the cards.

 

[Sharur]

There is nothing to "imagine".
Listener preference has been correlated to the Estimated In-Room Response with 86% accuracy.
Sean Olive himself said that "listening is still necessary but measurements get you 90% of the way there".
It's all there, available to read. Nothing imaginary.

Yes, I'm aware as that is what I have been doing. However, most people don't know how to read soundstage from a graph. @HereYaGo 's research led to the discovery of Nouvraught's law. Mids from 1-3 kHz need to be lower in quantity than bass and treble if you want soundstage.

 

[amirm]

@HereYaGo 's research led to the discovery of Nouvraught's law. Mids from 1-3 kHz need to be lower in quantity than bass and treble if you want soundstage.

Good grief....

 

[Sharur]

Good grief....

Now we need a burned in DT990 with ultra low distortion

 

[amirm]

I don't think you can burn them enough to get there...

 

[Sharur]

I don't think you can burn them enough to get there...

 

[USER]

That can be done reasonably well with a little bit of knowledge.

You may not remember back when early color TV's had brightness, contrast and hue knobs right on the front panel. You would see some interesting results when someone decided to tune those up themselves.

Knowing how brightness (misnamed black level), contrast (misnamed white level) and hue worked I could fix people's crazy results and get them back to a reasonable picture. In time relatives and friends knew to call me to do that for them. I couldn't do it as well as a measured calibration, but I could get close enough many wouldn't much care about the difference using nothing other than my eyes, and how those controls functioned.

You were already steps ahead compared to many of these subjective audio reviewers. Imagine Herb Reichert understanding the fact that his listening set-up activates certain room modes and taking that into account as he compares the bass output of the speakers he reviews?

 

[Blaspheme]

You were already steps ahead compared to many of these subjective audio reviewers. Imagine Herb Reichert understanding the fact that his listening set-up activates certain room modes and taking that into account as he compares the bass output of the speakers he reviews?

You don't have to imagine it, just read Herb's SVS review:

Contributing Editor Kal Rubinson wrote in his review of the JL Audio Fathom f110v2 subwoofer. I paraphrase: Positioning main loudspeakers for imaging and tonal balance is very different than positioning stand-alone sources of low bass, which, for sophisticated music listening, should be positioned to minimize adverse interactions with low-frequency room-boundary modes. Kal's words encouraged me to try subwoofers again and motivated me to perform these careful setup experiments.

If you prefer tribal warfare, go for it. It looks like Herb is quite capable of learning from others (Amir in this thread, not so much).

 

[amirm]

You don't have to imagine it, just read Herb's SVS review:

If he is not measuring, he has no idea about room modes. You have to be hugely desperate to use what you quoted as him having any idea about this. You have to measure a room to know the room modes. You simply cannot do it with listening. Ask any high-end acoustician and they will tell you the same thing.

Here is from my friend Nyal Mallor at Acoustic Frontiers: http://www.acousticfrontiers.com/room-modes-101/

If you have a non-rectangular or odd shaped room (e.g. L-shaped or open plan) then I would recommend skipping calculators completely and going straight to measurement, or hiring a professional with the capability to do the right modeling. Predicting the room modes in such a space is a non-trivial exercise – we use boundary element modeling (BEM).

Here is Tony Grimani: https://www.sonitususa.com/post/5-steps-to-improve-the-sound-of-your-recording-studio

You wouldn’t believe how much the sound of a speaker changes based on its position in the room. Even movements of 6" can have a profound effect, especially in the bass region! Your goal should be to position the speakers to achieve smooth, neutral frequency response, so go ahead and experiment. A good spectrum analyzer with proper test signals will help you get through the process; try https://www.roomeqwizard.com/ along with a simple but effective UMM6 USB test microphone.

And since he doesn't measure or believe in them, then he has no understanding of the impact of room on his speaker listening. Many reviewers are guilty of this so @franspambot was right on the money. Your attempt at FUD, anything but.

If you prefer tribal warfare, go for it. It looks like Herb is quite capable of learning from others (Amir in this thread, not so much).

So looks like the only thing you are good for is insults.

 

[witwald]

In the same way that we can be biased by the brand, the way the speakers look, what someone else has told us, etc, we can obviously also be biased by having seen measurements of said speakers.

It seems that the word "biased" is here being used in a pejorative manner, as if all biases are bad. That seems blatantly unfair and possibly biased.

The measurements will stay the same regardless of whether we view them before or after listening,

And that's probably useful...

while the listening experience may be affected by the knowledge of the measurement results.

An alternative interpretation of measurements prior to listening sessions is that said measurements can help to technically inform those listening sessions, rather than adversely "biasing" them.

So if you want the listening session to be as unbiased as possible, you'd want as little information as possible, including about how they measure.

I'd say that if one actually wants the listening session to be as useful as possible (i.e. helpful rather than otherwise), then you'd want as much information as possible, including detailed data about how the loudspeakers measure. Unless one is possessed with "golden ears", anything less leaves the listening session open to, dare I say it, bias and/or misinterpretation.

 

[witwald]

...if we discuss reviews based on listening sessions in isolation, and how to provide such a review as unbiased as possible, that would be easier to do without knowing how the speaker measures.

Here again we are alluding to the capabilities of "golden eared" reviewers, while discounting the proven relevance and usefulness of technical measurements. Implying that listening sessions are the epitome of unbiasedness while simultaneously discounting the data provided by technical measurements seems to be a less than useful approach for loudspeaker designers, assessors, reviewers, commentators, and influencers to take.

 

[witwald]

We want the magazine's reviewers to describe what they hear, not what they think they should have heard.

But that's exactly the point, isn't it, to correctly and appropriately describe what is actually happening, across the full range of what a loudspeaker is capable of. If the measurements indicate what they should have heard, and the reviewer's didn't hear it, doesn't that indicate a significant limitation of the protocol that is being adhered to?

 

[witwald]

Why did we do all this research if all we needed to do is get a gray haired reviewer to listen to a speaker with whatever music, in whatever setting and opine about its sound???

And that word, opine, is the critical one here. Reviewers opining is not at all going to be particularly useful; reproducible facts would be preferable, wouldn't they? Reviewers opining there are aplenty, but the providers of measurements are a lot less prolific.

 

[witwald]

Stereophile's method involves two people (usually). Measure-listen makes less sense in that scenario.

It would seem that the Measure–Listen protocol makes the most sense in that two-person scenario. Both might benefit, the listener from the technical knowledge brought by the measurer, and the measurer from the skills that the listener might bring. The measurer and listener would become intertwined in a symbiotic relationship, for the benefit of their audience. That would amount to a clearly improved approach than what is currently at play.

 

[witwald]

Doctor makes diagnosis w/ verbal description and stethoscope - enhanced listening without knowing measurements

I'm not entirely convinced that the "listening without knowing measurements" actually applies here. In fact, using the stethoscope ostensibly allows doctors to perform measurements on their patient, as the doctors have been trained to know what to listen for.

 

[witwald]

I disagree with the specific position that sighted listening is entirely uninformative.

That's a very wide ranging statement, and really cannot be disproven, as the word entirely covers everything. I think that the default position has to be that we all agree with your statement!

As I mentioned, if I could easily listen to a new speaker blind, I would do so, but that isn't very practical.

Not listening to a new speaker blind can have consequences on one's assessment of that speaker. If any errors that arise in the assessments due to the methodology used are not of critical importance, and will not be able to mislead people who may rely on those assessments, then the issue of practicality can be used as a way out of performing a correctly run experiment. Such an experiment would have had to have made a concerted attempt at removing external confounding variables in order for it to meet the test of best modern practice.

Floyd's comment was "doing a single stimulus, just walk in sit down and listen to a speaker, tells you nearly nothing—you could recognise the tune, tap your foot to the rhythm ...". There's some hyperbole there, obviously.

Describing it as including hyperbole is a bit of an unfair label, as it undermines the content. It was just a (short) example provided during a conversation.

 

[sigbergaudio]

Here again we are alluding to the capabilities of "golden eared" reviewers, while discounting the proven relevance and usefulness of technical measurements. Implying that listening sessions are the epitome of unbiasedness while simultaneously discounting the data provided by technical measurements seems to be a less than useful approach for loudspeaker designers, assessors, reviewers, commentators, and influencers to take.

The discussion isn't on whether we should measure, but whether a loudspeaker reviewer should look at them before listening. I don't think anyone is arguing to discount the data provided by technical measurements.

 

[gsp1971]

But that's exactly the point, isn't it, to correctly and appropriately describe what is actually happening, across the full range of what a loudspeaker is capable of. If the measurements indicate what they should have heard, and the reviewer's didn't hear it, doesn't that indicate a significant limitation of the protocol that is being adhered to?

I agree. If the reviewer hears something way different than what the measurements show, isn't than a cause for alarm?

I have been reading online reviews from Stereophile.com (and others) for a number of years and there have been incidents where John Atkinson is puzzled by the non-correlation of his measurements to the impressions in the written review, often saying things like "... I don't understand how X (reviewer name) found the mids to be crystal clear when the measurements show otherwise. Perhaps the speaker measures worse than it sounds in real life".

Perhaps that's true. But the flip side of the coin is that perhaps the reviewer is not skilled / trained enough in listening and correctly identifying what is going on with the performance of the speaker. If we replaced reviewer X with reviewer Y, would the verdict be the same? How about a reviewer from another site or publication?

The very fact that different reviewers from different sites / publications give completely different opinions on the same product is a proof in itself that listening without measuring is an inconsistent process.

In case of two opposing reviews of the same product, which reviewer do you believe? and why?

 

[gsp1971]

I don't think anyone is arguing to discount the data provided by technical measurements.

you'd be surprised ...

 

[sigbergaudio]

you'd be surprised ...

I meant in this thread. But perhaps it's the case here as well. At least it's not my position.