• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

[YouTube] The Big Measurement & Listening Mistake Some Hi-Fi Reviewers Make - SoundStage! Real Hi-Fi

Status
Not open for further replies.

beeface

Senior Member
Joined
Dec 5, 2017
Messages
321
Likes
773
Firstly, my apologies for yet another thread that's just about someone on YouTube espousing an opinion about a subject that's ASR-adjacent. I'm sure a lot of people are sick of these videos, but I am curious to hear what others think.

Anyway, here's the video:


tl;dw - reviewers shouldn't measure before they perform subjective evaluation because the measurements are a source of "bias".

Here's the video that preceded it in the series:


tl;dw - measurements are good: they keep manufacturers honest and anti-measurement reviewers are ignorant.

Here's my take:

Measuring a device before subjective evaluation helps the reviewer provide an informed opinion on the product.

If we assume that measurements are indeed a source of bias, then surely it's a very innocuous guise of bias. There are other external factors that influence reviewers that are in my opinion significantly worse, such as fear of pissing off an advertiser with a negative product review.
 
Last edited:

Katji

Major Contributor
Joined
Sep 26, 2017
Messages
2,990
Likes
2,273
Sorry, I don't do any of them, ever. I read, I dislike watching people talk, it's too slow, I lose focus. And just the pictures of it - him, and the cheap advert style text, and "The Big Mistake" - my immediate reaction is to think "Ahh f### off man."
He's just trying to get attention for his Youtube shit.
What's the point? ..It's like discussing that people make shit pop music for the general masses population.
 

Sombreuil

Active Member
Joined
Nov 20, 2020
Messages
236
Likes
239
Note to myself, if the words hi-fi or audiophile appear in the title of a video, watching it is a waste of time.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,388
Location
Seattle Area
tl;dw - reviewers shouldn't measure before they perform subjective evaluation because the measurements are a source of "bias".
He evaluates a speaker knowing fully the brand, color, design, manufacturer, etc. but they don't count but knowing the measurements does? If bias is such a bad thing in speaker evaluations, then he should never review a product sighted.

Somehow, measurements which provide reliable indication of sound are a bad bias, but the others are not.

Let me ask this: who do you trust the most? A reviewer who is giving you opinion of sound with no objective means? Or one that bases it on objective measurements augmented with subjective listening? Personally, I have zero use for pure subjective listening data. It is just random junk assessment that has no chance of being right.

BTW, Doug stormed out of here because he could not defend his arguments. He doesn't understand measurements that well because that is contracted out to NRC for soundstage. They are also not producing proper spinorama measurements as we are.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,388
Location
Seattle Area
One more thing: there are a lot of reviewers who cheat on this front. It is very common in headphone reviews. Reviewer says when listening, "there was too much in the 300 to 400 Hz." Oh really? You have ears that good that they are spectrum analyzer? Invariably they then let on that measurements showed the same thing. They must think we don't have brains or something. Any time specific frequencies are used in subjective listening descriptions, it is immediate sign that cheating was done in my book.
 

abdo123

Master Contributor
Forum Donor
Joined
Nov 15, 2020
Messages
7,424
Likes
7,940
Location
Brussels, Belgium
One more thing: there are a lot of reviewers who cheat on this front. It is very common in headphone reviews. Reviewer says when listening, "there was too much in the 300 to 400 Hz." Oh really? You have ears that good that they are spectrum analyzer? Invariably they then let on that measurements showed the same thing. They must think we don't have brains or something. Any time specific frequencies are used in subjective listening descriptions, it is immediate sign that cheating was done in my book.

not here to argue with your point, but I have seen several music producers/mixers (on Youtube) telling exactly what's wrong with the 'music' from just listening to it. and it often checks out with a spectrum plotter/analyzer.

it's definitely a skill a person can learn, specially if their career and livelihood depends on it. Not sure how many Youtube reviewers actually have such a skill though.
 
Last edited:

MarkS

Major Contributor
Joined
Apr 3, 2021
Messages
1,062
Likes
1,502
Didn't JA at Stereophile make it a practice to listen before measurements, back when he was doing reviews there?

To the extent that subjective judgement is useful at all, I do think it is good practice to listen and judge before measurement. Contra Amir, "knowing fully the brand, color, design, manufacturer, etc." is IMO not nearly as likely to influence what you hear as knowing that the measurements show that the speaker "should" sound bright or whatever.
 

Martin

Major Contributor
Forum Donor
Joined
Mar 23, 2018
Messages
1,896
Likes
5,536
Location
Cape Coral, FL
That’s one thing I always liked about “he who must not be named or linked”’s reviews. He listens to the speakers first before he measures them. Then he goes back and correlates his measurements with his listening notes. Seems like a smart way of doing things.

Martin
 

fieldcar

Addicted to Fun and Learning
Joined
Sep 27, 2019
Messages
821
Likes
1,258
Location
Milwaukee, Wisconsin, USA
It looks like he posted a video prior to the one listed by OP. It's a defense of objective measurements.
I think his issue is mostly with measuring before listening to gather a subjective opinion. I tend to agree with this, as I've been guilty of only judging by a graph for an IEM(I have way too many), only to find that I enjoy what I thought I would dislike. But there is a big difference between having a personal preference for tuning and going on youtube trying to convince an audience that your tastes are any sort of useful benchmark.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,388
Location
Seattle Area
To the extent that subjective judgement is useful at all, I do think it is good practice to listen and judge before measurement. Contra Amir, "knowing fully the brand, color, design, manufacturer, etc." is IMO not nearly as likely to influence what you hear as knowing that the measurements show that the speaker "should" sound bright or whatever.
What do you do then when the reviewer says it sounded fine but measurements show it was bright? What do you believe and why?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,388
Location
Seattle Area
That’s one thing I always liked about “he who must not be named or linked”’s reviews. He listens to the speakers first before he measures them. Then he goes back and correlates his measurements with his listening notes. Seems like a smart way of doing things.

Martin
Same question. What happens if said measurements don't correlate with prior listening test results? And how do you know prior test was not influenced by person not liking horns, flat panels, ported vs not, bookshelf vs stand mount, price, etc.?
 

tomtoo

Major Contributor
Joined
Nov 20, 2019
Messages
3,607
Likes
4,514
Location
Germany
What do you do then when the reviewer says it sounded fine but measurements show it was bright? What do you believe and why?

I think the revier had a bad day and believe the measurements until i listen by myself.
 

tomtoo

Major Contributor
Joined
Nov 20, 2019
Messages
3,607
Likes
4,514
Location
Germany
What do you do then when the reviewer says it sounded fine but measurements show it was bright? What do you believe and why?

I think the revier had a bad day and believe the measurements until i listen by myself.
But at the end i will always be the opinion, listen first then measure. We are humans and graphics can heavy influence what we hear.
 

Martin

Major Contributor
Forum Donor
Joined
Mar 23, 2018
Messages
1,896
Likes
5,536
Location
Cape Coral, FL
Same question. What happens if said measurements don't correlate with prior listening test results? And how do you know prior test was not influenced by person not liking horns, flat panels, ported vs not, bookshelf vs stand mount, price, etc.?

Then you find out he has hearing problems or biases he is not being honest about.

That’s one thing that always bugged the hell out of me about Stereophile reviews. The subjective reviewer praises a speaker to high Heaven only to have JA come in and show through measurements that it’s actually a hot mess. JA should call out the reviewer but he seems to demure and waffle. Reviews like that only prove to me the subjective reviewer cannot be trusted.

Martin
 

MaxBuck

Major Contributor
Forum Donor
Joined
May 22, 2021
Messages
1,515
Likes
2,117
Location
SoCal, Baby!
Same question. What happens if said measurements don't correlate with prior listening test results? And how do you know prior test was not influenced by person not liking horns, flat panels, ported vs not, bookshelf vs stand mount, price, etc.?
I react no differently from how I react to any other subjective review of speakers: "How nice." Then decide what speakers I want to listen to based primarily on measured performance.
 

sigbergaudio

Major Contributor
Audio Company
Forum Donor
Joined
Aug 21, 2020
Messages
2,639
Likes
5,397
Location
Norway
The responses in this thread surprised me. In the same way that we can be biased by the brand, the way the speakers look, what someone else has told us, etc, we can obviously also be biased by having seen measurements of said speakers.

The measurements will stay the same regardless of whether we view them before or after listening, while the listening experience may be affected by the knowledge of the measurement results. So if you want the listening session to be as unbiased as possible, you'd want as little information as possible, including about how they measure.
 
Status
Not open for further replies.
Top Bottom