Blind listening test of speakers......what went wrong?

[Blumlein 88]

https://worldradiohistory.com/Archive-All-Audio/Stereophile/90s/Stereophile-1993-09.pdf

Starts on page 107. A bit hard to read, but I couldn't find it on the Stereophile web site.

Vandersteen 2CE were given as a reference to judge against. Arbitrarily given a score of 5. Everything else was rated less, but a fairly narrow range with only one speaker falling below a 4. Each speaker was measured as they did things in 1993. Wildly different measurements gave some strangely similar scores.

So what went wrong in this test?

 

[Audioagnostic]

There seems to be little consistency between listeners in which speaker is considered best. Table 4. That would be my take home message.
Have to admit I skipped many details about the individual speakers ...

I am not sure you should disregard this study. It seems a solid setup with multiple replications.

It bring me to a question though. The Harman preference model is taken for gospel here but I have often wondered if there has been any independent confirmation.

 

[thewas]

Yes, quite some significant variation of the listeners individual scores (am sure that Toole and Olive would comment about their training level in regards of discrimination variation and coloration detection) but in the end also there a relatively smooth measuring Harman loudspeaker got on average the highest score and the Alon with the most significant on-axis deviations the lowest.

 

[EEE272]

There are two aspects here that are also discussed in his book and some were supported by studies as well.
1) the switching of loudspeakers was manual here and our auditory memory is very short. This is the reason why HK did go through such efforts of having this very quick loudspeaker switching setup. With a break in between, it is likely that scores become more random or at least less precise.
2) the use of a pair of loudspeakers significantly reduces the possibility to hear the quality of the speaker. For surround it is even less possible. That is why HK used single speaker setups to evaluate quality.

The second point, I always wondered about because having the loudspeaker in the middle of the room will lead to different reflections, which seems to imply that we would require a different weighting than the symmetric one that is used by Olive, in case that we want to evaluate stereo quality.

 

[EEE272]

It bring me to a question though. The Harman preference model is taken for gospel here but I have often wondered if there has been any independent confirmation.

If I remember correctly, they first matched one part of their data and got something in the high 90s as correlation.
Then they used a different data set to verify (with 14 speakers?), where the correlation was high 80s.
So, the model is probably not perfect but a reasonable indicator. I still wonder how it would change in a different room or when treating certain aspects of the room. There seems to be little done in this direction.

Just as food for thought; if I would absorb (with a giant absorber) the first horizontal reflection, it would not matter how the loudspeaker emits sound in this direction.

 

[Blumlein 88]

If I remember correctly, they first matched one part of their data and got something in the high 90s as correlation.
Then they used a different data set to verify (with 14 speakers?), where the correlation was high 80s.
So, the model is probably not perfect but a reasonable indicator. I still wonder how it would change in a different room or when treating certain aspects of the room. There seems to be little done in this direction.

Just as food for thought; if I would absorb (with a giant absorber) the first horizontal reflection, it would not matter how the loudspeaker emits sound in this direction.

They got a correlation of just short of 100 % using bookshelf speakers. The next group in the mid 80% area used more or less full (fuller?) range larger speakers.

 

[Audioagnostic]

If I remember correctly, they first matched one part of their data and got something in the high 90s as correlation.
Then they used a different data set to verify (with 14 speakers?), where the correlation was high 80s.
So, the model is probably not perfect but a reasonable indicator. I still wonder how it would change in a different room or when treating certain aspects of the room. There seems to be little done in this direction.

Just as food for thought; if I would absorb (with a giant absorber) the first horizontal reflection, it would not matter how the loudspeaker emits sound in this direction.

That would not count as a true independent confirmation in science though since it is the same group reporting.

It is a pity other manufacturers do not publish their research.

 

[EEE272]

That would not count as a true independent confirmation in science though since it is the same group reporting.

It is a pity other manufacturers do not publish their research.

I totally agree.
The result close to 100% looks less impressive to me because the coefficients are defined directly based on the data.
The second part is the more interesting one in my eyes but I absolutely agree that it would be nice to see a verification study.

 

[Blumlein 88]

My opinion about why the correlation was better with bookshelves is the lack of bass. Uneven response below 200 hz is less of a factor if the speaker barely goes below 100 hz. Even good speakers in a room will have uneven bass, and they found bass contributes 30% to the subjective rating of speakers.

 

[Blumlein 88]

As for this test I'm not sure what to think. Two obvious problems are the long switching time and the fact they listened to long parts of music before switching. Both are known to drastically lower acuity.

So as a blind test I think those two factors invalidate it completely in terms of effectiveness judging differences.

 

[Andysu]

harman blind listening with blindfold that covers the ears ?

 

[Kal Rubinson]

Waste of time unless it is a joke. Even then.

 

[Andysu]

Waste of time unless it is a joke. Even then.

it is pure total bs trolling that video is lol

 

[krabapple]

Stereophile did not include the reference Vandersteens in the actual 'blind' part of the test (i.e., include it as a 'self' control). Instead it was reintroduced as a sighted reference halfway through each day's comparison of 8 (!) loudspeakers (a number that itself is nuts).

here's the reasoning (p 107):

In the past the reference was auditioned in the open at the beginning of each day to form a baseline, then auditioned in the blind later in the day. The main disadvantage of this procedure is that by the time you've listened to several loudspeakers, the sound of the reference becomes a fading memory. Ideally the reference would be auditioned before each unknown. But, due to time constraints, this is wildly impractical in a daylong test of seven or eight unknowns [....] [Auditioning them a third time in the blind was ruled out because minimizing panel fatigue was deemed more important than gathering blind data on the already thoroughly reviewed [i.e., sighted -- krab] Vandersteens.

What a waste of time. And there's no way I'm going to waste mine in close reading of eight audiophile-adjective-laden 'takes' by the likes of clowns like Harley, when it's not even established that they could rate the reference consistently, blind.

 

[Andysu]

Stereophile did not include the reference Vandersteens in the actual 'blind' part of the test (i.e., include it as a 'self' control). Instead it was reintroduced as a sighted reference halfway through each day's comparison of 8 (!) loudspeakers (a number that itself is nuts).

here's the reasoning (p 107):




What a waste of time. And there's no way I'm going to waste mine in close reading of eight audiophile-adjective-laden 'takes' by the likes of clowns like Harley, when it's not even established that they could rate the reference consistently, blind.

i said it before . each time one goes to a cinema is blind listening . most have no idea where the screen channels are placed or what make model unless they noticed the surrounds which tell them the name but i doubt even that .

 

[azzy_mazzy]

harman blind listening with blindfold that covers the ears ?

not the Scientific Audiophile finest work but the whole channel is mostly sarcasm and trolling.

 

[krabapple]

i said it before . each time one goes to a cinema is blind listening . most have no idea where the screen channels are placed or what make model unless they noticed the surrounds which tell them the name but i doubt even that .

And how has that 'observation' got sweet FA to do with anything here?