• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Steve Guttenberg compares subjective and objective reviews

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
Here is one of the setups when I took the harman blind test (after the reveal/curtain opening):

harman speaker test blind.jpg


You can see the Martin Logan, B&W and JBL speaker that is frequently used in the tests.
 

RayDunzl

Grand Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
13,198
Likes
16,981
Location
Riverview FL
*waits for the "Caption the Photo" contest to start

"And this, well, nevermind..."
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
*waits for the "Caption the Photo" contest to start

"And this, well, nevermind..."

And this needs a better amplifier.

And this sounds better in Florida.

And this is so different people don't realize how good it sounds.

And this takes room correction very well.

With apologies to Ray from another electrostatic fan.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
And this needs a better amplifier.
I had a ton of subjectivists dismiss the tests on the basis of the amplifier used in Harman tests!
 

dragonspit4

Member
Joined
Dec 7, 2018
Messages
83
Likes
26
The other myth here is that we all have different preferences (hence his comment on who to listen to). That is just not so when we test speakers blind. Here again is Harman's research and preference for four speakers among different groups of listeners:

View attachment 19866

Note that the relative scoring of each speaker remains the same no matter which group is selected.

HARman trained listeners are more picky but have the same order preference that reviewers and academics had as I have highlighted.

If everyone have the same preference then the lines should be perfectly horizontal.
Just by looking at the graph, it shows that we do have individual preference.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,521
Likes
37,050
If everyone have the same preference then the lines should be perfectly horizontal.
Just by looking at the graph, it shows that we do have individual preference.
They key thing is all groups rated the speakers in the same order. Trained listeners were more picky and the ratings varied by a larger amount which is what one would expect.

So untrained, trained, musicians, salespeople, reviewers, all, all groups picked the same kind of speaker as best. This is what you would expect. They all picked the same as worst (unfortunately for Ray and myself). So everyone has the same ordered preference for the same qualities, but trained and experienced people are more picky about it and indicate larger differences in their evaluation of the sound.

All seems perfectly reasonable to me.
 

dragonspit4

Member
Joined
Dec 7, 2018
Messages
83
Likes
26
They key thing is all groups rated the speakers in the same order. Trained listeners were more picky and the ratings varied by a larger amount which is what one would expect.

So untrained, trained, musicians, salespeople, reviewers, all, all groups picked the same kind of speaker as best. This is what you would expect. They all picked the same as worst (unfortunately for Ray and myself). So everyone has the same ordered preference for the same qualities, but trained and experienced people are more picky about it and indicate larger differences in their evaluation of the sound.

All seems perfectly reasonable to me.
I understand, but my point still stand, the graph shows that we do have individual preference.
Even the order of the preference on the graphic is not absolute. If you look at the graph, it shows that for Acad10, loudspeaker I & loudspeaker P is reversed compared to everyone else. (".......So everyone has the same ordered preference for the same qualities......"<--- incorrect)
Which proves my point that we do have individual preference.
 

dragonspit4

Member
Joined
Dec 7, 2018
Messages
83
Likes
26
The other myth here is that we all have different preferences (hence his comment on who to listen to). That is just not so when we test speakers blind. Here again is Harman's research and preference for four speakers among different groups of listeners:

View attachment 19866

Note that the relative scoring of each speaker remains the same no matter which group is selected.

HARman trained listeners are more picky but have the same order preference that reviewers and academics had as I have highlighted.

Actually the relative scoring of each speaker is different. (does not remain exactly the same) I measured the relative difference using a ruler. This is the graph.
3.1cm , 1cm , 3.7cm

Harman Preferences(mod).png
 

SoundAndMotion

Active Member
Joined
Mar 23, 2016
Messages
144
Likes
111
Location
Germany
I understand, but my point still stand, the graph shows that we do have individual preference.
Even the order of the preference on the graphic is not absolute. If you look at the graph, it shows that for Acad10, loudspeaker I & loudspeaker P is reversed compared to everyone else. (".......So everyone has the same ordered preference for the same qualities......"<--- incorrect)
Which proves my point that we do have individual preference.

Actually the relative scoring of each speaker is different. (does not remain exactly the same) I measured the relative difference using a ruler.

Hi Dragon,

@Blumlein 88 and @amirm may have been a little careless with some word choices, but the thrust of what they are saying is that "relative" choices are generally preserved between groups. "Relative" means the rank is preserved, but not the absolute value or absolute difference. The P - I distinction deserves special consideration.

One can see from the graph the significant result:
P,I >* B >** M
* for 15/16 individual groups, and the combined group (p<0.0001)
** for all individual groups, and the combined group (p<0.0001)

Even though individual scores and differences between scores vary, the rank is well preserved.

The difference between P and I is less clear. Since the tabular data equivalent to the graph is not in the paper, one has to eyeball it. But it is clear that for 9/16 groups there is not a statistically significant difference. For 4/16 groups there is clearly a statistically significant difference, with 3 preferring P and 1 I. For 3/16 the significance is unclear from the graph (since I don't trust pixel measurements). When all the groups are combined the mean difference between P and I is 0.336 (from the paper), with a significance of p=0.0214.

Some will argue that is significant, but from my experience with this type of data, since the p=0.05 cutoff is arbitrary, other ways of viewing the data are relevant, and I would prefer further tests to draw a conclusion about P vs. I.

Cheers, SAM
 
Last edited:

andreasmaaan

Master Contributor
Forum Donor
Joined
Jun 19, 2018
Messages
6,652
Likes
9,399
I think what the graph shows is that all the groups showed similar preferences.

I don't have a copy of Dr Toole's book and in any case I'm not sure how much is explained by the research in it, but unlike @dragonspit4 my concern is not so much about the relative scores varying (sure, the do, but not very significantly, at least in the case of B vs M vs P/I). My concern is that the tests all took place in the same room.

I'd like to see how different rooms affected preferences, and to what extent the results hold when the room is varied.

Perhaps there is more on this in the book, or in Harman's published research?
 

dragonspit4

Member
Joined
Dec 7, 2018
Messages
83
Likes
26
Hi Dragon,

@Blumlein 88 and @amirm may have been a little careless with some word choices, but the thrust of what they are saying is that "relative" choices are generally preserved between groups. "Relative" means the rank is preserved, but not the absolute value or absolute difference. The P - I distinction deserves special consideration.

One can see from the graph the significant result:
P,I >* B >** M
* for 15/16 individual groups, and the combined group (p<0.0001)
** for all individual groups, and the combined group (p<0.0001)

Even though individual scores and differences between scores vary, the rank is well preserved.

The difference between P and I is less clear. Since the tabular data equivalent to the graph is not in the paper, one has to eyeball it. But it is clear that for 9/16 groups there is not a statistically significant difference. For 4/16 groups there is clearly a statistically significant difference, with 3 preferring P and 1 I. For 3/16 the significance is unclear from the graph (since I don't trust pixel measurements). When all the groups are combined the mean difference between P and I is 0.336 (from the paper), with a significance of p=0.0214.

Some will argue that is significant, but from my experience with this type of data, since the p=0.05 cutoff is arbitrary, other ways of viewing the data are relevant, and I would prefer further tests to draw a conclusion about P vs. I.

Cheers, SAM

Thx for the write up SAM.

I agree with you.
My interpretation of data presented here is as follows:
1. In absolute terms, people generally have different preference in Speaker.
2. Relative scoring of each speaker remains somewhat the same no matter which group is selected
3. All groups rated the speakers in the same order most of the time (but not always). <---this means that people generally speaking have the same preference but not always)
4. Trained listeners were more picky therefore the ratings are generally lower compared to other groups.
 

dragonspit4

Member
Joined
Dec 7, 2018
Messages
83
Likes
26
I think what the graph shows is that all the groups showed similar preferences.

I don't have a copy of Dr Toole's book and in any case I'm not sure how much is explained by the research in it, but unlike @dragonspit4 my concern is not so much about the relative scores varying (sure, the do, but not very significantly, at least in the case of B vs M vs P/I). My concern is that the tests all took place in the same room.

I'd like to see how different rooms affected preferences, and to what extent the results hold when the room is varied.

Perhaps there is more on this in the book, or in Harman's published research?
That's interesting.
How about no room!
Test the speakers outside, would the results still be the same?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
I don't have a copy of Dr Toole's book and in any case I'm not sure how much is explained by the research in it, but unlike @dragonspit4 my concern is not so much about the relative scores varying (sure, the do, but not very significantly, at least in the case of B vs M vs P/I). My concern is that the tests all took place in the same room.

I'd like to see how different rooms affected preferences, and to what extent the results hold when the room is varied.
Harman has three separate rooms where this type of research has been performed in. I showed the smallest one we sat in when we tested. Here is another where they test in-wall speakers and EQ products:
Harman Reference Room - small.jpg


That is Dr. Olive there and the session I mentioned where he tested us as a group. Behind the screen is a triangular section of the wall that rotates with different speakers mounted on it. You can see it in this private harman presentation:
Harman Multi-channel Room.PNG


Then there is a larger room where they test multichannel speaker systems:

Harman Shuffler Testing Room #2.PNG


Harman Shuffler Testing Room.PNG


All of this research is extensively documented across countless papers from Dr. Toole and Olive over a number of decades. I have quoted many of them in the past and can quote more. It all points to the same consistent story that when tested blind, most of us like the same type of speaker response: ones with smooth and well behaved frequency response. Make variations from this and in controlled tests, listeners don't like the sound.

I have taken the blind test twice and both times voted the same as what their research indicates.
 

dragonspit4

Member
Joined
Dec 7, 2018
Messages
83
Likes
26
Harman has three separate rooms where this type of research has been performed in. I showed the smallest one we sat in when we tested. Here is another where they test in-wall speakers and EQ products:
View attachment 19999

That is Dr. Olive there and the session I mentioned where he tested us as a group. Behind the screen is a triangular section of the wall that rotates with different speakers mounted on it. You can see it in this private harman presentation:
View attachment 20002

Then there is a larger room where they test multichannel speaker systems:

View attachment 20000

View attachment 20001

All of this research is extensively documented across countless papers from Dr. Toole and Olive over a number of decades. I have quoted many of them in the past and can quote more. It all points to the same consistent story that when tested blind, most of us like the same type of speaker response: ones with smooth and well behaved frequency response. Make variations from this and in controlled tests, listeners don't like the sound.

I have taken the blind test twice and both times voted the same as what their research indicates.
Nice pictures!

Can we measure all the frequency response of different speakers (from same brand or different brand), and determine which one sounds the best? (determine best speaker by frequency response) and make a objective claim that one speaker is superior in sound compared the others?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,384
Location
Seattle Area
Can we measure all the frequency response of different speakers (from same brand or different brand), and determine which one sounds the best? (determine best speaker by frequency response) and make a objective claim that one speaker is superior in sound compared the others?
You can but you need an anechoic chamber with measurements every few degrees in both horizontal and vertical direction. You then apply a special weighting based on which ones come at you directly versus reflected from horizontal and vertical directions (indirect sounds). Once there, you get extremely high correlation to listening tests. It is not perfect and listening tests needs to confirm but it is a very good predictor.

Here is a picture of the setup I took while visiting Harman in one of their anechoic chambers:

Harman Microphone Array Anechoic Chamber.jpg


That is the vertical arc with microphone array placed at the precise angles. The speaker is put on a turntable that is spun and the output of the microphones is captured. Once done, you have a sphere or response all around the speaker where you can now begin go create the composite score of all the direct and indirect sounds per above.
 
Top Bottom