• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). Come here to have fun, be ready to be teased and not take online life too seriously. We now measure and review equipment for free! Click here for details.

Steve Guttenberg compares subjective and objective reviews

Joined
Dec 7, 2018
Messages
77
Likes
10
#61
Hi Dragon,

@Blumlein 88 and @amirm may have been a little careless with some word choices, but the thrust of what they are saying is that "relative" choices are generally preserved between groups. "Relative" means the rank is preserved, but not the absolute value or absolute difference. The P - I distinction deserves special consideration.

One can see from the graph the significant result:
P,I >* B >** M
* for 15/16 individual groups, and the combined group (p<0.0001)
** for all individual groups, and the combined group (p<0.0001)

Even though individual scores and differences between scores vary, the rank is well preserved.

The difference between P and I is less clear. Since the tabular data equivalent to the graph is not in the paper, one has to eyeball it. But it is clear that for 9/16 groups there is not a statistically significant difference. For 4/16 groups there is clearly a statistically significant difference, with 3 preferring P and 1 I. For 3/16 the significance is unclear from the graph (since I don't trust pixel measurements). When all the groups are combined the mean difference between P and I is 0.336 (from the paper), with a significance of p=0.0214.

Some will argue that is significant, but from my experience with this type of data, since the p=0.05 cutoff is arbitrary, other ways of viewing the data are relevant, and I would prefer further tests to draw a conclusion about P vs. I.

Cheers, SAM
Thx for the write up SAM.

I agree with you.
My interpretation of data presented here is as follows:
1. In absolute terms, people generally have different preference in Speaker.
2. Relative scoring of each speaker remains somewhat the same no matter which group is selected
3. All groups rated the speakers in the same order most of the time (but not always). <---this means that people generally speaking have the same preference but not always)
4. Trained listeners were more picky therefore the ratings are generally lower compared to other groups.
 
Joined
Dec 7, 2018
Messages
77
Likes
10
#62
I think what the graph shows is that all the groups showed similar preferences.

I don't have a copy of Dr Toole's book and in any case I'm not sure how much is explained by the research in it, but unlike @dragonspit4 my concern is not so much about the relative scores varying (sure, the do, but not very significantly, at least in the case of B vs M vs P/I). My concern is that the tests all took place in the same room.

I'd like to see how different rooms affected preferences, and to what extent the results hold when the room is varied.

Perhaps there is more on this in the book, or in Harman's published research?
That's interesting.
How about no room!
Test the speakers outside, would the results still be the same?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
17,436
Likes
14,921
Location
Seattle Area
#63
I don't have a copy of Dr Toole's book and in any case I'm not sure how much is explained by the research in it, but unlike @dragonspit4 my concern is not so much about the relative scores varying (sure, the do, but not very significantly, at least in the case of B vs M vs P/I). My concern is that the tests all took place in the same room.

I'd like to see how different rooms affected preferences, and to what extent the results hold when the room is varied.
Harman has three separate rooms where this type of research has been performed in. I showed the smallest one we sat in when we tested. Here is another where they test in-wall speakers and EQ products:
Harman Reference Room - small.jpg


That is Dr. Olive there and the session I mentioned where he tested us as a group. Behind the screen is a triangular section of the wall that rotates with different speakers mounted on it. You can see it in this private harman presentation:
Harman Multi-channel Room.PNG


Then there is a larger room where they test multichannel speaker systems:

Harman Shuffler Testing Room #2.PNG


Harman Shuffler Testing Room.PNG


All of this research is extensively documented across countless papers from Dr. Toole and Olive over a number of decades. I have quoted many of them in the past and can quote more. It all points to the same consistent story that when tested blind, most of us like the same type of speaker response: ones with smooth and well behaved frequency response. Make variations from this and in controlled tests, listeners don't like the sound.

I have taken the blind test twice and both times voted the same as what their research indicates.
 
Joined
Dec 7, 2018
Messages
77
Likes
10
#64
Harman has three separate rooms where this type of research has been performed in. I showed the smallest one we sat in when we tested. Here is another where they test in-wall speakers and EQ products:
View attachment 19999

That is Dr. Olive there and the session I mentioned where he tested us as a group. Behind the screen is a triangular section of the wall that rotates with different speakers mounted on it. You can see it in this private harman presentation:
View attachment 20002

Then there is a larger room where they test multichannel speaker systems:

View attachment 20000

View attachment 20001

All of this research is extensively documented across countless papers from Dr. Toole and Olive over a number of decades. I have quoted many of them in the past and can quote more. It all points to the same consistent story that when tested blind, most of us like the same type of speaker response: ones with smooth and well behaved frequency response. Make variations from this and in controlled tests, listeners don't like the sound.

I have taken the blind test twice and both times voted the same as what their research indicates.
Nice pictures!

Can we measure all the frequency response of different speakers (from same brand or different brand), and determine which one sounds the best? (determine best speaker by frequency response) and make a objective claim that one speaker is superior in sound compared the others?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
17,436
Likes
14,921
Location
Seattle Area
#65
Can we measure all the frequency response of different speakers (from same brand or different brand), and determine which one sounds the best? (determine best speaker by frequency response) and make a objective claim that one speaker is superior in sound compared the others?
You can but you need an anechoic chamber with measurements every few degrees in both horizontal and vertical direction. You then apply a special weighting based on which ones come at you directly versus reflected from horizontal and vertical directions (indirect sounds). Once there, you get extremely high correlation to listening tests. It is not perfect and listening tests needs to confirm but it is a very good predictor.

Here is a picture of the setup I took while visiting Harman in one of their anechoic chambers:

Harman Microphone Array Anechoic Chamber.jpg


That is the vertical arc with microphone array placed at the precise angles. The speaker is put on a turntable that is spun and the output of the microphones is captured. Once done, you have a sphere or response all around the speaker where you can now begin go create the composite score of all the direct and indirect sounds per above.
 

RayDunzl

Major Contributor
Central Scrutinizer
Joined
Mar 9, 2016
Messages
6,251
Likes
2,298
Location
Riverview, FL
#66
Is the floor missing something in that photo?
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
17,436
Likes
14,921
Location
Seattle Area
#67
To paint the larger picture, what you hear from a speaker is the sum total of direct sound coming at you from the speaker, and reflections from the room surfaces. Speaker response can vary from direct to reflected so any proper model of the speaker needs to include those. Using perceptual and practical modeling of rooms, Harman has created the weighting factor for those reflections by correlating the objective measurements against subjective listening tests.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
17,436
Likes
14,921
Location
Seattle Area
#68
Is the floor missing something in that photo?
Anechoic chambers don't have floors. It is usually a mesh. That room was small and I could not get far enough back to show the floor.
 
Joined
Dec 7, 2018
Messages
77
Likes
10
#69
You can but you need an anechoic chamber with measurements every few degrees in both horizontal and vertical direction. You then apply a special weighting based on which ones come at you directly versus reflected from horizontal and vertical directions (indirect sounds). Once there, you get extremely high correlation to listening tests. It is not perfect and listening tests needs to confirm but it is a very good predictor.

Here is a picture of the setup I took while visiting Harman in one of their anechoic chambers:

View attachment 20016

That is the vertical arc with microphone array placed at the precise angles. The speaker is put on a turntable that is spun and the output of the microphones is captured. Once done, you have a sphere or response all around the speaker where you can now begin go create the composite score of all the direct and indirect sounds per above.
This is pretty interesting.
How about for headphones, can the same thing be done
Can we measure all the frequency response of different headphones (from same brand or different brand), and determine which one sounds the best? (determine best headphone by frequency response) and make an objective claim that one headphone is superior in sound compared the others?
I am assuming that measuring headphone's frequency response would be easier? since there is no reverberation (echo) of sound coming from the walls?
 

Cosmik

Major Contributor
Joined
Apr 24, 2016
Messages
2,558
Likes
1,265
Location
UK
#70
...what you hear from a speaker is the sum total of direct sound coming at you from the speaker, and reflections from the room surfaces.
What reaches your ear is what you describe above. What you hear is dependent on functions of the ears and brain, and possibly what you consciously choose to focus on.
 

flipflop

Active Member
Joined
Feb 22, 2018
Messages
229
Likes
187
#71
Can we measure all the frequency response of different headphones (from same brand or different brand), and determine which one sounds the best?
Yes, to a large degree. From 'A Statistical Model That Predicts Listeners’ Preference Ratings of In-Ear Headphones: Part 2 – Development and Validation of the Model' (Sean E. Olive, Todd Welti, and Omid Khonsaripour):
The correlation between the predicted and measured preference ratings was r = 0.91 with a residual error of about 5.5% or 5.5 points on a 100-point preference scale.
I am assuming that measuring headphone's frequency response would be easier?
It poses different obstacles due to leakage effects and driver position variability.
 
Joined
Jul 21, 2018
Messages
84
Likes
174
#72

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
17,436
Likes
14,921
Location
Seattle Area
#73

vert

Member
Joined
May 30, 2018
Messages
62
Likes
28
Location
Switzerland
#74
Reading from the website of Paradigm, their research with the Canadian institute had similar results in that listeners expressed some definite preferences in speaker profiles.
 

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
17,436
Likes
14,921
Location
Seattle Area
#75
Reading from the website of Paradigm, their research with the Canadian institute had similar results in that listeners expressed some definite preferences in speaker profiles.
Paradigm is a spin off from NRC research team (of which so is Dr. Toole). PSB is another company in the same regard.
 

Arnandsway

Active Member
Joined
Sep 13, 2018
Messages
112
Likes
77
#76
Paradigm is a spin off from NRC research team (of which so is Dr. Toole). PSB is another company in the same regard.
Little bit offtopic, but I just looked up PSB and saw that they just released a new Alpha series of budget-friendly speakers. They look nice :)
 
Top Bottom