• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). There are daily reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Stereophiles editor Jim Austin publicly disagreeing with Kal Rubinson

Status
Not open for further replies.

617

Major Contributor
Forum Donor
Joined
Mar 18, 2019
Messages
1,977
Likes
4,261
Location
Somerville, MA
Nope. One can structure a controlled and double-blinded test in which individual subjective listening impressions are collected, compiled and analyzed to provide statistically-supported and useful data. Not a trivial task but doable.
Has this actually been done with loudspeakers? The famous harman trials were blind, correct?
 

kemmler3D

Addicted to Fun and Learning
Joined
Aug 25, 2022
Messages
665
Likes
1,264
Location
San Francisco
Can you explain what a subjective opinion of a device that does exactly the same thing as another device would tell us?
You could use that to measure the subjective effect on perceived sound of cool-looking housings, for example. You think I'm being funny, but that would be a very worthwhile test for someone running product for an audio company.

"What finish on this housing sounds best" is not a goofy question at all if you understand and appreciate the importance of placebo effect in the audio industry!
 

SIY

Master Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
9,002
Likes
20,687
Location
Alfred, NY
Nope. One can structure a controlled and double-blinded test in which individual subjective listening impressions are collected, compiled and analyzed to provide statistically-supported and useful data. Not a trivial task but doable.
Thanks; and yes, hedonic testing is a bear.

"Subjective" is a word that's so abused that (unfortunately) one cannot assume it passes the Inigo Montoya test.
 

Kal Rubinson

Major Contributor
Industry Insider
Joined
Mar 23, 2016
Messages
4,709
Likes
8,244
Location
NYC/CT
Interesting! Are we talking about trained listeners? Or are we talking about off-the-street-college-kids-and-Joe-the-plumber type listeners?
Either. It depends on what you want to know.
And what sort of listening impressions are we considering as having value? I have always considered descriptions referenced to measurements to have some validity. But I have always considered impressions (and resulting descriptions) such as " .... "bright", "hollow", "dry", "forward", "tubby" and "loose", "incisive" and "warm" as having no value, simply because no two people define them the same way nor hear them the same way.
It may involve defined choice constraints or prior training to insure agreement on terminology. It definitely will not allow for free-form commentary. That is all part of what experimental design is about.
 

Jim Taylor

Major Contributor
Forum Donor
Joined
Oct 22, 2020
Messages
1,587
Likes
3,443
Either. It depends on what you want to know.

It may involve defined choice constraints or prior training to insure agreement on terminology. It definitely will not allow for free-form commentary. That is all part of what experimental design is about.

Subsequent posts provided me with the answers I sought. Thank you. Jim
 
Last edited:

617

Major Contributor
Forum Donor
Joined
Mar 18, 2019
Messages
1,977
Likes
4,261
Location
Somerville, MA
Interesting! Are we talking about trained listeners? Or are we talking about off-the-street-college-kids-and-Joe-the-plumber type listeners?

And what sort of listening impressions are we considering as having value? I have always considered descriptions referenced to measurements to have some validity. But I have always considered impressions (and resulting descriptions) such as " .... "bright", "hollow", "dry", "forward", "tubby" and "loose", "incisive" and "warm" as having no value, simply because no two people define them the same way nor hear them the same way.

So my curiosity is definitely piqued ;) ; what sort of subjective listening impressions support an analysis that provides statistically-supported and useful data?

Jim
Subjective trials are done all the time in the social sciences and, you know, marketing. A group of people tasting different spaghetti sauces in a mall can provide very good subjective data about preferences, and their adjectives can be useful data points. You just need a ton of data, to account for the fact that different people use the same word to mean different things.

Experimental design, and questionnaire design is really a fascinating topic, and is used all the time to measure things far more difficult to describe than what Dire Straits sounds like. Just think of those mental health screening questionnaires.
 

Jim Taylor

Major Contributor
Forum Donor
Joined
Oct 22, 2020
Messages
1,587
Likes
3,443
To those who notice that a post seems lost ..... it is.

In answer to the post by @Kal Rubinson:

" Nope. One can structure a controlled and double-blinded test in which individual subjective listening impressions are collected, compiled and analyzed to provide statistically-supported and useful data. Not a trivial task but doable."..........

..... I posted the following. Then I deleted it, thinking it was worthless and no one had noticed it .
I was wrong.... :oops: .... again. :facepalm:

For those who want to see the post, here's what I deleted, in its entirety:

Interesting! Are we talking about trained listeners? Or are we talking about off-the-street-college-kids-and-Joe-the-plumber type listeners?

And what sort of listening impressions are we considering as having value? I have always considered descriptions referenced to measurements to have some validity. But I have always considered impressions (and resulting descriptions) such as " .... "bright", "hollow", "dry", "forward", "tubby" and "loose", "incisive" and "warm" as having no value, simply because no two people define them the same way nor hear them the same way.

So my curiosity is definitely piqued ;) ; what sort of subjective listening impressions support an analysis that provides statistically-supported and useful data?

Jim

p.s. - @SIY mentioned hedonic testing, which, in my understanding, only relates to acceptance based on like/dislike, not accurate/inaccurate. Is this the analysis to which you refer?
 
Last edited:

kemmler3D

Addicted to Fun and Learning
Joined
Aug 25, 2022
Messages
665
Likes
1,264
Location
San Francisco
Subjective trials are done all the time in the social sciences and, you know, marketing. A group of people tasting different spaghetti sauces in a mall can provide very good subjective data about preferences, and their adjectives can be useful data points. You just need a ton of data, to account for the fact that different people use the same word to mean different things.

Experimental design, and questionnaire design is really a fascinating topic, and is used all the time to measure things far more difficult to describe than what Dire Straits sounds like. Just think of those mental health screening questionnaires.
Dead-on.

Also, it's instructive to change nothing and then ask people what their impressions are. I once shipped a firmware update that changed a few things, but not the sound, on some bluetooth headphones. Most users reported the sound got better anyway.

In order to make subjective impressions statistically useful, you usually have to use multiple-choice questionnaires. So the questions would probably look something like:

Speaker A:
I could clearly hear the trumpets: Strongly disagree --|-- neither agree nor disagree --|-- Strongly agree
I could clearly hear the drums: Strongly disagree --|-- neither agree nor disagree --|-- Strongly agree
I thought it was easy to understand the lyrics: Strongly disagree --|-- neither agree nor disagree --|-- Strongly agree
I heard more details in the sound: Strongly disagree --|-- neither agree nor disagree --|-- Strongly agree

Speaker B:
etc.
 

poxymoron

Senior Member
Joined
May 7, 2021
Messages
335
Likes
379
"Listening to music is believing"
Love it. JackL is a funny guy.
Reminds me of a fond subjectivist audiophile colleague of mine who always ends our Hifi discussions (debates) with "trust your ears" or "it's all about the music".
 

birdog1960

Senior Member
Joined
Oct 18, 2022
Messages
304
Likes
308
Location
Virginia

Mr. Widget

Addicted to Fun and Learning
Joined
Oct 11, 2022
Messages
575
Likes
696
Location
SF Bay Area
Dead-on.

Also, it's instructive to change nothing and then ask people what their impressions are. I once shipped a firmware update that changed a few things, but not the sound, on some bluetooth headphones. Most users reported the sound got better anyway.

In order to make subjective impressions statistically useful, you usually have to use multiple-choice questionnaires. So the questions would probably look something like:

Speaker A:
I could clearly hear the trumpets: Strongly disagree --|-- neither agree nor disagree --|-- Strongly agree
I could clearly hear the drums: Strongly disagree --|-- neither agree nor disagree --|-- Strongly agree
I thought it was easy to understand the lyrics: Strongly disagree --|-- neither agree nor disagree --|-- Strongly agree
I heard more details in the sound: Strongly disagree --|-- neither agree nor disagree --|-- Strongly agree

Speaker B:
etc.
Absolutely... clinical drug trials are always set against a harmless placebo.

We humans are amazingly sensitive to micro changes or differences, but we are also far from infallible.
 

birdog1960

Senior Member
Joined
Oct 18, 2022
Messages
304
Likes
308
Location
Virginia
To those who notice that a post seems lost ..... it is.

In answer to the post by @Kal Rubinson:

" Nope. One can structure a controlled and double-blinded test in which individual subjective listening impressions are collected, compiled and analyzed to provide statistically-supported and useful data. Not a trivial task but doable."..........

..... I posted the following. Then I deleted it, thinking it was worthless and no one had noticed it .
I was wrong.... :oops: .... again. :facepalm:

For those who want to see the post, here's what I deleted, in its entirety:

Interesting! Are we talking about trained listeners? Or are we talking about off-the-street-college-kids-and-Joe-the-plumber type listeners?

And what sort of listening impressions are we considering as having value? I have always considered descriptions referenced to measurements to have some validity. But I have always considered impressions (and resulting descriptions) such as " .... "bright", "hollow", "dry", "forward", "tubby" and "loose", "incisive" and "warm" as having no value, simply because no two people define them the same way nor hear them the same way.

So my curiosity is definitely piqued ;) ; what sort of subjective listening impressions support an analysis that provides statistically-supported and useful data?

Jim

p.s. - @SIY mentioned hedonic testing, which, in my understanding, only relates to acceptance based on like/dislike, not accurate/inaccurate. Is this the analysis to which you refer?
hedonistic testing is best. if it feels good, it is good.
 

birdog1960

Senior Member
Joined
Oct 18, 2022
Messages
304
Likes
308
Location
Virginia
You could use that to measure the subjective effect on perceived sound of cool-looking housings, for example. You think I'm being funny, but that would be a very worthwhile test for someone running product for an audio company.

"What finish on this housing sounds best" is not a goofy question at all if you understand and appreciate the importance of placebo effect in the audio industry!
Bang and olufsen anyone? But I love speakers that look like furniture. Double art. Is that wrong?
568C4759-A9FA-4247-97E6-82DEFED45C3B.png
 
Last edited:

Jim Taylor

Major Contributor
Forum Donor
Joined
Oct 22, 2020
Messages
1,587
Likes
3,443
hedonistic testing is best. if it feels good, it is good.

I agree. However, I'll stick with hedonic testing. Less trouble with the cops outside the tavern lane on Saturday night. :eek: Jim
 

dorakeg

Active Member
Joined
Jul 20, 2022
Messages
225
Likes
116
Problem is, Kal was telling people not to go out and waste their time and money on things that don't add to the musical enjoyment.
Jim Austin is saying that all you have to do is tell people that they can enjoy music without wasting money and it will cause them to not enjoy their music.:eek: And Jim Austin is right, because Stereophile made those people.

Oh ok. Thanks for pointing it out to me. I really do not see why one has to spend lots of money to enjoy music and audio quality need not be a factor. My wife is a living example.

She has her favourite singers and songs. She enjoys listening it from her phone speakers!! Eeks! For her, she is more interested in the lyrics (the meaning of it) and the way the song was sung ( tune I guess?). She is not interested in how it sound (as in audio quality). When I played it on my system, she did admit it did sound a bit better but she didn't care. She still prefers her phone.

I think i can safely say she is not alone and there are many out there like her.
 

birdog1960

Senior Member
Joined
Oct 18, 2022
Messages
304
Likes
308
Location
Virginia
Subjective trials are done all the time in the social sciences and, you know, marketing. A group of people tasting different spaghetti sauces in a mall can provide very good subjective data about preferences, and their adjectives can be useful data points. You just need a ton of data, to account for the fact that different people use the same word to mean different things.

Experimental design, and questionnaire design is really a fascinating topic, and is used all the time to measure things far more difficult to describe than what Dire Straits sounds like. Just think of those mental health screening questionnaires.
Psychiatric studies are almost all subjective. Not worthless but subjective. Hamilton depression scores are neither sensitive nor specific but CAGE screenings for alcoholism are pretty good. Go figure. 2/3 of depressed pts got better on placebo in one study. It is fascinating and you make a great point. Ok, I'll stop now.
 

preload

Major Contributor
Forum Donor
Joined
May 19, 2020
Messages
1,539
Likes
1,664
Location
California
Nope. One can structure a controlled and double-blinded test in which individual subjective listening impressions are collected, compiled and analyzed to provide statistically-supported and useful data. Not a trivial task but doable.
Exactly.
Has this actually been done with loudspeakers? The famous harman trials were blind, correct?
Yes and yes. This is how we know how well objective loudspeaker (and headphone) measurements correlate with subjective listener preferenes.
People get upset when I point out that the correlation is not perfect, and that Harman's paper could only show that objective loudspeaker measurements explained a lot (but not all) of the differences in subjective listener ratings, so I'll stop now.
 
Status
Not open for further replies.
Top Bottom