Limitations of blind testing procedures

[Jakob1863]

"The second approach would rely on the fact that there exists a reality independent from the subjective impression of a human but reflected by the subjective impressions. From that point of view a listening test is obejctive if it measures the subjective response with a repeatable, reliable method.
Although getting the exact same result even in a repetition is quite unlikely, this uncertainty is just the property of the detector (listener) and will be addressed in the same way as the uncertainty from detectors in other fields."

That was the whole paragraph; i really think that it doesn´t make sense to split it apart and then commenting the snippets without proper consideration of the thoughts behind.
In this framework a listening test (well designed, well planned and well executed, which means is objective valid and reliable) is objective although the output of the detector (i.e. the listener) reflects a subjective impression.

 

[amirm]

In this framework a listening test (well designed, well planned and well executed, which means is objective valid and reliable) is objective although the output of the detector (i.e. the listener) reflects a subjective impression.

It doesn't matter how you want to think about it. No listening test regardless of rigor is called objective. Audio research papers refer to any listening test as subjective regardless of confidence in the outcome. You can draw scientific conclusions out of it. But the test itself is not referred to as "objective."

I think the fact that listening tests are so much part and parcel of "objectivity" in audio that people have equated the two. But again, in formal settings, research papers, etc. that is not done and I like us to use the correct term.

 

[Cosmik]

If a DB listening test asking "Which do you prefer: A or B?" is objective, then using it we can deduce *objectively* if one tune is better than another; one voice is better than another; one key signature is better than another?

If not, why not?

 

[amirm]

 

[Jakob1863]

It doesn't matter how you want to think about it. No listening test regardless of rigor is called objective. Audio research papers refer to any listening test as subjective regardless of confidence in the outcome. You can draw scientific conclusions out of it. But the test itself is not referred to as "objective."

That´s the opinion you offered before and the reason why i responded to your post was, as written before, that two different point of views exist. One, for example in the audiology field, that segmented in the same way that you´ve described and the other that offered a different point of view.

I thought, and at that point it might matter what i think , that readers might be interested to learn that at least some scientists have published the diverging second opinion.

I think the fact that listening tests are so much part and parcel of "objectivity" in audio that people have equated the two. But again, in formal settings, research papers, etc. that is not done and I like us to use the correct term.

I hoped that even in the short summary of the author´s point it would be clear that their line of reasoning is consistent.
But, as someone afair already mentioned, it is more a philosophical question and we will do controlled listening tests in the future regardless of whether called "subjective" or "objective".

 

[Jakob1863]

If a DB listening test asking "Which do you prefer: A or B?" is objective, then using it we can deduce *objectively* if one tune is better than another; one voice is better than another; one key signature is better than another?

If not, why not?

Usually the most important point in preference tests is to find out if a preference exists. That´s not the same as stating that one preference is "better" than the other. The preference choice might even be different in different countries or population segments.

 

[Cosmik]

Usually the most important point in preference tests is to find out if a preference exists. That´s not the same as stating that one preference is "better" than the other. The preference choice might even be different in different countries or population segments.

But if the very purpose of the 'thing' (music, voice, poem, painting) is to be liked by a human, then the preferred option is by definition 'better' than the non-preferred option. And if you are not interested in whether a human likes something, it would be stupid to ask one whether they prefer one type over another. "Do you prefer blue or green wires?" is meaningless.

But, in your world of standalone preferences, if tests did find a preference for blue or green wires (and it is quite possible), you could publish a paper on it...

 

[Jakob1863]

And now it seems that we should really begin a discussion about the meaning of words, as in the world of sensory testing there is a clear distinction between "preference" and "like" . People might prefer one option but don´t really like both (if two options are offered).

Furthermore i think to state objectively that a preference exists is different from stating that one option is objectively "better" than another. Maybe i am mistaken, it is a foreign language.....

And you did extend in your post, that i responded to, the meaning imo even further, as you included "signatures" and "voices".

 

[amirm]

That´s the opinion you offered before and the reason why i responded to your post was, as written before, that two different point of views exist. One, for example in the audiology field, that segmented in the same way that you´ve described and the other that offered a different point of view.

I thought, and at that point it might matter what i think , that readers might be interested to learn that at least some scientists have published the diverging second opinion.

They have not provided a diverging opinion. You posted about a *measurement* example, and as I repeated, that is considered objective. So there was no disagreement there.

Our discussions here about listening tests are not at all of measurement types so your comment is non sequitur regardless.

 

[amirm]

I hoped that even in the short summary of the author´s point it would be clear that their line of reasoning is consistent.
But, as someone afair already mentioned, it is more a philosophical question and we will do controlled listening tests in the future regardless of whether called "subjective" or "objective".

The point of using proper accepted terminology is that one day you may find yourself outside of these forums in real life talking about these issues with real researchers. There, you want to use the correct terminology or you will find them very puzzled at best, or very dismissive of your views if you say an audio listening test is objective.

There are countless papers in audio and they keep the domains separate on purpose and for clarity. Many papers provide listening tests and couch them as subjective data, followed by objective measurements and such.

Here is one great example from AES paper, The Subjective and Objective Evaluation of Room Correction Products

Sean E. Olive, John Jackson, Allan Devantier, David Hunt and Sean M. Hess

See the clarity this presents?

 

[Jakob1863]

They have not provided a diverging opinion. You posted about a *measurement* example, and as I repeated, that is considered objective. So there was no disagreement there.

Your recollection is inaccurate at that point.
In my post, cited below, i was writing about two different approaches; i described the first one (means the first approach) in the bold sentences and the second approach i described in the cursive part, the latter is the diverging point of view.

There are (at least) two different approaches; the first one used in audiology where a diagnostic routine is called subjective if it relies on the direct reactions from the person like audiometry. If it is independent from a (conscious) response then it is called objective, for example the measurement of otoacoustic emissions.

The second approach would rely on the fact that there exists a reality independent from the subjective impression of a human but reflected by the subjective impressions. From that point of view a listening test is obejctive if it measures the subjective response with a repeatable, reliable method.
Although getting the exact same result even in a repetition is quite unlikely, this uncertainty is just the property of the detector (listener) and will be addressed in the same way as the uncertainty from detectors in other fields.

Our discussions here about listening tests are not at all of measurement types so your comment is non sequitur regardless.

I´m quite confident, if you reread the cursive part, you´ll relialize that it is about _listening_ _tests_ ....., so no logical fallacy committed.

 

[Jakob1863]

The point of using proper accepted terminology is that one day you may find yourself outside of these forums in real life talking about these issues with real researchers. There, you want to use the correct terminology or you will find them very puzzled at best, or very dismissive of your views if you say an audio listening test is objective.

My experience with real researchers is indeed that they are often less stubborn and open for ideas of other scientists, especially if supported by a consistent argumentation.
Does it mean that the diverging opinion surely will become the prevalent opinion? Of course not.

There are countless papers in audio and they keep the domains separate on purpose and for clarity. Many papers provide listening tests and couch them as subjective data, followed by objective measurements and such.

Here is one great example from AES paper, The Subjective and Objective Evaluation of Room Correction Products

Sean E. Olive, John Jackson, Allan Devantier, David Hunt and Sean M. Hess

View attachment 7798

See the clarity this presents?

Isn´t it obvious that a lot of papers exist - and must exist - that share that point of view?
Otherwise the second approach wouldn´t be the diverging opinion .....

Wrt to your last question, would you say that "evaluation of Room Correction Products by measurements and listening tests" provides less clarity?

 

[amirm]

My experience with real researchers is indeed that they are often less stubborn and open for ideas of other scientists, especially if supported by a consistent argumentation.

What ideas? We are not discussing any ideas. We are discussing correct terminology. You are trying to rename them without a reference, clinging to fantastical notions of detection of human perception beneath what they say, with zero references to back what you say. Any science domain would be in turmoil if it allowed this kind of discourse.

In your next post, provide references in papers using your terminology as it relates to audio tests of interest to our members, and then we would have something to chew on. Until then, insisting that you are right doesn't get us anywhere.

 

[Jakob1863]

Oh, you mean you missed to ask for a reference?

Weinzierl, S. and Maempel, H.-J. (2012): "Sind Hörversuche subjektiv? Zur Objektivität akustischer Maße" Fortschritte der Akustik: Tagungsband der 38. DAGA, pp. 315-316.

And, as said before, i was reporting the diverging opinion of scientists in the audio field.
Surely you are able to accept that your assertion that it was my terminology isn´t correct?

 

[amirm]

Oh, you mean you missed to ask for a reference?

There you go trying to be cute again.

Weinzierl, S. and Maempel, H.-J. (2012): "Sind Hörversuche subjektiv? Zur Objektivität akustischer Maße" Fortschritte der Akustik: Tagungsband der 38. DAGA, pp. 315-316.

And, as said before, i was reporting the diverging opinion of scientists in the audio field.
Surely you are able to accept that your assertion that it was my terminology isn´t correct?

You would do well reading your own references before putting them forward.

Here we are discussing the usage of English terminology as it relates to describing type of research data. And you go and just put forward a German paper as your backup???

That aside, that is a 2-page opinion piece, not audio listening test results of interest to the membership I asked about.

The author's case is that since there is unreliability in objective data, perhaps they are not that reliable/objective. So what? Of course there is unreliability in objective measurements. It is up to reader/researcher to determine that degree of error before drawing conclusions.

Nothing in that paper says we should stop using those terms to refer to listening tests and measurements with the terms we have been using. Indeed, both authors are editors of proceeding of a symposium where they fully follow what I have stated:

In there there are a number of papers, and my scan of a number of the shows that they are using the term subjective to refer to listening tests, and objective to measurements.

Here is an example of many:

So once again, this topic is not up for debate. Audio research has standardized on the term "subjective" for listening tests and "objective" for measurements. You no more can try to redefine them than you would in use of English words in Law. They are what they are and are not up for mis-use as that just creates confusion.

 

[Fitzcaraldo215]

With respect to detecting audio fidelity differences, there is no method to detect that alternate reality or we would not bother with blind tests and such.

I dunno. I watch the Alternative Reality Channel on my TV all the time. It's right there along with the Alternative Facts Channel.

 

[Fitzcaraldo215]

Which measurements haven't correlated with your subjective experience?
Keith

Well, if buying an amp, will one that is flat to 100khz sound better to me than one flat to only 25k? We both know that particular spec in isolation does not matter much. But, similarly, should I pay $1,000 extra for an amp with 0.03% THD vs. one with 0.3% at maximum output. Ok, the audible sonic differences might be small to nonexistent, especially with amps these days.

I sort of know how to navigate specs and measurements, but the answers are not always clear, in my opinion. And, consider me old school, but I sure as hell would not buy an amp with good specs and measurements without first listening, any more than I would buy a car without first driving it.

 

[Purité Audio]

If it has good measurements it will sound good ,the two go hand in hand, you may prefer something with more distortion of course.
Keith

 

[Cosmik]

Please correct me if I am missing something in the following (I don't really pay much attention to the goings-on in the listening test world).

It seems to me that if I ask people for their preference on something completely meaningless, like "Do you prefer grey dust, or beige dust?", I may get a 98% preference for grey (like the question about vegetables where everyone says "carrot" or some such). If I asked people for the strength of their preference for dust, sensible people would say something like "51:49" but others would say "80%" or even "100%" (because they think being decisive makes them more impressive). In other words, on something totally pointless and meaningless, I get a strong repeatable 'preference', and even if I ask a further question regarding strength of preference the result may not even be an obviously weak preference - it could happen.

I could also ask them "Which of these two manhole covers do you prefer?". Or "Which of these two sounds do you prefer?" and play them the sound of a cat sneezing and a dog sneezing. Or I could ask them for their preference for a recorded voice played through two amplifiers, one of which is distorted slightly differently from the other.

Where I am going on this is that asking for preference only really makes sense when the choice is between two things that humans 'like'. (The dictionary definition of "prefer" is "to like one thing more than another").

Is 'an amplifier' something that a human 'likes'? I would say no. No more than an electricity substation, or a manhole cover. To ask two humans for their preference when the only difference is the amplifier makes no sense. They may still express a repeatable preference for one over the other, but this could be as meaningless as the earlier examples.

Preference, in itself, proves nothing - its only significance is the narrative that you put on it. If you are a manhole cover enthusiast with subjectivist tendencies then a strong apparent preference for a particular type is, to you, proof that regardless of the objective measurements, a type 37B is the only one that should be used. Whereas in fact, it would make no difference whatsoever to the lives of the people you stopped in the street because until you asked them about it, none of them had ever noticed a manhole cover before, and they just answered your question with an early 21st century Western human's slight bias towards a fashion for rounded corners.

 

[Thomas savage]

If it has good measurements it will sound good ,the two go hand in hand, you may prefer something with more distortion of course.
Keith

Well , if it measures well we've reproduced the signal faithfully.. after that it's up to subjective preference and not liking what you then hear is not the same as it being low fidelity.

Possibly one just does not like high fidelity sound... nothing wrong with that I guess