Can You Trust Your Ears? By Tom Nousaine

[DonH56]

"People hear what they see." - Doris Day

"The reason most women would rather have beauty than brains is because most men can see better than they can think." - Dr. Joyce Brothers (I think, paraphrased)

 

[Blumlein 88]

Er... what?

I had exactly the same thought. What? I read the sentence 4 times and then decided it actually didn't say anything.

 

[Blumlein 88]

The "totally junk" term implies that nobody is able to learn to minimize the impact of bias mechanism (up to a certain degree) even if aware of these bias mechanism.

BINGO! You were onto something there for a second.

 

[Blumlein 88]

P.S. If we want to take everyday life into consideration (as in forum discussions) we have to notice that "sighted" listening is often welcomed if not applauded, provided the results happen to fit the various beliefs of nonaudibility (in contrary to all "totally junk" ascriptions) .....

I guess it depends upon where and to whom you are talking. I have said about such sighted non-results they aren't surprising while also pointing out the procedure was just as flawed for reaching that conclusion as sighted listening is when differences are heard. Sighted listening is simply a terribly poor methodology for which it is used.

A more interesting area to me is recording, mixing and mastering. Those fellows suffer all the same issues with sighted listening to an extreme level and yet make their living making adjustments. Yet nearly all will report having made process changes, listened and heard the result before noticing they forgot to engage the change so nothing really was different.

 

[fas42]

Here is the deal and there is no getting around it: we can make arguments as you are making in favor of validity of sighted tests. The problem is that what I find in reality is that audiophiles as a group are horrible in hearing distortions that they talk about all the time. That is, they can't even hear the distortions when they are objectively there. Vast majority for example will fail tests of lossy compression against the original.

Of course this will fail - if one is trying to compare apples, with mushrooms. Both are a form of food - and. system playback is incorrect with a multitude of forms of distortion. If a person is not sensitive to distortion type F, it's not very good science, extrapolating to say that this individual can't register any of the distortion types A to Z.

Once you become aware of a particular distortion, work to eliminate that anomaly. Once achieved, it's highly likely you will become aware of more anomalies, that were, yes, masked up to that time - this becomes an iterative process, and each step on the path relies on the ears telling you "something is wrong!".

Any approach which relies on swapping between two alternatives, and ticking which is 'better' is doomed to fail, IMO - all that matters is whether there is a perceptible flaw in the sound; and here, stress testing by using 'difficult' recordings becomes the powerful tool of choice.

 

[SoundAndMotion]

If you ask someone for their perception of music in a listening test, which of those two are you measuring?

Due to the lack of specificity, that can’t be answered.
I’m not criticizing you or requesting specifics. I’m just pointing out that more info is needed about the setup, and there must be more than the question: “what is your perception of the music?”. Those details will answer the question.

 

[SoundAndMotion]

Taking a picture is about creating art. The job of audio reproduction is to preserve art. They are different things.

Amir, I wasn’t asking for a technical or philosophical comparison of photography and audio playback. Taking a picture can also be used to measure or to document in the lab. The point is: when choosing parameters for a measurement, whether it’s shutter speed, inter-interval delay or plate gap voltage, you must know what and how you are measuring, and a pat, cookbook answer won’t always work. The answer is: it depends. Giving an example, or many examples, of when short times are needed does not show it is always the case. It is not.

The problem is that doing it sighted will magnify those problems a million times.

IMHO, hyperbole here weakens your point. You certainly can’t document a 10^6-fold difference.
But as the Harmon study you do mention shows: if you are able to fix all other variables, removing knowledge (“blinding”) will improve the results over having knowledge (“sighted”).

It is worth remembering here that “blind” and “sighted”, in this context, refers to knowledge about the DUT, not about the sense of vision, i.e. using the eyes. Many people say “I know that”, but don’t seem to, when they incorrectly use the McGurk effect as an example of why “sighted” testing suffers from the biases of DUT knowledge.

 

[Cosmik]

Due to the lack of specificity, that can’t be answered.
I’m not criticizing you or requesting specifics. I’m just pointing out that more info is needed about the setup, and there must be more than the question: “what is your perception of the music?”. Those details will answer the question.

The point about music is that it is 'art', and not only does a human's hearing respond to it, but also their mind at many different levels. You are proposing to test their hearing by asking them, effectively, how they feel in their mind about some art. If you were recommending that the stimulus was bleeps and clicks, I might be more persuaded that this was science. If the idea is "Well the system is going to be used for reproducing music, so we need to know how people respond to it with music", I would say that that was fine, but it isn't science.

 

[Jakob1863]

We don't operate from the point of view of zero knowledge in audio or any other sciences. What you say we don't know is the consensus point of view of literally thousands of engineers and researchers in this field. This is not an unknown field where all outcomes have reasonable chances of being true. There are immutable laws and rules that govern audio and it is not reasonable to throw them out in evaluating likeliness of outcomes.

We both know that literally thousands of other engineers think differently about at least a couple of items from your list.
Maybe we should start another thread to revive the discussion about Popper´s falsification theory, enhanecments of it or about philosophy of science, as it might be to much for this thread?

An impact on what?

An impact on the internal criteria of listeners participating in a controlled listening test. As stated earlier, people were doing a lot of different experiment within the Signal Detection Theory (SDT) in the 1950-1960s and compared the results of trials with and without a bonus.

I put money forward because I know the likelihood of losing it is zero.

Yeah, but is it a good idea?
Again a matter of bias and you have to think that this sort of bias will not have an influence, or as it seems to be, you simply don´t care which is a matter of objectivity. As someone else said before in this thread (cosmik?) if you do experiments mainly to confirm your belief risk to do something wrong is much higher.

I know that based on full consensus of the entire engineering and scientific community of audio. Per my other post, we don't get to throw that out in assessing someone's notions of improving audio. We don't live in a bubble in audio where no rules would apply to it.

I wouldn´t do that (means throwing out) either, but have seen and read in the recent ~35 years a lot from the "engineering community" that does not really reflect proper scientific reasoning. I´d blame the fact that attending courses about philosophy of science isn´t mandatory (at least it wasn´t in Germany and i got the impression that it isn´t in other countries as well).

You mean in sighted testing in high-end audio as performed by audiophiles? Where is the data to back that to borrow your line?

There seems to be an obsession with audiophiles.
Couldn´t we leave that aside? I could rant about pseudoobjectionists doing (and demanding) DBTs without knowing even about the most basic things in test theory, but that will not help either.

A proper definition of the term "audiophile" is missing. In the 1930s - 1950s it meant afaik people not so much interested in music but in an exaggerated sound (means more than in reality).
From a more neutral point of view, it would describe people who are concerned about the quality of sound, which includes musicians, recordings engineers, developers of audio gear and consumers as well.

 

[Jakob1863]

@ Don Hills,

Er... what?

Who could resist to respond to such eloquent asking.......

The term "quantitative" describes a class of methods/tests that uses standardized procedures to get numerical results on single features (or a small number of features) that can be analyzed with the usual statistics. To this class belong the tests routinely used like ABX, AB, triangle, sort the odd one out etc. . This sort of tests is normally getting more and more precise if the sample size is raised.

The term "qualitative" describes another class, that is more interested in a descriptive and holistic assessment (as an example) of the "thing" under examination, uses questionaires and in opposite of the other class results will not getting more precise if the sample size will be raised above a certain point. Goal is to get deeper/better insight about the processes working .

There is sort of an overlap between classes, means they are not always clearly distinct, and sometimes work better if used in conjunction.

@ Blumlein88,

I had exactly the same thought. What? I read the sentence 4 times and then decided it actually didn't say anything.

I don´t mind, as it is a nice example of bias at work.....
I mean, today access to the largest pool of knowledge ever is available at our fingertips; if you had just googled something like "quantitative qualitative research" .....

 

[SoundAndMotion]

The point about music is that it is 'art', and not only does a human's hearing respond to it, but also their mind at many different levels.

Music is art and music is sound. Use as needed.

You are proposing to test their hearing by asking them, effectively, how they feel in their mind about some art.

I have never proposed such a thing.
I don’t ask you to bother to understand what I write… UNLESS you respond to what I write. Then, I respectfully ask you do try to understand. Ask what I mean if I’m unclear. Or ignore me.

If you were recommending that the stimulus was bleeps and clicks, I might be more persuaded that this was science. If the idea is "Well the system is going to be used for reproducing music, so we need to know how people respond to it with music", I would say that that was fine, but it isn't science.

Your criteria for “what is science and what isn’t” does not show much knowledge of “audio science”.
As you describe it, a test with bleeps and clicks, where I ask “how does that make you feel?” might be science, but when Amir compares 320kbps MP3 to CD and correctly identifies it 18/20 times, “it isn't science” if he used music?

Maybe you and I shouldn’t argue about what science is, but rather just agree that music can bring enjoyment and go enjoy some.

 

[Cosmik]

Music is art and music is sound. Use as needed.

I have never proposed such a thing.
I don’t ask you to bother to understand what I write… UNLESS you respond to what I write. Then, I respectfully ask you do try to understand. Ask what I mean if I’m unclear. Or ignore me.
Your criteria for “what is science and what isn’t” does not show much knowledge of “audio science”.
As you describe it, a test with bleeps and clicks, where I ask “how does that make you feel?” might be science, but when Amir compares 320kbps MP3 to CD and correctly identifies it 18/20 times, “it isn't science” if he used music?

Maybe you and I shouldn’t argue about what science is, but rather just agree that music can bring enjoyment and go enjoy some.

At the end of the day, what are you hoping to use your "audio science" for? I think usually it is to prove audiophiles wrong, whereas I am interested in developing better gear. If real science can be used for that, then count me in. I just won't be wasting my time running listening tests if I can just make objective measurements or work it out on paper.

The soon-to-be-obsolete issue of lossy encoding is an example where listening to it gives a good indication of whether or not it is working. But like the designer of that Schiit amp who said he had demonstrated that it wouldn't shut itself down for real music, you cannot know what 'music' people are going to listen to. Maybe your lossy encoding algorithm works great for Bing Crosby but has some nasty artefacts on a khongou (thank you Wikipedia). And if it works fine for the khongou, it may fall flat on its face with a nyckelharpa. Listening to music alone will not help you understand what is going on. And people can learn to prefer the sound of MP3 over unencoded because maybe that was the sound of their youth. Like my example of using the scientific method to find the 'best' shop lighting for fashion retailing, the use of the scientific method doesn't make it science. Real, actual science doesn't have room for such ephemeral, 'cultural' aesthetic stuff.

 

[Blumlein 88]

@ Don Hills,


Who could resist to respond to such eloquent asking.......

The term "quantitative" describes a class of methods/tests that uses standardized procedures to get numerical results on single features (or a small number of features) that can be analyzed with the usual statistics. To this class belong the tests routinely used like ABX, AB, triangle, sort the odd one out etc. . This sort of tests is normally getting more and more precise if the sample size is raised.

The term "qualitative" describes another class, that is more interested in a descriptive and holistic assessment (as an example) of the "thing" under examination, uses questionaires and in opposite of the other class results will not getting more precise if the sample size will be raised above a certain point. Goal is to get deeper/better insight about the processes working .

There is sort of an overlap between classes, means they are not always clearly distinct, and sometimes work better if used in conjunction.

@ Blumlein88,



I don´t mind, as it is a nice example of bias at work.....
I mean, today access to the largest pool of knowledge ever is available at our fingertips; if you had just googled something like "quantitative qualitative research" .....

From this

https://crlte.engin.umich.edu/wp-co...-Basics-A-Guide-for-Engineering-Educators.pdf

I read:

Much qualitative research is based on an interpretivist perspective, which states that truth is contextual, depending on the situation, the people being observed, and even the person doing the observation.

Then I think: thanks, but no thanks. I do not buy the epistemology of it. It will take some convincing for me to think much of this approach in the context of high fidelity sound reproduction. Maybe in the context of desired preferences of reproduction it would have some use.

 

[Cosmik]

From this

https://crlte.engin.umich.edu/wp-co...-Basics-A-Guide-for-Engineering-Educators.pdf

I read:

Much qualitative research is based on an interpretivist perspective, which states that truth is contextual, depending on the situation, the people being observed, and even the person doing the observation.

Then I think: thanks, but no thanks. I do not buy the epistemology of it. It will take some convincing for me to think much of this approach in the context of high fidelity sound reproduction. Maybe in the context of desired preferences of reproduction it would have some use.

That looks like a good read. I'll have a look later.

 

[SoundAndMotion]

At the end of the day, you've laid out your agenda quite clearly. And I don't think our agendas overlap much.
But since you ask:

At the end of the day, what are you hoping to use your "audio science" for?

"Audio science" is not mine, but it is the name of the forum. I use "my science" to try to understand multisensory information processing in the brain. Over the years I have used neurophysiology, motor physiology and psychophysics to advance my work. You know the "ephemeral, 'cultural' aesthetic stuff". In the engineering firm where I now work, my model is being combined with an optimization algorithm to provide the most realistic multisensory experience for our customers. But no fashion retailing.
Unfortunately, I find the goal of "proving audiophiles wrong" drives the anti-science movement further away from science, rather than helping people understand the importance of science.
So, have fun, and good luck selling your view of real science. I know others share it.

 

[Jakob1863]

<snip>

Much qualitative research is based on an interpretivist perspective, which states that truth is contextual, depending on the situation, the people being observed, and even the person doing the observation.

Then I think: thanks, but no thanks. I do not buy the epistemology of it. It will take some convincing for me to think much of this approach in the context of high fidelity sound reproduction. Maybe in the context of desired preferences of reproduction it would have some use.

Wow, fast transition from "doesn´t mean anything" to "do not buy the epistemology of it" ....

But to be serious, the sentence you´ve quoted is strongly related to everyday practice of "blind tests" in the audio field. If you don´t buy it, you have to believe that
-) the difference between a test situation and the "normal listening" can´t have any influence (never ever, period)
-) the "people overserved" i.e. the participants will work absolutely reliable exactly in the same manner, regardless of .....
-) even if the experimenter i.e. "the person doing the observation" have told upfront explicitely what the result of the experiment has to be, it will not have any impact on the participants (never ever, period)

PS. and we know already from other studies in cognity psychology that these things do have an impact (which means, might have/can have an impact)

I admit having a bit exaggerated for the reason of clarity, but if you think about the premise for demanding "blind" tests (i.e. participants can´t do anything against the "i know" bias) it is surprising that these same participants, who were just a moment ago helplessly struggling with the "i know" bias, are all of a sudden able to control every bias else if only the "i know" bias is removed.

@ SoundAndMotion,

maybe there is a misunderstanding with cosmik´s point of view; imo he thinks that audio science is mostly used to "prove audiophiles wrong" but that seems to be of no interest to him, because he is mainly interested in making audio gear better. I´m not sure where this idea about the main goal of audio science is emerging .....

 

[SoundAndMotion]

@ SoundAndMotion,

maybe there is a misunderstanding with cosmik´s point of view; imo he thinks that audio science is mostly used to "prove audiophiles wrong" but that seems to be of no interest to him, because he is mainly interested in making audio gear better. I´m not sure where this idea about the main goal of audio science is emerging .....

That can't be!! I just got on Cosmik's case for not reading my post carefully...

Oops, I think you're right. And so I must apologize to Cosmik for both misreading and having such a hostile tone. I edited the post a bit. And thanks Jakob for pointing this out.

-) the difference between a test situation and the "normal listening" can´t have any influence (never ever, period)
-) the "people observed" i.e. the participants will work absolutely reliably exactly in the same manner, regardless of .....
-) even if the experimenter i.e. "the person doing the observation" have told upfront explicitely what the result of the experiment has to be, it will not have any impact on the participants (never ever, period)
[snip]
but if you think about the premise for demanding "blind" tests (i.e. participants can´t do anything against the "i know" bias) it is surprising that these same participants, who were just a moment ago helplessly struggling with the "i know" bias, are all of a sudden able to control every bias else if only the "i know" bias is removed.

IME, all 3 of the above are false. But no one is expecting that removing the "I know" bias removes ALL biases. You try to remove as many biases as you can, and then try to mitigate the rest, the best you can. The "I know" bias actually accounts for many cognitive biases on its own. Look at the chart of cognitive biases DallasJustice posted a month ago. Many are removed, if "I don't know".
You are not arguing that unless you can remove ALL biases, you might as well not remove any, are you?

 

[Cosmik]

I use "my science" to try to understand multisensory information processing in the brain. Over the years I have used neurophysiology, motor physiology and psychophysics to advance my work.... In the engineering firm where I now work, my model is being combined with an optimization algorithm to provide the most realistic multisensory experience for our customers.

That's marvellous, but is it applicable to "high fidelity"? We are working to a pretty straightforward definition of what 'hi-fi' means, and I, personally, would always be a purist, avoiding any type of psychoacoustic what-not. I want the system to be the proverbial straight wire with gain - an idea most people espouse until the speakers. I include the speakers too.

 

[Cosmik]

That can't be!! I just got on Cosmik's case for not reading my post carefully...

Oops, I think you're right. And so I must apologize to Cosmik for both misreading and having such a hostile tone. I edited the post a bit. And thanks Jakob for pointing this out.

Not a problem, honestly.

 

[Jakob1863]

<snip>

IME, all 3 of the above are false.

That´s my stance on it too.
But then imo you have to "buy the epistomology of it" means to refrain from positivism (at least partly).
There might exist an invariable truth, but we can´t access it; usually all we can hope for is an approximation. I happen to believe that it is better to avoid a strict dogmatic approach to the various school of thought as we imo in reality always work with a mixture.

But no one is expecting that removing the "I know" bias removes ALL biases. You try to remove as many biases as you can, and then try to mitigate the rest, the best you can.

Of course, i only tried to point out that the "totally junk" descriptor (wrt sighted listening) imo inevitably leads to the notion that the "i know bias" is the only uncontrollable bias while other remaining must be controllable (because several are always at work). Which means humans can learn to handle (up to a certain degree) some bias mechanism, but not the "i know bias".

´<snip>You are not arguing that unless you can remove ALL biases, you might as well not remove any, are you?

No, i´m not. :=)
I try to point out a logical fallacy.

As said before, internal validity of a sighted listening test can´t shown (without further examination using "blind" controlled listening tests) so we have to use the "blind" property in tests. The golden rule applies, block out what you can and randomize what you can´t block out.

But, just as an example the bias effect mentioned in another thread which is either a time error or a presentation order effect or even a combination, shows that we can´t avoid it (as simultaneous presentation of stimuli isn´t possible) and even randomization might not be sufficient. The proposed fix was repetition of the first stimulus after the second, but afaik there is no evidence if that is really a cure.....