On Peer Reviewed Science

[SIY]

Hedonic analysis is the term you're looking for.

See Stone, H and Sidel, JL. 1993. Sensory Evaluation Practices. 2nd ed. Academic Press: San Diego for a good treatment.

 

[Cosmik]

A hypothetical example derived from that earlier comment of mine:

You are a speaker maker, and curious about what could be achieved by a unique combination of motion feedback and neural network correction of a bass driver. You work on it for a while and then listen to it. It's astounding; it sounds more like real bass than you've ever heard. The motion feedback on its own got close, but the neural network refinement compensates for some of the unavoidable cone flexing at high accelerations and the result is a step change in bass reproduction.

At this stage, your experiment is genuinely objective, scientific: you haven't 'voiced' the speaker by listening to it; you have merely closed the loop with feedback on the measurements and you can document everything as being 'objective'.

But you decide to do some real 'science'. You assemble a panel of listeners and, because you are doing science, you don't tell them what they are listening for. You play to them a selection of commercial speakers (volume matched to within 0.1 dB, etc. - all that good stuff) and also your new speaker. You ask them to give their 'preferences'. Sound is the only variable in play.

You put the results through the statistical analysis. To your dismay, you find that your new speaker is in the lower reaches of the rankings. Science has proven that your idea was schiit. You abandon the idea and scrap the project.

Your paper is published though - you have to get something out of it. Other speaker makers and scientists read it and take it as a warning; passive bass reflex boxes actually sound superior. Science has proved it.

What happened? The answer is that the listeners came with preconceived ideas. They knew what a 'good' audio system sounded like, because they own expensive ones themselves or have heard them at shows. Your speaker sounded nothing like those 'good' systems, and in fact shocked them; they had never heard those recordings sound like that before. Your speaker must just be a freakily bad speaker.

By committing your work to 'science' you fooled yourself. You took an objectively correct system and put it through a subjective non-scientific experiment, but included the 'usual' sleight of hand of putting the participants' subjectivity in a black box.

Had you, instead, declared in a big advert that you had produced a radically better bass driver, and asked listeners to evaluate it on that basis i.e. told them what to expect (shock horror - non-science!!!), you might have had a smash hit on your hands and been able to sell the idea under licence to other speaker manufacturers.

But looking on the bright side, you got your name on a published paper.

 

[SIY]

You put the results through the statistical analysis. To your dismay, you find that your new speaker is in the lower reaches of the rankings. Science has proven that your idea was schiit. You abandon the idea and scrap the project.

Well, the aim of the project as detailed was to determine consumer preference. You determined it using methods developed by people doing science. You may not like the answer you got, but it is the answer. Your idea may have been schiit from a commercial POV, but if there's scientific merit to it, you can design experiments demonstrating or refuting that.

I had a similar experience in a haptics project. I was designing a computer pointing device and the test used (called a Fitt test) involved chasing down targets. I showed that the target acquisition time was better with the force-to-velocity algorithm I liked, but our consumer test panel hated it. So it would not be a commercially successful product, though from a purely objective POV, my algorithm was better. I dumped my algorithm and went on to sell quite a few million of that device.

Think of the "loudness wars" versus objective accuracy.

 

[andreasmaaan]

Another interesting scientific experiment based on your example @Cosmik would be to test for audibility of the measured improvements of the new device. If listeners are unable to pick a difference between corrected and uncorrected bass drivers under controlled conditions, then the new design may cost more and be more complex to design, but seems to offer no audible advantage over previous designs.

 

[Cosmik]

Well, the aim of the project as detailed was to determine consumer preference. You determined it using methods developed by people doing science.

You simply wanted to develop the most popular sound with listeners, and you assumed that the way to determine it was to isolate sound as the only variable in a test with listeners. Science gave you an off-the-shelf methodology.

But your experiment was flawed, because although sound was the only explicit variable in your experiment, the participants brought in a whole load of stuff in their heads and you had no idea of the nature of that stuff. You fooled yourself by believing your own sleight of hand; that 'preference' meant unconditional 'best' or 'most popular' or 'most pleasurable'. The listeners responded, in fact, to familiarity.

The same listeners in ten years time, or in another country, or a different bunch of listeners in different weather, etc. would have responded differently - and non-scientists can see that a mile off. People heavily involved in science maybe can't..!

The result didn't mean that your sound wasn't best/most popular/most pleasurable, and nor did it mean that your sound wouldn't be the most popular with customers. It's just that science couldn't give any useful/stable/universal/timeless answer. Science is not applicable to aesthetic judgement even if an experiment can be framed so that it 'passes' the 'science test' with other similarly-inclined 'peers'.

 

[andreasmaaan]

You simply wanted to develop the most popular sound with listeners, and you assumed that the way to determine it was to isolate sound as the only variable in a test with listeners. Science gave you an off-the-shelf methodology.

But your experiment was flawed, because although sound was the only explicit variable in your experiment, the participants brought in a whole load of stuff in their heads and you had no idea of the nature of that stuff. You fooled yourself by believing your own sleight of hand; that 'preference' meant unconditional 'best' or 'most popular' or 'most pleasurable'. The listeners responded, in fact, to familiarity.

The same listeners in ten years time, or in another country, or a different bunch of listeners in different weather, etc. would have responded differently - and non-scientists can see that a mile off. People heavily involved in science maybe can't..!

The result didn't mean that your sound wasn't best/most popular/most pleasurable, and nor did it mean that your sound wouldn't be the most popular with customers. It's just that science couldn't give any useful/stable/universal/timeless answer. Science is not applicable to aesthetic judgement even if an experiment can be framed so that it 'passes' the 'science test' with other similarly-inclined 'peers'.

I completely agree, but I think you're misconstruing the purpose of the test.

The test is designed to determine listener preference. It is not designed to be able to determine the underlying factors resulting in listener preference. So the experiment was not flawed, it's conclusions were just more narrow than you had hoped.

EDIT: and this is why audibility studies are more powerful and interesting, from my point of view, than preference studies. Audibility studies allow us to understand human hearing itself, whereas preference studies are well-executed and controlled market research.

 

[SIY]

So the experiment was not flawed, it's conclusions were just more narrow than you had hoped.

Or, more accurately, the experiment was specifically designed to determine listener preference. If you want other data, you need to design a different experiment. I have a problem with the idea that there's some "universal" experiments or standards- they all have to be looked at on a case-by-case level, rather than have a Procrustean approach.

This is a general statement, not peculiar to audio. If I want to measure the e/m ratio of an electron, I should not be disappointed that I didn't determine spin from that same experiment.

 

[andreasmaaan]

Or, more accurately, the experiment was specifically designed to determine listener preference. If you want other data, you need to design a different experiment. I have a problem with the idea that there's some "universal" experiments or standards- they all have to be looked at on a case-by-case level, rather than have a Procrustean approach.

I thought this was precisely what I said in the portion of my post that you didn't quote

 

[SIY]

Think of it as expansion.

 

[Cosmik]

I completely agree, but I think you're misconstruing the purpose of the test.

The test is designed to determine listener preference. It is not designed to be able to determine the underlying factors resulting in listener preference. So the experiment was not flawed, it's conclusions were just more narrow than you had hoped.

That's why I'm not arguing against the fact that it passes some people's 'science test'.

Science, from what I am learning in this thread, can literally be the declaration that "These people on this day under these conditions said this when showed this". A claim so narrow as to be almost nonexistent, and yet, apparently, still science. At that point, I have to give up on arguing that listening tests are not science.

So my new angle is whether or not certain types of science are useful. If they are not useful, and never can be (I'm not talking about no one having an immediate use for a laser in 1947!) then why do the science?

You, as a speaker maker, were suggesting that scientific preference tests might be a good thing in developing your speakers a few posts back. So it was obviously you I had you in mind with my story

 

[andreasmaaan]

That's why I'm not arguing against the fact that it passes some people's 'science test'.

Science, from what I am learning in this thread, can literally be the declaration that "These people on this day under these conditions said this when showed this". A claim so narrow as to be almost nonexistent, and yet, apparently, still science. At that point, I have to give up on arguing that listening tests are not science.

So my new angle is whether or not certain types of science are useful. If they are not useful, and never can be (I'm not talking about no one having an immediate use for a laser in 1947!) then why do the science?

You, as a speaker maker, were suggesting that scientific preference tests might be a good thing in developing your speakers a few posts back. So it was obviously you I had you in mind with my story

Ha, thanks

I just think we're discussing preference tests a lot here, when the real value of science in audio is in testing for audibility.

 

[Cosmik]

Ha, thanks

I just think we're discussing preference tests a lot here, when the real value of science in audio is in testing for audibility.

Well, OK. But can you answer this question:

Does the knowledge that you are performing in an experiment affect your ability to hear small differences?

 

[andreasmaaan]

Well, OK. But can you answer this question:

Does the knowledge that you are performing in an experiment affect your ability to hear small differences?

It certainly may, yes, although I think most evidence suggests it does so for the better (i.e. more sensitive).

 

[Cosmik]

It certainly may, yes, although I think most evidence suggests it does so for the better (i.e. more sensitive).

How was that determined?

 

[andreasmaaan]

Audibility tests also often train listeners to hear the difference being tested for beforehand, and are structured to confine testing to short periods in which concentration is at a maximum, or to allow participants to run tests at their own pace. These measure also help to ensure that listeners' ears are more sensitive than average listeners under normal conditions.

 

[andreasmaaan]

How was that determined?

There's a whole discussion on this somewhere else on this forum! Let me see if I can find it.

 

[andreasmaaan]

Ok, it's a long thread..

Further discussions of the same topic here and here.

 

[Cosmik]

Ok, it's a long thread..

Further discussions of the same topic here and here.

There may be discussions, but I wasn't aware they contained any useful nuggets! I'm just intrigued as to how a scientific experiment on audibility is performed without alerting the participants that they are taking part in an experiment.

 

[andreasmaaan]

There may be discussions, but I wasn't aware they contained any useful nuggets!

Here's a useful nugget from @amirm summarising the David Clark study:

"All in all, the position of audio science on this matter is clear: fast AB switching is far more revealing than any long term tests. No evidence has ever been presented to show otherwise or to demonstrate anything based on psychoacoustics why that would be so."

 

[Cosmik]

Here's a useful nugget from @amirm summarising the David Clark study:

"All in all, the position of audio science on this matter is clear: fast AB switching is far more revealing than any long term tests. No evidence has ever been presented to show otherwise or to demonstrate anything based on psychoacoustics why that would be so."

Yep. That's what I thought! There is no evidence to the contrary. But no actual evidence. Maybe a Truman Show type experiment would work..?