Limitations of blind testing procedures

[March Audio]

But didnt the Toole work demonstrate that people actually prefer speakers that were nuetral / sound power and dispersion characteristics that make technical sense?

Yeah sure some people like stuff that diverges from normal and technically correct, but are they just defective individuals?

 

[Cosmik]

But didnt the Toole work demonstrate that people actually prefer speakers that were nuetral / sound power and dispersion characteristics that make technical sense?

Yeah sure some people like stuff that diverges from normal and technically correct, but are they just defective individuals?

So why do we carry on doing the listening tests? If you trust Toole's conclusions, why not just go with the most neutral specs from now on?

 

[amirm]

Is 'an amplifier' something that a human 'likes'? I would say no. No more than an electricity substation, or a manhole cover.

Average person would be that way. But high-end audiophiles who think night and day happened when they changed amps, certainly cherish the differences they think, or are hearing.

 

[Thomas savage]

Average person would be that way. But high-end audiophiles who think night and day happened when they changed amps, certainly cherish the differences they think, or are hearing.

We know the real ' night and day ' happens when you start wearing anti static underpants ( y-fronts)

 

[Fitzcaraldo215]

If it has good measurements it will sound good ,the two go hand in hand, you may prefer something with more distortion of course.
Keith

Yes, of course, Keith. I would not even waste my time listening to stuff that did not have "good" measurements in the first place. So, getting to "sounds good" is not the problem. A lot of stuff sounds good these days. The problems are (1.) how far does one pursue the ever better measurements idea? Example: .3% THD, 0.03% or .003%? Or, at what point does that chase become useless and irrelevant? And, (2.) Does one actually sound better, A or B? Or, do they sound the same in spite of one having an order of magnitude lower THD? An octave more ultrasonic HF response? Etc.? And, (3.) are there other measurements that might be more revealing beyond the commonly available measurements in the audio mags?

Believe me, as an audiophile for decades, everything I ever bought sounded "good". Was there better? Maybe. But, I never had any regrets about my own choices in terms of how it sounded within my budget.

I am comfortable wth my selection skills and how they have become more sophisticated over time. So, these are not exactly burning issues for me personally. But, it is never as simple as just buying from the spec sheet or the measurements. Listening has to be involved, of course, even if only for final confirmation, blind or double blind if possible. But, even sighted if that is the best you can do.

Incidentally, do your customers ask for blind AB tests and/or do you suggest them? Not a trick question or trying to perpetate the argument. Just curious.

 

[Cosmik]

Believe me, as an audiophile for decades, everything I ever bought sounded "good". Was there better? Maybe. But, I never had any regrets about my own choices in terms of how it sounded within my budget.

Not so for me. The light bulb moment was realising that passive bass reflex speakers are miles off 'neutral' by design.

 

[Fitzcaraldo215]

Not so for me. The light bulb moment was realising that passive bass reflex speakers are miles off 'neutral' by design.

I realize you are a fan of actives. I am definitely in favor of that technology and its pursuit by more manufacturers. But, I did not realize your concerns went to bass reflex vs. sealed, as well. All else being equal, I would prefer the sealed box, as well.

In any case, the available measurements out there do not seem to give a hint of any problems for many or at least some passive, bass reflex speakers on the market. Certainly, there is nothing consistent that I see.

 

[Cosmik]

In any case, the available measurements out there do not seem to give a hint of any problems for many or at least some passive, bass reflex speakers on the market. Certainly, there is nothing consistent that I see.

I think you are right - measurements may not tell the whole story unless particularly comprehensive.

A short cut might be to try to understand how a particular technology works. In the case of bass reflex (I am not an expert) for the port to actually work its frequency response-centric magic, it has to corrupt the time domain rendition. This would then show up in the right set of measurements - although you wouldn't bother to measure it if, like me, you are against them on principle!

 

[Purité Audio]

Yes, of course, Keith. I would not even waste my time listening to stuff that did not have "good" measurements in the first place. So, getting to "sounds good" is not the problem. A lot of stuff sounds good these days. The problems are (1.) how far does one pursue the ever better measurements idea? Example: .3% THD, 0.03% or .003%? Or, at what point does that chase become useless and irrelevant? And, (2.) Does one actually sound better, A or B? Or, do they sound the same in spite of one having an order of magnitude lower THD? An octave more ultrasonic HF response? Etc.? And, (3.) are there other measurements that might be more revealing beyond the commonly available measurements in the audio mags?

Believe me, as an audiophile for decades, everything I ever bought sounded "good". Was there better? Maybe. But, I never had any regrets about my own choices in terms of how it sounded within my budget.

I am comfortable wth my selection skills and how they have become more sophisticated over time. So, these are not exactly burning issues for me personally. But, it is never as simple as just buying from the spec sheet or the measurements. Listening has to be involved, of course, even if only for final confirmation, blind or double blind if possible. But, even sighted if that is the best you can do.

Incidentally, do your customers ask for blind AB tests and/or do you suggest them? Not a trick question or trying to perpetate the argument. Just curious.

I encourage customers to listen to components in their own homes, especially loudspeakers , I don't feel I can insist on unsighted comparison, but am more than happy to arrange level matched unsighted listening here.
Speakers like the Kii's and Dutch &Dutch are obviously better even than really decent passives like the Revels I heard again today.
Keith

 

[Jakob1863]

<snip>Are you just trying to be rude?

Here we are discussing the usage of English terminology as it relates to describing type of research data. And you go and just put forward a German paper as your backup???

You wanted a reference and you got one.

Btw, you post included:
"Listening tests are subjective"
and i cited it and reported another point of view wrt your statement.

I responded to your post, gave an example for an audio field that follows the same approach as you did and then i reported another point of view, where the authors explained why they think a listening test is objective (if done in a sound way).

That aside, that is a 2-page opinion piece, not audio listening test results of interest to the membership I asked about.

Of course it is an opinion piece in which the authors argue why they consider listening tests (if representing good scientific practice) as being objective. Their headline included the question "are listening tests subjective?"

The author's case is that since there is unreliability in objective data, perhaps they are not that reliable/objective. So what? Of course there is unreliability in objective measurements. It is up to reader/researcher to determine that degree of error before drawing conclusions.

You are mistaken; that´s not the authors point. The author´s point is that, if you use a listener as a detector in a good scientific test then this test should be considered as objective because the test is valid, reliable and objective (within the limits of the detector).

Nothing in that paper says we should stop using those terms to refer to listening tests and measurements with the terms we have been using.

It´s the other way round, as the authors indeed said in their next to last paragraph (translation by me):
"Suggested Terminology

It seems to be sensible not to differentiate between "objective" and "subjective" but instead to discern between "technical" and "perceptual" or between "physical" and "psychological" measurements. As the objectivity can be ensured by carefully and reproduceable execution for both, is it important to evaluate which degree of validity and reliability is needed wrt to the hypothesis under research"

Indeed, both authors are editors of proceeding of a symposium where they fully follow what I have stated:

Obviously the authors realized that their proposed terminology wasn´t accepted by the majority (up to now?)!?

<snip>
So once again, this topic is not up for debate. Audio research has standardized on the term "subjective" for listening tests and "objective" for measurements. You no more can try to redefine them than you would in use of English words in Law. They are what they are and are not up for mis-use as that just creates confusion.

If someone uses the terminology "measurements and perceptual evaluation" i assume that most (if not all) real researchers understand what is meant.....

 

[Thomas savage]

You wanted a reference and you got one.

Btw, you post included:
"Listening tests are subjective"
and i cited it and reported another point of view wrt your statement.

I responded to your post, gave an example for an audio field that follows the same approach as you did and then i reported another point of view, where the authors explained why they think a listening test is objective (if done in a sound way).



Of course it is an opinion piece in which the authors argue why they consider listening tests (if representing good scientific practice) as being objective. Their headline included the question "are listening tests subjective?"



You are mistaken; that´s not the authors point. The author´s point is that, if you use a listener as a detector in a good scientific test then this test should be considered as objective because the test is valid, reliable and objective (within the limits of the detector).



It´s the other way round, as the authors indeed said in their next to last paragraph (translation by me):
"Suggested Terminology

It seems to be sensible not to differentiate between "objective" and "subjective" but instead to discern between "technical" and "perceptual" or between "physical" and "psychological" measurements. As the objectivity can be ensured by carefully and reproduceable execution for both, is it important to evaluate which degree of validity and reliability is needed wrt to the hypothesis under research"



Obviously the authors realized that their proposed terminology wasn´t accepted by the majority (up to now?)!?



If someone uses the terminology "measurements and perceptual evaluation" i assume that most (if not all) real researchers understand what is meant.....

I don't think we are getting anywhere here, please can we move on.

Thanks .

 

[Jakob1863]

I don't think we are getting anywhere here, please can we move on.

Thanks .

I´d appreciate it.

 

[Jakob1863]

<snip>

So I say, testing the human limits of what *can* be heard is all very nice as science, but for real impact in our strange little hobby, subjecting actual self-professed audiophiles -- say , Michael Fremer -- to proper DBTs, would be vastly more influential ....not to mention, far more entertaining, than academic research. A pity we no longer have Tom Nousaine to lead the way there.

Regardless whether one agrees or diagrees with Michael Fremer´s views, he should acknowledge that Fremer was willing to participate in controlled listening tests and had both times quite disappointing experiences as his opponents weren´t really playing fair.
He was quite confident being able to hear differences between power amplifiers and participated in a listening test hold during an AEC convention in the 1980s, got his score (answers to same/different correct in 5 trials out of five, and correct identifications within these trial in 4 of five trials, three amplifiers under test) but was later dismissed as the "lucky coin" as a hundred (or so) other participants weren´t able to detect a difference. He stepped up a second time only to see Randi playing unfair (means weaseled out), we discussed that briefly over at hydrogenaud .

 

[Purité Audio]

Insisting that the cables had the same electrical parameters is hardly 'weasling out', there was an interesting unsighted amplifier comparison carried y one of the UK mags some years ago and conducted by Martin Colloms, I will try and find some more details.
Keith

 

[Jakob1863]

Insisting that the cables had the same electrical parameters is hardly 'weasling out',...

I totally agree but that insisting on same electrical parameters didn´t take place (very surprising at that point from an engineering point of view) .....

there was an interesting unsighted amplifier comparison carried y one of the UK mags some years ago and conducted by Martin Colloms, I will try and find some more details.
Keith

That would be interesting; i remember that John Atkinson wrote about a power amplifier listening test (blind or doubleblind) conducted by ?Martin Colloms? that he attended in the 1980s.

 

[tomelex]

blind listening tests will reveal differences in amplifiers, or any other components, if there is enough of a difference for ANY individual under tests hearing to hear. I hope there is nobody here who declares that all amps sound the same when simple "standard" measurements show them to be very similar. I do not consider spectrum plots to be simple, although for the last 15 years or so atleast they have been easy enough to do by the amateur.

MF hearing differences is no big deal, (he declared an amp with like 5% distortion to be the best amp he ever heard) however, when Carver did his challenge, no one could hear the difference between a solid state amp and a tubey tube amp, that was a demonstration worth doing and done well. Despite later retractions by the golden eared brigade, that was proof enough that a master audio engineer can make an amp "sound" however you want, no matter if you want an inaccurate to the source amplifier or not.

 

[Cosmik]

I hope there is nobody here who declares that all amps sound the same when simple "standard" measurements show them to be very similar....

...when Carver did his challenge... that was proof enough that a master audio engineer can make an amp "sound" however you want, no matter if you want an inaccurate to the source amplifier or not.

But wasn't the so-called Carver challenge a simulation of various amps based on simple standard measurements alone?

I do not buy into the mythology of the 'Carver Challenge'. It is a combination of the following:
(1) Human audiophiles' inability to hear easily-measurable differences, particularly when being tested
(2) People being convinced that amplifiers are a lot more complex than they are ("Ooh, someone has made a $200 amplifier sound like one that costs $10,000. He must be a genius.")
(3) People being convinced that amplifiers are a lot more simple than they are ("By hacking about with a few components I can simulate the behaviour of a completely different amp").

Put it all together, and people can read into it what they like - and they do.

 

[SoundAndMotion]

But wasn't the so-called Carver challenge a simulation of various amps based on simple standard measurements alone?

No, not really. It's an interesting story(link) and worth reading.

 

[Jakob1863]

blind listening tests will reveal differences in amplifiers, or any other components, if there is enough of a difference for ANY individual under tests hearing to hear. I hope there is nobody here who declares that all amps sound the same when simple "standard" measurements show them to be very similar. I do not consider spectrum plots to be simple, although for the last 15 years or so atleast they have been easy enough to do by the amateur.

It´s quite rare that controlled listening tests will seek for differences that any individual will detect; closest to that are imo the sometimes quite large scale tests done on lossy compression codecs, as the goal was indeed to find some that are audibly transparent for ~95% of people.
The listening tests we deal routinely with are of a different kind and the interests are diverging; some want to "proof audiophiles" wrong, some want to test listeners, some want to find out if a perceptable difference exists between DUTs (at least to one listener) and so on.
But as fair the controvery started in the 1960s and was from the beginning disputing the fact that the measurements indicated, if compared to he known thresholds of hearing, that no perceptable difference should exist.
The measurements routinely done in the 1960s were surely a lot easier then today, although spectral analysis already existed, and were constantly improved over the decades.

MF hearing differences is no big deal, (he declared an amp with like 5% distortion to be the best amp he ever heard) however, when Carver did his challenge, no one could hear the difference between a solid state amp and a tubey tube amp, that was a demonstration worth doing and done well. Despite later retractions by the golden eared brigade, that was proof enough that a master audio engineer can make an amp "sound" however you want, no matter if you want an inaccurate to the source amplifier or not.

We briefly discussed the Carver challenge a couple of months ago; imo the most surprising thing in the challenge was the reception of the actual tests done.
The stereophile people did what they always did (and do), they listened sighted to the amplifiers. They were confident to hear a difference in the beginning and were still confident at the first Carver approach and at the end confident that they couldn´t hear any difference.
To use your term, i´d say that the "non golden ear brigade" was more than happy to accept the stereophile approach of sighted listening although it was constantly condemned before and after.

 

[DonH56]

Carver matched transfer functions to make his SS amp sound like a tube amp. I wonder what the results would have been if he had been challenged to make the tube amp sound like the SS? Seems like a tougher task to me... IIRC one of the biggest features of the resulting SS amplifier product was a resistor to raise the output impedance of the SS amp, not sure what else. Long ago.