Limitations of blind testing procedures

[watchnerd]

I don't think blind tests are very useful any more because all the major audio debates that we could use ABX/DBT to solve are resolved.

We now know pretty well how good stuff needs to get for transparency to kick in, both for electronics and for media.

The remaining big problems in audio playback are so huge you don't need blind testing, namely the speakers and the room.

If I expose someone to something that truly advances the experience, like HRTF, you don't need blind testing to discriminate it from something else.

Blind testing is for minutiae, and we've solved most of those to the degree than any engineering time spent on them is arguing over angels on a pinhead.

 

[tomelex]

I am still saying that blind tests are useful to remove as many forms of bias from the brains hearing system. It has been proven that hearing involves firing lot more places in the brain than visual stimulus.

Here is a simple example, I was at a club, acoustic musicians, pretty near the front, but they were at the narrow end of the room and say to the left mostly as I looked at them. When the drummer hit the drum, the sound came from the drum kit he was sitting at, when I closed my eyes, the sound now came from the wall to the right side, bouncing off from there, but my brain using visual inputs "moved" the sound to the drum kit. Taking away bias, due to the various areas all interacting on audio stimuli, is essential to really hear what the "sound" is.

Even not so well run blind tests always reveal your brains bias, and in the case of audiophiles, their real "lack" of audio acuity in many cases. Limitations of blind listening tests are far and away less significant than the positives of them IMO.

I will and always will use them whenever and wherever I can, because I am into audio, not visual effects. Yes, agreed there are plenty of things that don't even need a blind test, its when, IMO, you are spending hard earned cash, no one wants to be made the fool. Except those that want the mental masturbation of buying the "best" based only on price and looks, which is fine for them but not me.

 

[March Audio]

I don't think blind tests are very useful any more because all the major audio debates that we could use ABX/DBT to solve are resolved.

We now know pretty well how good stuff needs to get for transparency to kick in, both for electronics and for media.

The remaining big problems in audio playback are so huge you don't need blind testing, namely the speakers and the room.

If I expose someone to something that truly advances the experience, like HRTF, you don't need blind testing to discriminate it from something else.

Blind testing is for minutiae, and we've solved most of those to the degree than any engineering time spent on them is arguing over angels on a pinhead.

Whilst I sort of agree with this you still need some kind of basis to compare two bits of, how shall I put it, "competing" equipment. If you know brand, price and aesthetics you will be biased.

 

[March Audio]

I am still saying that blind tests are useful to remove as many forms of bias from the brains hearing system. It has been proven that hearing involves firing lot more places in the brain than visual stimulus.

Here is a simple example, I was at a club, acoustic musicians, pretty near the front, but they were at the narrow end of the room and say to the left mostly as I looked at them. When the drummer hit the drum, the sound came from the drum kit he was sitting at, when I closed my eyes, the sound now came from the wall to the right side, bouncing off from there, but my brain using visual inputs "moved" the sound to the drum kit. Taking away bias, due to the various areas all interacting on audio stimuli, is essential to really hear what the "sound" is.

Even not so well run blind tests always reveal your brains bias, and in the case of audiophiles, their real "lack" of audio acuity in many cases. Limitations of blind listening tests are far and away less significant than the positives of them IMO.

I will and always will use them whenever and wherever I can, because I am into audio, not visual effects. Yes, agreed there are plenty of things that don't even need a blind test, its when, IMO, you are spending hard earned cash, no one wants to be made the fool. Except those that want the mental masturbation of buying the "best" based only on price and looks, which is fine for them but not me.

This.

 

[watchnerd]

Whilst I sort of agree with this you still need some kind of basis to compare two bits of, how shall I put it, "competing" equipment. If you know brand, price and aesthetics you will be biased.

I can't think of a single category of traditional audio electronics that isn't a 'solved problem' at this point, and thus no longer warrants wasting time listening on research-grade ABX/DBT tests, unless the product is either intentionally colored or incompetently /cheaply engineered. Even The Absolute Sound is calling amplifiers a 'solved problem'.

Same is true of digital music formats: we know high grade lossy vs Redbook vs high resolution is transparent to pretty much everybody.

This leaves transducers, DSP and acoustics as the problem areas that need all the R&D attention / money.

 

[watchnerd]

Even not so well run blind tests always reveal your brains bias, and in the case of audiophiles, their real "lack" of audio acuity in many cases. Limitations of blind listening tests are far and away less significant than the positives of them IMO.

But here's the rub:

Audio engineers have done enough blind tests to know the most relevant parameters related to electronics and digital music. We don't need more blind listening studies on amps, lossy vs lossless, or jitter to advance the art on the electronic side.

We do need more listening studies for speakers, acoustics, VR, etc.

As for the placebophiles, they don't want to hear it.

You're not going to convert die-hard subjectivists by doing more blind tests because they disregard results that create cognitive dissonance.

 

[fas42]

OTOH, hearing a system get "messy" doesn't need blind testing to pick - the only debate then is the why ...

 

[Blumlein 88]

If it's all so simple, then why use music as your test signal and not tones, noise, clicks and bleeps? This is the point where the whole "It's real science" thing falls apart. It turns a scientific experiment into a beauty parade and, as we know, beauty exists only in the eye of the beholder. The result can be that a real difference is masked by the 'emotional' content of the music, and that supposed preferences are really just a response to novelty or fashion. You can launder the results into statistics with six decimal places of course.

Actually tones, noise, clicks, pops, bleeps etc are better for testing limits of hearing and really seeing what can be heard. Immediately you get people then complaining it is artificial and doesn't mean anything because people listen to music not test beeps. SIGH!

 

[Blumlein 88]

I don't think blind tests are very useful any more because all the major audio debates that we could use ABX/DBT to solve are resolved.

We now know pretty well how good stuff needs to get for transparency to kick in, both for electronics and for media.

The remaining big problems in audio playback are so huge you don't need blind testing, namely the speakers and the room.

If I expose someone to something that truly advances the experience, like HRTF, you don't need blind testing to discriminate it from something else.

Blind testing is for minutiae, and we've solved most of those to the degree than any engineering time spent on them is arguing over angels on a pinhead.

I agree. Other than testing to convince the unconvinced. The only place with room to make big improvements is the tranducers at each end.

 

[watchnerd]

Other than testing to convince the unconvinced.

Does this actually work?

For every subject you get who admits he/she can't differentiate Redbook from high resolution, it seems like you get many more who refuse to accept the results, claim the test is faulty, claim ABX is a flawed mechanism, etc etc

 

[Blumlein 88]

Does this actually work?

For every subject you get who admits he/she can't differentiate Redbook from high resolution, it seems like you get many more who refuse to accept the results, claim the test is faulty, claim ABX is a flawed mechanism, etc etc

Well in maybe 2% of the cases it works. Works better if you can manage to provide the experience more than once. Definitely a low percentage play among audiophiles.

Now I don't think audiophiles realize what their public image is. People who spend crazy amounts on whacky ideas comes close to the picture in the general public's mind. What I hear repeatedly among 'civilians' is a variation on "it makes sense the ear can only hear so much. What are they thinking?" Not a pretty picture actually.

 

[watchnerd]

People who spend crazy amounts on whacky ideas comes close to the picture in the general public's mind.

I wouldn't disagree with that definition.

I no longer consider myself an audiophile because I no longer have a shared worldview or belief system with current audiophile culture.

I'm an "audio enthusiast".

 

[RayDunzl]

I'm an "audio enthusiast".

http://www.enthusiastnetwork.com/

SHUTTERBUG
STEREOPHILE
SOUND & VISION
INNERFIDELITY
AUDIOSTREAM
ANALOGPLANET

Try again.

 

[watchnerd]

http://www.enthusiastnetwork.com/

SHUTTERBUG
STEREOPHILE
SOUND & VISION
INNERFIDELITY
AUDIOSTREAM
ANALOGPLANET

Try again.

Well, crap.

I hate to use the term music lover because it only covers half of it.

 

[RayDunzl]

I'll go with audiophile and point out the ignorant nature of those who think there is a negative connotation.

Audiophile
NOUN
informal
A hi-fi enthusiast

hi-fi
ADJECTIVE
Of, used for, or relating to the reproduction of music or other sound with high fidelity.
NOUN
A set of equipment for high-fidelity sound reproduction

Nobody ever says it anyway, and if they do, who cares?

 

[watchnerd]

Nobody ever says it anyway, and if they do, who cares?

Oh I hear it all the time, although it's usually brandished at hipsters audiophiles who do things like bring Crosley turntables to parties.

 

[March Audio]

I can't think of a single category of traditional audio electronics that isn't a 'solved problem' at this point, and thus no longer warrants wasting time listening on research-grade ABX/DBT tests, unless the product is either intentionally colored or incompetently /cheaply engineered. Even The Absolute Sound is calling amplifiers a 'solved problem'.

Same is true of digital music formats: we know high grade lossy vs Redbook vs high resolution is transparent to pretty much everybody.

This leaves transducers, DSP and acoustics as the problem areas that need all the R&D attention / money.

So if thats the case I would ask you what is the minimum price I need to pay for a dac to acheive state of the art performance?

I think the bit you are missing here is its not about research. Equipment is still built to a price point and compromises are always made. So whilst I have some agreement with your assertion that a lot of electronics is a solved problem, the reality is not all equipment is equal.

A simple example is for the amplifiers you mention. Reduce the transformer size to reduce cost but in doing so reduce drive capability for difficult speakers.

 

[Blumlein 88]

So if thats the case I would ask you what is the minimum price I need to pay for a dac to acheive state of the art performance?

We talking state of the audible art or state of the art? My guess on the SOTA is possibly the Benchmark DAC3. They get near 22 bit performance for $2k. Only a small number of DACs very slightly edge them out. They cost anywhere from about $20k to more than $100k (not counting any I may not be aware of naturally).

Audible SOTA maybe $500 for the Emotiva Stealth. Or perhaps a similar amount for some prosumer DACs. Possibly between $500 or $1000 for various versions of the TEAC offerings.

 

[oivavoi]

I would also assume the dac in the Emotiva Stealth to be the limit of audibility. Another question is whether there's any point in using independent dacs for each channel in actively run systems. The Kii Three do that, as do Linn in their active systems. My hunch would be that it's not necessary. But I struggle to believe that Kii and Bruno Putzeys would do something just to show off. Maybe it does measure demonstrably better?

 

[Blumlein 88]

So if thats the case I would ask you what is the minimum price I need to pay for a dac to acheive state of the art performance?

I think the bit you are missing here is its not about research. Equipment is still built to a price point and compromises are always made. So whilst I have some agreement with your assertion that a lot of electronics is a solved problem, the reality is not all equipment is equal.

A simple example is for the amplifiers you mention. Reduce the transformer size to reduce cost but in doing so reduce drive capability for difficult speakers.

The other end of things, but 2 or 3 years ago Sound on Sound recorded an electronically controlled grand piano in a large studio (possibly Abby Road). They used a couple different pair of mics and each of those when thru AD/mic pre units ranging between $450 and something like $10-12 k. They put the files up and let people vote on them for preference. There was also a discussion on their forums for identifying which was which. No consensus and not many takers originally. Once they had a fair number of votes for preference the results were split pretty evenly. The number one vote getter: the $450 unit. 2nd was a moderately expensive unit, and third was the 2nd least expensive unit. The real take away was no unit got overwhelming support as best. Now this may not mean they sound the same as much as different people had different preferences. However, this along with a few similar tests posted simply for difference finding indicates ADC's at any level from somewhere below $1k is a solved problem. Microphone pre's unless you are looking for character and color are a solved problem. I think the playback end is similar. Pre's, DAC's and such are all a solved problem unless you purposely skew the device for character. Amps come close, but I agree they interact with speakers or at least often can. So they might not be solved for all speakers, but it is getting very close to a solved problem. Transducers are the areas yet to be solved and may be somewhat unsolvable in the near future.

I posted the 8th gen copies and the digital originals using a pair of $400 recording interfaces. That is in my opinion pretty convincing evidence lower price digital does little harm.