Relevance of Blind Testing

[solderdude]

One should only blind test, and do it properly, when you yourself are interested in finding your truth.

When you want it to serve as 'evidence' substantial testing in the company of someone else administering the test multiple times over a longer period will give evidence you can go to court with.

In the end, people want something they like. For this blind tests are not needed. Just don't go posting here something IS good or better than based on one's subjective findings. Proper blind testing can be illuminating for the individual doing the test.

 

[andreasmaaan]

Regarding measurements, I have to quote Jonathan Novick: "not all that matters can be measured, not all that can be measured, matters".

If the first part of that sentence were true, wouldn't you expect there to be examples of identical-measuring components that were nevertheless subjectively distinguishable under controlled conditions?

Are you aware of any such examples?

 

[BaaM]

If the first part of that sentence were true, wouldn't you expect there to be examples of identical-measuring components that were nevertheless subjectively distinguishable under controlled conditions?

Are you aware of any such examples?

It's not me saying it, it's Jonathan Novick from AudioPrecision, who sells measuring devices, so I take this as a serious testimonial, but maybe he's wrong.
No, indeed, I don't have any examples to give, but you also have to take into consideration that the vast majority of audio blind testings result in failure, no matter what you compare.
If I remember correctly, Amir said that he had taken a training course to practice ABXs, I don't think the majority of us are trained to pass these tests.

 

[ahofer]

These are fair points. It is easier and cheaper to substitute the placebo effect of beyond-audible perfect measurements for the kilobuck steampunk placebo, if you go at it for a while.

 

[polmuaddib]

That is one of the things that interest me. When you “uncover” the differencies in sq or lack of, is a person susceptible to other influences or more objective?
I know we are not cybernetic organisms (yet) and can never be fully objective, but to some degree?
Also, looking at measurements here must play a role in our hearing?

 

[brimble]

Maybe this is slightly off topic, but I've been wanting to write up my buying philosophy. It's this:

I'm sure that for me, equipment that looks better sounds better, subjectively, regardless of its objective properties. Therefore, for me, blind testing isn't the whole story ...

Fortunately, it's easy to get equipment that looks good AND does well in blind testing at a fairly low price.

So I always (except when I make mistakes!) buy things that look good AND test well (and are likely to be reliable), and then within those criteria I get things that are as cheap as possible.

 

[andreasmaaan]

It's not me saying it, it's Jonathan Novick from AudioPrecision, who sells measuring devices, so I take this as a serious testimonial, but maybe I'm wrong.

When it comes to designing low noise/distortion circuits, I would take what J Novick says extremely seriously. But when it comes to questions concerning psychoacoustics, I would prefer to base my views on evidence from experts in that field

 

[Frank Dernie]

It really depends on what sort of difference you’re testing for. If you want to test discrimination of low-level sensory features then rapid-switching and relatively brief presentations are the way to go as our memory of such raw sensory experience decays rapidly. But if you’re interested in higher-order effects then it might be more powerful to use far longer presentations, as memory of such mental states can be encoded semantically and thus persist far longer. When designing such experiments it’s always important to have a very clear idea of precisely what you’re trying to test.

I am not convinced any more.
For years I thought long term listening was needed to appreciate the enjoyment side, and convinced myself that my system needed to be left on for a day before sounding optimum after I had been away.
10 years ago I did a test which convinced me that it was pointless buying any different electronics.
I still think it sounds better sometimes but I haven't changed anything since so it won't be doing, it is mood or something, not the hifi, which changes.

 

[BDWoody]

Having said that, I'm not aware of any evidence that audiophile-length presentation times (days, weeks or months) have ever been shown to allow for better discrimination. Has anyone come across any evidence of this?

Based on how many reviewers notice immediate and obvious differences, then who claim you need extended listening to be able to discern differences in 'fatigue', etc., the evidence is being danced all around, it's just not favorable to those who rely on obfuscation.

 

[solderdude]

That is one of the things that interest me. When you “uncover” the differencies in sq or lack of, is a person susceptible to other influences or more objective?
I know we are not cybernetic organisms (yet) and can never be fully objective, but to some degree?
Also, looking at measurements here must play a role in our hearing?

That's the thing. Not only does one need to know (involves measuring) how to properly test one also needs to be aware of their own hearing abilities.
This is not a thing you learn in a day.
So I always encourage people to 'discover' their hearing limits and when you test blind need to think about what you want to test, how to do it and to prevent as many biases from creeping in there.
A proper AB of anything but interlink cables is not easy to do properly.

I would not recommend any comparative testing before one understands their own hearing abilities.
When I was young I could hear softer sounds and higher frequencies but it took time to learn to hear certain types of distortions and where tonal issues come from etc. You can learn to become a better listener despite your ears getting crappier you can still detect things younger untrained ears can not.

 

[BaaM]

When it comes to designing low noise/distortion circuits, I would take what J Novick says extremely seriously. But when it comes to questions concerning psychoacoustics, I would prefer to base my views on evidence from experts in that field.

So what he says is false, I'm willing to hear that, even if I admit that I would like to know his arguments, It's his conclusion of the RMAF15 conference.
But this sentence is related to measurements on the one hand, and psychoacoustics on the other, not only psychoacoustics!

So who're the experts that makes you think otherwise?

 

[andreasmaaan]

So what he says is false, I'm willing to hear that, even if I admit that I would like to know his arguments in relation to this conclusion of the RMAF15 conference.
But this sentence is related to measurements on the one hand, and psychoacoustics on the other, not only psychoacoustics!

I'd also like to hear his arguments, and that's basically why I added the question to my earlier post, "Are you aware of any such examples?"

I realise that part of his statement relates to interpreting measurements, but would argue that his expertise lies in producing equipment to make measurements, rather than in interpreting the output.

So who're the experts that makes you think otherwise?

The main source of my knowledge of psychoacoustics comes from Zwicker and Fastl, who wrote what is arguably the bible on it. Their research (and research they cite) in that book paints a picture human hearing ability that, impressive as it is, falls far short of what something like an APx can measure.

 

[magicscreen]

A sugar pill can actually cure several illnesses if the subject believes it will work,

Wait a minute, I do not understand one thing. Why didn't the doctors make a sugar pill at March this year for cure the coronavirus?
They would had lied about that it is a perfect antiserum. Everybody believes and nobody gets coronavirus and nobody dies.
Problem solved.

 

[raistlin65]

One thing I've always been interested in seeing research on is at what point does noise and/or distortion affect listening to the music as an aesthetic experience?

For example, DAC A measures extremely accurate, and in DBT's with other extremely accurate DACs, DAC A is indistinguishable from the others.

DAC B has some minor amount of measured noise and/or distortion that could be audible. And in DBT, subtle differences between DAC A and DAC B are able to be picked out by most people.

So in our tests, each day for a month, listeners engage in a long listening session without knowing which DAC they are using. They are discouraged from listening to the gear through the music (they shouldn't even know what DACs are being used in the study). But rather to be submerged in the music listening session itself and then rate the aesthetic music listening experience at the end of each session. Might even be useful to include talk aloud protocols that have the listener state what they are feeling.

After a month of such tests, would we find a correlation between the enjoyment of the music and which DAC was used?

We'd have to develop a fairly sophisticated measure for various ways to assess music enjoyment. But it would be interesting to find out the results.

 

[Frank Dernie]

Wait a minute, I do not understand one thing. Why didn't the doctors make a sugar pill at March this year for cure the coronavirus?
They would had lied about that it is a perfect antiserum. Everybody believes and nobody gets coronavirus and nobody dies.
Problem solved.

I presume you are joking.

You could have taken a sugar pill in March if you thought it would work, and maybe it would have.

 

[brimble]

After a month of such tests, would we find a correlation between the enjoyment of the music and which DAC was used?

I would love to see research on this. And maybe there is some. I don't know.

But there would be a BILLION relevant variables. For example, the type of music.

I find I need good equipment to listen to organ music and orchestral music and choral music. For jazz, good equipment is nice but matters less. For some types of rock music, good equipment is disastrous - for example, some rock music has double-tracked vocals which are two voices that are meant to sound like one voice. And on cheap equipment (the type they're mixed and mastered for) it does sound like one voice, but on good equipment you can hear both voices and the effect is completely different from, and much worse than, what the musicians intended. So, complicated.

 

[raistlin65]

I would love to see research on this. And maybe there is some. I don't know.

But there would be a BILLION relevant variables. For example, the type of music.

I find I need good equipment to listen to organ music and orchestral music and choral music. For jazz, good equipment is nice but matters less. For some types of rock music, good equipment is disastrous - for example, some rock music has double-tracked vocals which are two voices that are meant to sound like one voice. And on cheap equipment (the type they're mixed and mastered for) it does sound like one voice, but on good equipment you can hear both voices and the effect is completely different from, and much worse than, what the musicians intended. So, complicated.

Yep. That's why a very sophisticated measuring tool is needed. In this type of qualitative research, I think you'd have to anticipate collecting a lot of data and looking for trends.

 

[BostonJack]

Indeed.
The placebo effect is very real, yes, and powerful.
A sugar pill can actually cure several illnesses if the subject believes it will work, so it is entirely likely that a piece of hifi will sound better to somebody who expects it to do so.

When making equipment choices, I believe in factoring in feelings or inclinations that would be ascribed to sighted bias in a listening test. Not paying attention to what brings you satisfaction is like going to the doctor and thinking "I don't really have any evidence that this doctor is capable of curing me, so I better discount him or her". Better to embrace the placebo effect, believe in the physician, swallow the tiny bitter pills with confidence. After all, placebo effects are the route to healing in many instances.

 

[pkane]

o in our tests, each day for a month, listeners engage in a long listening session without knowing which DAC they are using. They are discouraged from listening to the gear through the music (they shouldn't even know what DACs are being used in the study). But rather to be submerged in the music listening session itself and then rate the aesthetic music listening experience at the end of each session. Might even be useful to include talk aloud protocols that have the listener state what they are feeling.

I came across a description of a test that seemed to be designed to validate the "long-term evaluation is better" hypothesis:

This is from an otherwise good article on audio testing by Stuart Yaniger: https://linearaudio.net/sites/linearaudio.net/files/LA Vol 2 Yaniger(1).pdf

 

[levimax]

Having recently completed a blind ABX test of radically different amplifiers and not being able to tell any difference it is interesting to see that the reaction of many people that haven't tried ABX testing for them selves is to try to discredit the test method or make up hypotheses about magic stories that need further testing. For me the ABX test was great.... I can stop wasting money and time on "solved problems" like DAC's and Amp's and concentrate on things that do make a difference like speakers and rooms and good recordings. In addition by eliminating worrying about expensive magic that I feared I was missing out on I can just relax and enjoy the music. Intellectually I had accepted blind ABX testing for some time but experiencing it first hand, even though I was not surprised by the results, was quite an eye opening experience.