Blind tests.....

[dweeeeb2]

Stumbled across a scenario that I'd like your feedback / experience on. Brief background:
Looking for differences in DAC's. Used Wiim for streaming, optical out into splitter, optical into 2 dacs, each dac into same amp via different inputs. Volume conveniently appeared matched by ear. 3 DACs on test (each wildly different in price), so rotated around a couple of times. Had my wife connect the DACs so I did not know which input had which DAC.

I have done this routine 2 times now. First time was 3 weeks ago. I believed that generating an accurate analogue signal from a digital source could be quite hard to get right. When I did this scenario back then my expensive DAC won. On a side note it also won when I used a completely different amp/speaker combo (not detailed for brevity but one was amp /speaker, one was HP amp / headphones).

Fast forward 3 weeks, listening to you bunch and studying multiple graphs I am now of the understanding that DACs have pretty much been figured out. Tonight I repeated my tests. This time I hear no obvious difference between all 3 DACs.

In both scenarios I found the result I wanted.

Now I bet the answer is that it's not truly a blind test, and so be it,....but, What is peoples understanding around proving no difference when there actually is no difference? We seem to automatically accept these results.

 

[amirm]

They key is repeating the test at least 10 times and see if you can get at least 8 guesses correctly. This rules out perceived variations that are not durable/reliable.

 

[solderdude]

All 3 DACs will definitely be different in measurements. If only due to not the same chips/reconstruction filters being used and small differences (that are not obvious as level differences) may skew the results.

Also listening fu may differ from day to day.

For checking audible differences nulling recordings of 3 DACs with the same music to the source and each other and listening to the nulls can help determine differences.

Also... you should be using 44.1 material. When using 96kHz or even higher the possible audible differences that may be there (for young ears) at 44.1 may not be there when playing 96kHz files for instance.

 

[dweeeeb2]

They key is repeating the test at least 10 times and see if you can get at least 8 guesses correctly. This rules out perceived variations that are not durable/reliable.

I must admit “correctly” is the hard one when saying “no difference”. I didn’t do 10 tests

 

[BDWoody]

Volume conveniently appeared matched by ear.

You might want to use a multimeter to get those levels closer than you can otherwise.

Can you bribe any family members to do a few trials?

 

[dweeeeb2]

Also listening fu may differ from day to day.

I find this to be very true. Primarily I find some days Im hyper focused on treble, others bass

Also... you should be using 44.1 material. When using 96kHz or even higher the possible audible differences that may be there (for young ears) at 44.1 may not be there when playing 96kHz files for instance.

Thanks, will do

 

[Westsounds]

Blind tests with most if not all modern DACs is almost impossible to tell much of a difference with many, repeated tests, if anything, just confuse the listener even more.

How there are so many lengthy reviews on how DACs sound is just unfathomable.

 

[fpitas]

How there are so many lengthy reviews on how DACs sound is just unfathomable.

Sometimes I suspect people have a monetary incentive.

 

[Westsounds]

Sometimes I suspect people have a monetary incentive.

Unfortunately, that is the truth

 

[fpitas]

Unfortunately, that is the truth

If nothing else, it makes for a crappy review to say: "This one sounds like all the rest"

 

[fpitas]

As for blind tests, I just close my eyes to evaluate changes. Works for me!

 

[Westsounds]

I’ve tested a few recent DACs, ranging costs from £100 to £1000, when level matched as best as possible and unknowingly mixed and connected to separate switchable inputs on an amplifier, it is pretty much impossible to tell any difference!

The BS that reviewers spew out is nothing but storytelling.

 

[Westsounds]

If nothing else, it makes for a crappy review to say: "This one sounds like all the rest"

There are actually a few around that do this They are not as popular funny enough Consumers are their own worst enemy, we are all a bit of a sucker for a good story!

 

[fpitas]

There are actually a few around that do this They are not as popular funny enough Consumers are their own worst enemy, we are all a bit of a sucker for a good story!

Well yes, Amir admits that the good ones sound alike. But he is firmly in the minority. And incurs the wrath of countless Golden Ear sorts.

 

[DVDdoug]

Volume conveniently appeared matched by ear.

If you fail to hear a difference then I'd say "OK". But if you CAN reliably hear a difference it would be better to measure with a multimeter to eliminate that variable. People sometimes describe the louder source as being "more dynamic" or some other characteristic without realizing it's just a simple volume difference, and maybe a very-small volume difference.

I didn’t do 10 tests

ABX tests [link corrected] are "statistical". With just one trial there is a 50/50 chance of getting "X" correct by guessing. It doesn't prove (or "statistically prove") anything. With more repetitions the statistics are more reliable but there is always SOME chance that you were just lucky. (There is never "absolute proof".) If there is a group test, there's a better chance that someone will get statistically better results... That person might have better hearing or maybe they were lucky...

Here is an ABX probability chart. It says if you get 10 of 10 correct there is a 1 out of 1000 chance that you were guessing. If you get 9 of 10 correct there is about a 1% chance that you were guessing. That's a "statistically significant" result but I'd say if you can't hear a difference 100% of the time, the difference isn't THAT obvious and it's probably not worth paying more for the "better" DAC.

...An ABX test isn't telling you which one is better, only if there is an audible difference.

 

[sprellemannen]

I’ve tested a few recent DACs, ranging costs from £100 to £1000, when level matched as best as possible and unknowingly mixed and connected to separate switchable inputs on an amplifier, it is pretty much impossible to tell any difference!

The BS that reviewers spew out is nothing but storytelling.

No surprise you did not hear any difference. Your "problem" is that you are honest and not so affected by bling. If you start listening (testing) to DACs costing £10k or higher (hello dCS, Gryphon, Burmester, ...) you can replace £1000 by that number and your conclusion will be the same.

The "hi-fi society" is indeed weird/crazy. For instance people (and even worse: reviewers) giving detailed comments on how one DAC sounds compared to another one they listened to years ago.

I find this craziness fascinating as well in some way...
For instance people buying the Gryphon Ethos CD-player: It costs approximately £40k and a key person in that company is the sales person Rune Skov who previously worked for Nordost. Many such customers claim huge audible improvements. Are these customers' mind extremely influenced by marketing? or the look of the products?
Similarly regarding extremely expensive cables.

 

[sprellemannen]

An ABX test isn't telling you which one is better, only if there is an audible difference.

If you can not hear a difference, then none of them is better.

Say an ABX test between two products indicates you hear a difference. Then it makes sense to go on testing which one is better.

 

[Westsounds]

No surprise you did not hear any difference. Your "problem" is that you are honest and not so affected by bling. If you start listening (testing) to DACs costing £10k or higher (hello dCS, Gryphon, Burmester, ...) you can replace £1000 by that number and your conclusion will be the same.

The "hi-fi society" is indeed weird/crazy. For instance people (and even worse: reviewers) giving detailed comments on how one DAC sounds compared to another one they listened to years ago.

I find this craziness fascinating as well in some way...
For instance people buying the Gryphon Ethos CD-player: It costs approximately £40k and a key person in that company is the sales person Rune Skov who previously worked for Nordost. Many such customers claim huge audible improvements. Are these customers' mind extremely influenced by marketing? or the look of the products?
Similarly regarding extremely expensive cables.

Yes, it's an absolute con!

 

[sprellemannen]

Yes, it's an absolute con!

Yes. And the hi-fi market is full of it. Must be hard for the few serious companies to "navigate" in the hi-fi market.

 

[sprellemannen]

If nothing else, it makes for a crappy review to say: "This one sounds like all the rest"

For most of the readers, you are right.
I think ASR has done an amazing job in bringing sanity into hi-fi.
A further improvement could have been to categorize (based on the measurements) each product (*) into one out of three categories like "transparent" (perfect performance), "medium" (room for audible improvement, but not terrible) and "terrible".

(*) Excluding loudspeakers and headphones.