DAC listening test files - update

[rdaow]

I was not able to hear any difference in the 6J1 vs Topping test
I would be interested in the mp3 128kbps vs wav, I was able to hear that easily when I was younger. Did not try it recently.

 

[Salt]

Have to confess, even with this second pair of music (that is much more convenient to me ), can't hear any significant difference ... but did not dig into hours of listening to really attempt an ABX.
So far, these are very insightful experiences, that can be followed by everyone, so thanks a lot for Your efforts.

 

[RandomEar]

6J1 pre-amp in loop: 8/16 (p = 59.8%). Clearly, night and day differences!

D10s vs. DacMagic+: 10/16 (p = 22.7%). After getting 6/6 in training mode and being quite certain that there was something... Well, if you combine the logged run with training, it would be 16/22 (p = 2.62%). But of course anybody could say that they aced the training. Maybe I'll try this test again at some point.

 

[krabapple]

I also purposefully didn't search for very low amplitude moments in the test signal, where noise floor or low amplitude conversion artifacts could be present. If you have to wait for the last few seconds of a fade-out or reverb tail to distinguish two pieces of equipment, I don't believe that can be pointed out as a timbre difference, considering the musical program material humans use audio equipment for. Unless that low level artifact clearly sounds worse or less faithful, that's in the realm of testing lab equipment in my opinion.

Thank you! This has always been my response to people whom claim that 16 bits sound different than 24 or 32.
When you drill down, the only time they can provably do it, is by very carefully listening to tails, played loud, usually on headphones (and the material usually wasn't properly dithered)

It's an audio parlor trick.

That means, "No, Mr. GoldenEar, you do not hear a difference in your normal listening to music, much less 'right away'. And neither does your wife."

 

[Holy Spirit]

BTW, back in 2009 I think, I bought CS4398 DAC, removed output filter and installed a cathode follower based on 6N6P. And of course I have heard a "massive improvement".

 

[pma]

@Josh83 , he got 9/10 in the dactest, very good!!!

Would you like to try the tubepreamp test as well?

/I am on vacations now, so not too many comments, however I can see that some AS members continue in “predictions” what I would say. Is it clever behaviour?/

 

[welwynnick]

@Josh83 , he got 9/10 in the dac test, very good!!!

That's a significant result, not just for the p value but because Josh was so reluctant to test anything with an additional ADC & DAC in the loop, the reasoning being that if you're listening for the differences between DACs, then surely two additional conversion processes are going to mask what you're looking for.

It suggests that the audible differences aren't caused by the conversion process itself (noise, distortion etc), but can be detected by a good ADC.
IIRC the ADC was run at a high sample rate, which points towards frequency domain artefacts - bandwidth, HF roll-off, frequency non-linearity, maybe phase non-linearity.

Nonetheless it's a slightly unexpected but very interesting outcome because:

• It's good evidence that good DACs sound different (in conjunction with the following)
• DAC differences are audible even through an ADC - DAC loop.
• The ADC - DAC loop is sufficiently transparent.
• ABX testing is possible without a hardware comparator.
• The files can be shared which helps to verify the test.
• This result will encourage other people to listen for DAC differences.

 

[pjug]

@Josh83 , he got 9/10 in the dactest, very good!!!

Would you like to try the tubepreamp test as well?

/I am on vacations now, so not too many comments, however I can see that some AS members continue in “predictions” what I would say. Is it clever behaviour?/

Yes, it is great that someone can pass that test.

edit: scratch what I said earlier this morning. I was looking at statistics of the wrong pair. This pair looks like another case where passband ripple is the big difference. However, these are 44.1K files so not completely within the spirit of Josh's article where 96K files were used to lessen reconstruction filter effects. Still, we can EQ out the passband ripple and see if that is what @Josh83 is hearing, if he would be interested in ABX on that.

 

[RandomEar]

@Josh83 , he got 9/10 in the dactest, very good!!!

Would you like to try the tubepreamp test as well?

/I am on vacations now, so not too many comments, however I can see that some AS members continue in “predictions” what I would say. Is it clever behaviour?/

Just to add the info: The log is in this post. The test was done using the Topping D10s vs. DacMagic+files. I have checked the signature using the foobar2000 tool and it appears valid. Would have been nice to have more than 10 trials, though.

 

[RandomEar]

That's a significant result, not just for the p value but because Josh was so reluctant to test anything with an additional ADC & DAC in the loop, the reasoning being that if you're listening for the differences between DACs, then surely two additional conversion processes are going to mask what you're looking for.

It suggests that the audible differences aren't caused by the conversion process itself (noise, distortion etc), but can be detected by a good ADC.
IIRC the ADC was run at a high sample rate, which points towards frequency domain artefacts - bandwidth, HF roll-off, frequency non-linearity, maybe phase non-linearity.

Nonetheless it's a slightly unexpected but very interesting outcome because:

• It's good evidence that good DACs sound different (in conjunction with the following)
• DAC differences are audible even through an ADC - DAC loop.
• The ADC - DAC loop is sufficiently transparent.
• ABX testing is possible without a hardware comparator.
• The files can be shared which helps to verify the test.
• This result will encourage other people to listen for DAC differences.

I think this is way, way too much interpretation for such a short test and without additional verification.

 

[NTTY]

I have asked a good friend of mine, whom I can trust not to cheat Foobar ABX results (now that it is published how to), to give a try to the first test shared by you @pma. I was happy he accepted because in my past ABX tests, he was the one to constantly perform much better than the others, hearing 100ns of Jitter or 0.4% of distortion into music for instance.
He scored 9/16 on your test first test, so he can't differentiate the two files.
Cheers

 

[welwynnick]

I think this is way, way too much interpretation for such a short test and without additional verification.

That's a fair comment (though I hope you appreciated that I deliberately avoided using the word "proof").
However, there does seem to be a pattern emerging:

On Nov 13 JM reported he scored 8/8 comparing the Matrix X-SABRE Pro with the Topping D900 when using the AVA ABX comparator.
On Dec 02 JM reported he scored 8/8 comparing the Gustard R30 with the Yggdrasil LIM when using the AVA ABX comparator.
On Jul 21 JM reported he scored 7/8 comparing the Topping D900 with the RME ADI-2 FS when using the AVA ABX comparator, and others.
On Aug 08 JM reported he scored 8/10 comparing DAC > ADC > DAC with just a DAC using Foobar2000 ABX.
On Aug 19 JM reported he scored 9/10 comparing Topping D10s with DacMagic using PMA's loopback files using Foobar2000 ABX.

Each test might not be convincing in isolation, as PMA often says, but the evidence is starting to mount up.

 

[Thomas_A]

I do not understand why Josh use a 10-trial test when he nailed all 16 trials for 5 DAC combinations. 16/16 implies a chance of success of 0.002% while 9/10 is 1.07%. Significant below 0.05 yes but a huge difference to 16/16 for five pair of DACs.

 

[RandomEar]

That's a fair comment (though I hope you appreciated that I deliberately avoided using the word "proof").
However, there does seem to be a pattern emerging:

On Nov 13 JM reported he scored 8/8 comparing the Matrix X-SABRE Pro with the Topping D900 when using the AVA ABX comparator.
On Dec 02 JM reported he scored 8/8 comparing the Gustard R30 with the Yggdrasil LIM when using the AVA ABX comparator.
On Jul 21 JM reported he scored 7/8 comparing the Topping D900 with the RME ADI-2 FS when using the AVA ABX comparator, and others.
On Aug 08 JM reported he scored 8/10 comparing DAC > ADC > DAC with just a DAC using Foobar2000 ABX.
On Aug 19 JM reported he scored 9/10 comparing Topping D10s with DacMagic using PMA's loopback files using Foobar2000 ABX.

Each test might not be convincing in isolation, as PMA often says, but the evidence is starting to mount up.

I would not consider the comparator results until the device has been measured by Amir. And as shown by the results of @NTTY in this thread, you can easily get 8/10 by clicking randomly. It may be a valid result, but the uncertainty is higher than the p-value itself might suggest. As criticized by @pma in this post, 10 runs is just a bit too few. You can also not combine results from totally different tests with different devices without some statistical trickery.

Maybe it will all turn out to be as you suggest in the end, maybe it won't. The uncertainty is still pretty high right now.

Apart from that, the statement "the audible differences aren't caused by the conversion process itself" sounds weird to me. The conversion process is what produces the sound, so if you assume audible differences do exist, they must happen in that process by logical exclusion.

 

[RandomEar]

@pma Do you happen to remember which filter was active on the DacMagic+ you used to capture the ABX test files? Because it looks like the minimum phase filter in that DAC can cause clipping:

 

[welwynnick]

Maybe it will all turn out to be as you suggest in the end, maybe it won't. The uncertainty is still pretty high right now.

There's lots of evidence yet to come, and I trust that you haven't already decided what the outcome will be. The journey is nowhere near over.

Apart from that, the statement "the audible differences aren't caused by the conversion process itself" sounds weird to me. The conversion process is what produces the sound, so if you assume audible differences do exist, they must happen in that process by logical exclusion.

I was mindful of the multi-loop testing done by AP Mastering where he demonstrated how little degradation there was after 1500 loopbacks through a high quality PrismSound Lyra 2 audio interface. That suggests DA conversion is highly transparent, so how to reconcile that with the possible audibility of a single conversion stage? Of course the noise and distortion must eventually add up. They're the consequences of each pass through the conversion process, but they're obviously very low. As long as a linear phase filter is used, the effects of the filter don't add up. The filtering is done once, then there is no more effect. Therefore it seems plausible to me that what's being heard are differences in frequency and phase response between different filters in different DACs. That would be consistent with GoldenSounds ABX test as well.

Alternatively Amir finds a problem with the AVA ABX box, and we're back to stage 1.

 

[pma]

@pma Do you happen to remember which filter was active on the DacMagic+ you used to capture the ABX test files? Because it looks like the minimum phase filter in that DAC can cause clipping:

I do not think there was any clipping. But yes, the biggest difference between D10s and DM+ was/is in filter’s amplitude/phase response. I would say that it was also shown in this thread, earlier. The DeltaWave difference delta spectrum would show it clearly.

Sorry, my mistake. I need to check the files after I return to my lab, next week.

 

[pjug]

Here is the delta of the spectrum. There is passband ripple and a gradual slope. If you EQ these out vthe matching is better than -110dB PK metric. So it seems it is more about amplitude than phase (in terms of what is seen in DeltaWave).

I should say that identifying this does not make me any less impressed with @Josh83 's ability to ABX the files!

Oh, @pma beat me to showing the graph. Or rather it looks like @pma showed a similar one from 96K files. Just to be clear, Josh's 9/10 ABX was with the 44.1K files, correct?

 

[RandomEar]

There's lots of evidence yet to come, and I trust that you haven't already decided what the outcome will be. The journey is nowhere near over.

I'm cautious, sceptical and want to see solid data and verified results. Neither me nor you can predict the future and it certainly seems ill advised to make judgements now based on hypothetical evidence you expect to come out at some point. Let's see your ABX results!

 

[pma]

Here is the delta of the spectrum. There is passband ripple and a gradual slope. If you EQ these out vthe matching is better than -110dB PK metric. So it seems it is more about amplitude than phase (in terms of what is seen in DeltaWave).

I should say that identifying this does not make me any less impressed with @Josh83 's ability to ABX the files!

Oh, @pma beat me to showing the graph. Or rather it looks like @pma showed a similar one from 96K files. Just to be clear, Josh's 9/10 ABX was with the 44.1K files, correct?

Yeah, you are right. Josh has taken the 2_DAC_test, in 44.1kHz. 44.1kHz/24bit. It is in the post #1 as part IV. I am sorry for confusion, I am on vacations with my iPad only, no test database. Please forget my plot, it was from another test.

But for sure there was no clipping. Peak value was -8dBFS.