• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). There are daily reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

DAC ABX Test Phase 1: Does a SOTA DAC sound the same as a budget DAC if proper controls are put in place? Spoiler: Probably yes. :)

OP
dominikz

dominikz

Senior Member
Forum Donor
Joined
Oct 10, 2020
Messages
487
Likes
1,692
Thank you, Audacity!

If you see a ridiculous result on an online test, make sure to bring the person offline and take the test.

If you take Spectrum Analyzer away from him, his ears will be a fool.
Sure - one can of course cheat at an informal online ABX.
The previous 16/16 result might still be valid - there's no way to know - but we should of course consider that it was the only such score out of a total of 72 (online test) attempts.
Since we have such a high number of recorded attempts by now, it should hopefully be clear that this is a difficult test for most human listeners - this should be the main takeaway IMHO. And it is difficult even though measurements indicate small frequency response differences that could be audible, plus very large differences in distortion levels of the two DACs.

In the end I just hope this casual and informal listening test helps spread the importance of blind, level-matched listening when doing subjective comparisons in audio :)

Note: Since you disclosed you used Audacity to analyze the stream, I won't include this latest result any future overview. Hopefully others trying something similar will be as honest. :)
 

Steve Dallas

Addicted to Fun and Learning
Joined
May 28, 2020
Messages
725
Likes
1,460
Location
A Whole Other Country
Listening with a Modius DAC into a Nova300 amp and BMR speakers, I was sure I could hear a difference in the hi-hats. I got 5 correct. Nope!

This does not surprise me, as I was able to correctly identify the Modius vs. the Nova DAC in a blind test on exactly 1 track that has synthesized crystal sounds in it, which turned out to be the tell. One song only. I failed miserably with all others.

I may try again tomorrow with headphones.

1645294752615.png
 

gomar

Member
Joined
Feb 7, 2022
Messages
6
Likes
7
So basically instead of getting an e50 dac I should just get this tiny box to feed the Topping PA5 into LS50 meta's.
 
OP
dominikz

dominikz

Senior Member
Forum Donor
Joined
Oct 10, 2020
Messages
487
Likes
1,692
So basically instead of getting an e50 dac I should just get this tiny box to feed the Topping PA5 into LS50 meta's.
I wouldn't necessarily recommend that - maximum output level of FiiO D03K (~1,53 Vrms at 1kHz with 0dBFS peak) would not be enough to drive PA5 to maximum output (PA5 can take up to 2.6Vrms input), and also FiiO D03K seems to have pretty high output impedance, making it somewhat load-sensitive (this would not be a problem if the amp has reasonably high input impedance - don't know the value for PA5).

But you don't necessarily need E50 either for transparent reproduction. :)
 

gomar

Member
Joined
Feb 7, 2022
Messages
6
Likes
7
I wouldn't necessarily recommend that - maximum output level of FiiO D03K (~1,53 Vrms at 1kHz with 0dBFS peak) would not be enough to drive PA5 to maximum output (PA5 can take up to 2.6Vrms input), and also FiiO D03K seems to have pretty high output impedance, making it somewhat load-sensitive (this would not be a problem if the amp has reasonably high input impedance - don't know the value for PA5).

But you don't necessarily need E50 either for transparent reproduction. :)
Hmm, I might pull the D90 from my desk to do the job in that case, I'm not very well versed in the impedance technicalities. 1.5Vrms seems a bit low indeed although that way the topping PA5 will not reach it's output levels where the distortion percentage goes up. I could use the output of an apple dongle into my A90 for my desk setup. The A90 has ridiculous amounts of power to amp up the 0.5v of the apple dongle (EU version).
 
OP
dominikz

dominikz

Senior Member
Forum Donor
Joined
Oct 10, 2020
Messages
487
Likes
1,692
Hmm, I might pull the D90 from my desk to do the job in that case, I'm not very well versed in the impedance technicalities. 1.5Vrms seems a bit low indeed although that way the topping PA5 will not reach it's output levels where the distortion percentage goes up. I could use the output of an apple dongle into my A90 for my desk setup. The A90 has ridiculous amounts of power to amp up the 0.5v of the apple dongle (EU version).
If your DAC output only goes up to 1,5Vrms and the amp can take up to 2.6Vrms your maximum SPL will be about 4,5dB lower than with a DAC that can produce 2.6Vrms.

Regarding impedance you can see here that PA5 should have 10kOhm input impedance (probably with balanced input), so you don't need to worry - that should be high enough for any reasonable DAC. :)
 

Steve Dallas

Addicted to Fun and Learning
Joined
May 28, 2020
Messages
725
Likes
1,460
Location
A Whole Other Country
I wouldn't necessarily recommend that - maximum output level of FiiO D03K (~1,53 Vrms at 1kHz with 0dBFS peak) would not be enough to drive PA5 to maximum output (PA5 can take up to 2.6Vrms input), and also FiiO D03K seems to have pretty high output impedance, making it somewhat load-sensitive (this would not be a problem if the amp has reasonably high input impedance - don't know the value for PA5).

But you don't necessarily need E50 either for transparent reproduction. :)

Right. Any of the objectively good $100 DACs will be sufficient for transparent audio.
 

Kegemusha

Senior Member
Joined
Jul 16, 2020
Messages
319
Likes
313
Nice test, I did it on my modest desktop setup (old dacmagic 100 DAC to a BasX emotiva amp and sony 7506 with EQ), was difficult to hear any difference, you really need to "hear", I guess when listening while working or doing some other things I would never hear any difference.
Capture.JPG
 

Grooved

Senior Member
Joined
Feb 26, 2021
Messages
423
Likes
264
@dominikz
Hi, just found your thread, very interesting.
Downloaded both files, didn't have time to do a full test but both look really similar at first. I will try later.

But at this moment, even I think it will be very hard to hear something, I find a problem taking a real conclusion with this test, and it's not the devices used or the process, it's the track selection.
As I already saw in AAC/FLAC tests (or MQA/FLAC), there are tracks where you can hear a difference, and there are other where you can't, and even among tracks you perfectly know. So with a track you're just discovering and that may not be the best to hear a difference...

To really validate this test, but it would be painful for you, each person should send you a track they want to use for the test and let you do both loopback recordings.
It's the only possibility that I see to start taking conclusion of anything.
 
Last edited:
OP
dominikz

dominikz

Senior Member
Forum Donor
Joined
Oct 10, 2020
Messages
487
Likes
1,692
@dominikz
Hi, just found your thread, very interesting.
Downloaded both files, didn't have time to do a full test but both look really similar at first. I will try later.

But at this moment, even I think it will be very hard to hear something, I find a problem with taking conclusion with this test, and it's not the devices used or the process, it's on the track selection.
As I already saw in AAC/FLAC tests (or MQA/FLAC), there are tracks where you can hear a difference, and there are other where you can't, and even among tracks you perfectly know. So with a track you're just discovering and that may not be the best to hear a difference...

To really validate this test, but it would be painful for you, each person should send you a track they want to use for the test and let you do both loopback recordings.
It's the only possibility that I see at least take conclusion of anything.
Thanks for your comments!

To be honest, I'm not really looking to make any conclusions - IMHO there's not a lot of new insights to be gained here. :)
Personally I think it is very indicative that the frequency response differences between these two DACs should be audible in certain cases and to some listeners.
But based on the results so far I hope it is also evident that we're surely not talking about night-and-day audible differences - as one might otherwise expect looking at the measured performance and difference in price. Speaking about audibility, even in this case realistically we're talking about tiny differences that some people should be able to detect under special circumstances. And what if instead we compare two solid performing DACs...? ;)

The track selection has been primarily motivated by the fact that I have distribution rights for this piece of music and which I have the original 24bit digital master file for (which was the source for this test).

IMHO it should also be an OK track for identifying FR differences as it has a pretty rich spectrum content (mainly due to distorted guitars and keys). This is based on research (quoted also in Toole's book) which found that a rich spectrum (such as in e.g. rock or modern pop music) is beneficial for identifying frequency response differences. That is probably because there is statistically more chance in that case for the sound to be affected by the frequency response deviation.
Note how pink noise is usually the most revealing of FR anomalies, and music with a lot of distortion (and therefore rich spectrum) will formally be more similar to pink noise than very 'clean' music. It is even said that well-recorded music is not necessarily the most revealing of deficiencies in the reproduction chain. :)

As far as dynamic range and distortion audibility is concerned, I agree that this track is probably not ideal - if nothing else, the large amount of distortion in the instruments is almost certainly masking the <-90dB distortion products of the DACs.

To conclude - even if this is not an ideal test, I still hope it is a useful exercise. :)
Probably a lot of people never tried ABX or to level-match to this level of precision, and I hope this test might pique their curiosity.
Note that by default Topping E50 output in unbalanced mode is ~2.7dB louder than FiiO D03K (both at 0dBFS) - that is sure to bias any uncontrolled listening evaluation towards E50!
 
Last edited:

Steve Dallas

Addicted to Fun and Learning
Joined
May 28, 2020
Messages
725
Likes
1,460
Location
A Whole Other Country
Thanks for your comments!

To be honest, I'm not really looking to make any conclusions - IMHO there's not a lot of new insights to be gained here. :)
Personally I think it is very indicative that the frequency response differences between these two DACs should be audible in certain cases and to some listeners.
But based on the results so far I hope it is also evident that we're surely not talking about night-and-day audible differences - as one might otherwise expect looking at the measured performance and difference in price. Speaking about audibility, even in this case realistically we're talking about tiny differences that some people should be able to detect under special circumstances. And what if instead we compare two solid performing DACs...? ;)

The track selection has been primarily motivated by the fact that I have distribution rights for this piece of music and which I have the original 24bit digital master file for (which was the source for this test).

IMHO it should also be an OK track for identifying FR differences as it has a pretty rich spectrum content (mainly due to distorted guitars and keys). This is based on research (quoted also in Toole's book) which found that a rich spectrum (such as in e.g. rock or modern pop music) is beneficial for identifying frequency response differences. That is probably because there is statistically more chance in that case for the sound to be affected by the frequency response deviation.
Note how pink noise is usually the most revealing of FR anomalies, and music with a lot of distortion (and therefore rich spectrum) will formally be more similar to pink noise than very 'clean' music. It is even said that well-recorded music is not necessarily the most revealing of deficiencies in the reproduction chain. :)

As far as dynamic range and distortion audibility is concerned, I agree that this track is probably not ideal - if nothing else, the large amount of distortion in the instruments is almost certainly masking the <-90dB distortion products of the DACs.

To conclude - even if this is not an ideal test, I still hope it is a useful exercise. :)
Probably a lot of people never tried to level-match to this level of precision and I hope this test might pique their curiosity.
Note that by default Topping E50 output in unbalanced mode is ~2.7dB louder than FiiO D03K (both at 0dBFS) - that is sure to bias any uncontrolled listening evaluation towards E50!

This has been a very interesting experience. Thank you for putting it together. I always think I can hear a difference, and the results always show I cannot.

The ONLY time I have scored above the guessing range, let alone a 10 out of 10, in a blind test between DACs is when this song was played. After 1:10, synthesized crystal sounds enter the track. I could recognize one DAC instantly when I heard them. For lack of a better description, they sounded "sharper" or "more distinct" on that DAC. Notice I did not say "better". Just different enough to tell the DACs apart 10 out of 10 times. In the end, I could not honestly call either DAC better. They were 99.5% the same, and I just got lucky in this track being included.


I want to run your test again with good headphones, but I do not care for the program material, and am not sure I am ready for a 2nd round of torture.
 

Grooved

Senior Member
Joined
Feb 26, 2021
Messages
423
Likes
264
Thanks for your comments!

To be honest, I'm not really looking to make any conclusions - IMHO there's not a lot of new insights to be gained here. :)
Personally I think it is very indicative that the frequency response differences between these two DACs should be audible in certain cases and to some listeners.
But based on the results so far I hope it is also evident that we're surely not talking about night-and-day audible differences - as one might otherwise expect looking at the measured performance and difference in price. Speaking about audibility, even in this case realistically we're talking about tiny differences that some people should be able to detect under special circumstances. And what if instead we compare two solid performing DACs...? ;)

The track selection has been primarily motivated by the fact that I have distribution rights for this piece of music and which I have the original 24bit digital master file for (which was the source for this test).

IMHO it should also be an OK track for identifying FR differences as it has a pretty rich spectrum content (mainly due to distorted guitars and keys). This is based on research (quoted also in Toole's book) which found that a rich spectrum (such as in e.g. rock or modern pop music) is beneficial for identifying frequency response differences. That is probably because there is statistically more chance in that case for the sound to be affected by the frequency response deviation.
Note how pink noise is usually the most revealing of FR anomalies, and music with a lot of distortion (and therefore rich spectrum) will formally be more similar to pink noise than very 'clean' music. It is even said that well-recorded music is not necessarily the most revealing of deficiencies in the reproduction chain. :)

As far as dynamic range and distortion audibility is concerned, I agree that this track is probably not ideal - if nothing else, the large amount of distortion in the instruments is almost certainly masking the <-90dB distortion products of the DACs.

To conclude - even if this is not an ideal test, I still hope it is a useful exercise. :)
Probably a lot of people never tried ABX or to level-match to this level of precision, and I hope this test might pique their curiosity.
Note that by default Topping E50 output in unbalanced mode is ~2.7dB louder than FiiO D03K (both at 0dBFS) - that is sure to bias any uncontrolled listening evaluation towards E50!

For sure, it's useful and already very informative. I thought you picked it because of rights over it ;)
I should have quoted one comment asking if the conclusion would make you buy the cheapest one instead of the other, because I think more test would be made with other tracks to be a bit more sure.
 
OP
dominikz

dominikz

Senior Member
Forum Donor
Joined
Oct 10, 2020
Messages
487
Likes
1,692
For sure, it's useful and already very informative. I thought you picked it because of rights over it ;)
I should have quoted one comment asking if the conclusion would make you buy the cheapest one instead of the other, because I think more test would be made with other tracks to be a bit more sure.
Actually I just recently bought the E50 (based on ASR measurements) for a specific music production use-case where I need very high SNR (high-gain guitar reamping). This can easily bring DAC noise-floor up to audible levels, even with pretty good DACs :)
I'm also using the E50 for audio measurements as a signal generator.
For these two use-cases it is really amazing, and significantly better than e.g. my RME Babyface (1st gen).

For simple music listening I have much more humble DAC performance demands. :D
 

dasdoing

Major Contributor
Joined
May 20, 2020
Messages
2,495
Likes
1,380
Location
Salvador-Bahia-Brasil
As I already saw in AAC/FLAC tests (or MQA/FLAC), there are tracks where you can hear a difference, and there are other where you can't
imo if you have to desperatly search for a situation/track/circumstance where you can hear a diference, there is efectivly none. even if you after hours of training and trail and error found something, it proves nothing since you would have never noticed it in real world listening. that's why I refuse to go too deep into abx tests. or I hear it fast, or I give up.
unfortunatly some guys think abx-ing is a courtship display.
 

Grooved

Senior Member
Joined
Feb 26, 2021
Messages
423
Likes
264
imo if you have to desperatly search for a situation/track/circumstance where you can hear a diference, there is efectivly none. even if you after hours of training and trail and error found something, it proves nothing since you would have never noticed it in real world listening. that's why I refuse to go too deep into abx tests. or I hear it fast, or I give up.
unfortunatly some guys think abx-ing is a courtship display.
I agree completely if you had to search a lot, it's like having no difference it's not like you can't enjoy listening to both versions.
Now, I didn't specify in my post that I was searching for a long time. It happened to find the difference in the first two seconds (depending on the content), I knew which one it was, and not specially in ABX, but just like listening in Roon where I saved in a playlist some tracks that were with two versions, and Roon playing it randomly. I'd think that, sometimes, when a track start and you think it's A version and click on the path spec to check the version and it was always the correct one, it means at least a bit, because you're not even comparing with the B version.
Now, thanks, it's not an obsession and I didn't want to check every time I was listening, because both versions was enjoyable as long as the song is good :)
 

yanm

Member
Forum Donor
Joined
Feb 10, 2021
Messages
50
Likes
41
Location
Switzerland
Another thing to consider is that, as far as I know, this kind of ABX test relies on our short term memory (20 seconds or so). It is thus key to be able to play the same short snippets back-to-back, which the internet tool does not easily allow unfortunately.

Ideally, and in my opinion, one should be presented with tens or hundreds of short snippets pairs and try to guess if they are the same or not (rather than being presented with a long song). If enough tests are taken, critical song portions could even be determined (critical = portions where distortions are making a audibly difference). Publishing only short-enough snippets may even help with copyright infringement law… but I’m no lawyer.
 
OP
dominikz

dominikz

Senior Member
Forum Donor
Joined
Oct 10, 2020
Messages
487
Likes
1,692
Another thing to consider is that, as far as I know, this kind of ABX test relies on our short term memory (20 seconds or so). It is thus key to be able to play the same short snippets back-to-back, which the internet tool does not easily allow unfortunately.

Indeed - this has been discussed a few times already in the thread, e.g. I mentioned it post #11:
Perhaps the only part missing is the ability to loop just a specific section of the test track (rather than looping the whole thing).

The author of the tool gave us a hint in post #20 that there is however a chance that this gets included in the future :):

In the meantime you can use the source audio files from post #1 and foobar2000 ABX comparator plugin which provides this functionality:
[EDIT 2021-12-29] For those wanting to use foobar2000 (in WASAPI exclusive mode) with ABX comparator plugin (16 trials suggested), here are direct links to the audio test files:
1. Topping E50 ABX sample file
2.
FiiO Taishan D03K ABX sample file
If you do a test please report your results (copy/paste of ABX comparator result output) by posting in this thread or via a private message. Thanks! :)
 
Last edited:

yanm

Member
Forum Donor
Joined
Feb 10, 2021
Messages
50
Likes
41
Location
Switzerland
Indeed - this has been discussed a few times already in the thread, e.g. I mentioned it post #11:


The author of the tool gave us a hint in post #20 that there is however a chance that this gets included in the future :):


In the meantime you can use the source audio files from post #1 and foobar2000 ABX comparator plugin which provides this functionality:
Thanks for the comprehensive reply.

I think that the ability to loop back to a given section is a step in the good direction. Human short term memory is however pretty poor (see e.g., Here). If one needs to select where to loop back at each repetition, the memory of the fine details of what was listened to may be lost. Also, this implies that the person taking the test knows, which snippets should be listen to detect possible difference. In other words, two tests are compounded: 1) the knowledge of where differences may arise (or the will to search for them), and 2) the ability to detect difference.

In my opinion, but I am no specialist though, it would be better if the test consisted of many very short snippets (randomly) selected along the song. There are less risks to alter the short term memory as the person taking the test does not need to act or set anything for the looping back.

Also, one of the points for selecting this song, was that you’re the copyright holder, right? Do you know, what are the copyright laws regarding sharing only short snippets of copyrighted material? My point, is that sharing very short material may allow to be use a larger selection of songs and styles… of course, it may makes more complicated for people that insist on downloadable copy of test material.

If you do a follow-up test, it would be also interesting to ask the people taking the test about the disortion level of their transducer (if they know). I am suspecting, that‘s actually where the limit is: if the used transducer is, say, distorting at level of 1% it would likely mask the distortion of the lower-performing DAC… for example, my loudspeakers are in the low 0.x%, which most likely already masks the DAC distortion.

Anyway, very interesting thread. Thanks @dominikz , very good initiative.

( unfortunately foobar2000 does not work for me as I’m on MacOS - also I would need to AirPlay to the E50 via the node, which may further alter things a bit )
 

flz

Active Member
Joined
Feb 27, 2022
Messages
154
Likes
115
I also could not tell the difference (9 out of 16 right). In another ABX test, I could not tell the difference between lossy & lossless. In my home hifi, I can't tell the difference between AAC 256 streamed over bluetooth (AudioEngine B1) and original CDs from my CD player. All of these suggest I have little or no reason to upgrade my DAC. Yet, there is a slight temptation which I'm so far resisting.
 
Top Bottom