DAC ABX Test Phase 1: Does a SOTA DAC sound the same as a budget DAC if proper controls are put in place? Spoiler: Probably yes. :)

[dominikz]

Confirmed.
I also note the professional trimming and level matching, thanks @dominikz for the almost flawless preparation and execution of the test. That's the level of thoroughness we need here.

Care to share the original snippet as well?

Thanks for the kind words

I have to say I find the music choice unsatisfiyng for these kinds of tests. Not a very nice recording to begin with, very bland contemporary pop sound production.

Sorry you didn't like the track, but thanks anyway for listening and commenting

3) Is the main downside for me. Roadblocks of this type quickly lead to fatigue and demotivation.

One point to mention :
4) Crossfade is normally forbidden for ABX as that may easily introduce phasing which can be a false clue even when the clocks are very close to each other.... if they are not, timing difference will be an ever greater false cue.
A silence gap is very annoying on the other hand. Crossfade through level-tracking noise would be the best option to provide better auditory stream continuity and resolve the phasing issue when the low clock offset condition is met.

Thanks for your thoughts, perhaps this might also be interesting feedback for @jaakkopasanen.

I've checked my loopback-recorded files and clock drift and notably clock offset is really surprisingly small (subsample regions) and therefore in this case the phasing is minimal but noticable when you know what to listen for and the x-fade happens to be at points in the files where the phasing is maximum.

Interesting!

Other than that, the measurements you posted contain a clear pointer what to listen for to start with... I suggest disclosing it as the goal of an ABX is maximum sensitivity which is readily increased with prior knowledge about the obvious differences.

Do you mean the D03K frequency response variations? If so, I disclosed that already in post #1:

Though I consider this to be a difficult test to most people, I expect that those with well-preserved high frequency hearing (>10kHz) might still be able to hear a difference between the two DACs due to differences in their frequency response. Note that FiiO Taishan D03K has a slight sub-LF loss and a small top-octave peak that should result in a slightly brighter sound than the Topping E50, which could then be used to identify the "X".

 

[KSTR]

Do you mean the D03K frequency response variations? If so, I disclosed that already in post #1:

Exactly. A gradual boost up to 0.7dB at 17kHz, starting at below 10kHz should be good enough for a significant clue with cymbals etc... when you can hear well up that high (I can't anymore but will try ABXing anyway in the coming days).

Assuming we get corresponding results, then downstream in the process we could verify if that FR clue is dominant (by emulating it directly on the source file) and we could also verify if other clues than FR might be dominant here, by un-embedding the FR difference (making FR -- mag and phase -- the same for both DACs). So that basically noise/distortion/jitter/whatever might contribute and not simple (and reversible) frequency response changes.

 

[dominikz]

Exactly. A gradual boost up to 0.7dB at 17kHz, starting at below 10kHz should be good enough for a significant clue with cymbals etc... when you can hear well up that high (I can't anymore but will try ABXing anyway in the coming days).

Indeed, I agree - this should IMHO be audible to people with well-preserved high-frequency hearing.
My HF hearing seems to end somewhere between 16-17kHz but I still couldn't hear this HF response difference and therefore failed the ABX - however that might also be due to my critical listening skill level (or possibly lack thereof ).

Assuming we get corresponding results, then downstream in the process we could verify if that FR clue is dominant (by emulating it directly on the source file) and we could also verify if other clues than FR might be dominant here, by un-embedding the FR difference (making FR -- mag and phase -- the same for both DACs). So that basically noise/distortion/jitter/whatever might contribute and not simple (and reversible) frequency response changes.

Good ideas, that would definitely be interesting to try!

I have downloaded the songs and I can't do better than guessing using foobar2000. : )

Thanks for reporting back! Perhaps you could take some comfort in the fact that you are not alone At least I cannot do better than guessing in this test, and seems this might also be true for most other participants so far (but not all).
As stated from the beginning, IMHO this is a pretty difficult test, even if it may not seem so from the outset. The very precise level-matching and lack of obvious distortions is what makes it difficult.

Can you do another one with strong sub bass (with lots of ~30hz or lower tones)? Like the one in https://www.audiosciencereview.com/forum/index.php?threads/bass.18999/post-1017710, https://www.audiosciencereview.com/forum/index.php?threads/bass.18999/post-924704, or other strong bass song in the BASS! thread?

Thanks!

TBH I wouldn't want to publicly post music files with content that I have no legal rights to distribute.
Also there may not be enough hours in a day to finish all of the little projects that are already in my buffer
Hope you understand!

 

[Pdxwayne]

Indeed, I agree - this should IMHO be audible to people with well-preserved high-frequency hearing.
My HF hearing seems to end somewhere between 16-17kHz but I still couldn't hear this HF response difference and therefore failed the ABX - however that might also be due to my critical listening skill level (or possibly lack thereof ).


Good ideas, that would definitely be interesting to try!


Thanks for reporting back! Perhaps you could take some comfort in the fact that you are not alone At least I cannot do better than guessing in this test, and seems this might also be true for most other participants so far (but not all).
As stated from the beginning, IMHO this is a pretty difficult test, even if it may not seem so from the outset. The very precise level-matching and lack of obvious distortions is what makes it difficult.


TBH I wouldn't want to publicly post music files with content that I have no legal rights to distribute.
Also there may not be enough hours in a day to finish all of the little projects that are already in my buffer
Hope you understand!

No worries.

From post #1's charts, it seems if I am to hear a different, it would be with songs with strong bass near 20 to 40 Hz. I can't really hear pass 15khz anyway. Still fun to try. Thanks!

 

[GaryH]

Since the cat's anyway out of the bag already let me start by providing links to original files for offline use with foobar2000 ABX comparator (as requested by @KSTR and @Pdxwayne as well):

1. Topping E50 sample file
2. FiiO Taishan D03K sample file

Thanks.

To be fair, there are currently really only 3 criticisms regarding abxtests.com tool as far as I can tell:
1) The streamed sound quality of the tool may be compromised with certain end-user setups and systems, since these are out of control of the test. This is of course a valid concern, but IMHO doesn't necessarily make the test invalid. Note that foobar2000 ABX comparator is also not immune to similar type of criticsm. More on this below.
2) The tool calculates p-value based on P(X=x) instead of P(X>=x). This alone doesn't make the test results themselves invalid, as we can still calculate P(X>=x) from the same test data.
3) The tool doesn't let you seek or select a smaller subset of the track to loop. While I'd love for the tool to have this functionality, it is an added feature and lack of such functionality doesn't imply invalid ABX methodology. It just makes it more difficult to identify differences.

These are big issues.

To clarify, I have nothing against foobar2000 ABX comparator and my motivation for using the abxtests.com tool instead was:
- Hope that it would make the test more accessible and easier to use

Except several users have reported bugs / it not working for them. Foobar ABX Comparator does not have these issues, and is known to be bit-perfect in WASAPI exclusive mode.

Note that we could make a similar argument that some participants might not use good enough transducers, don't have correct gain-staging, don't apply room correction with loudspeakers etc...

I have made precisely that argument before. That's why every time anyone posts an ABX log they should also state what their playback chain and conditions are, so these can be checked if needed (e.g. looking at components' measured performance on here) for potential confounding variables.

Given that this thread is not meant to be a scientific article, I hope we can live with some level of uncertainty and still find some value in the data provided

I see this type of argument on here so many times, and it's a poor one. Science literally means knowledge. Knowledge is justified true belief. A belief is an assertion of truth. So logically everyone must want their belief-forming processes to reliably lead them to truth. We know from hundreds of years of experience that the scientific method is the most reliable belief-forming process we have. So it should, and can, be used to form all our beliefs in every area of life, whether we are a scientist or not, whether we're writing an academic paper or a forum post. That's the beauty of science, anyone can do it. If you don't maintain basic scientific principles, you can end up forming false conclusions that are then just repeated by others and misinformation is spread, which happens all too often these days at speed courtesy of the internet and a worrying lack of scientific literacy.

 

[pkane]

That's the beauty of science, anyone can do it. If you don't maintain basic scientific principles, you can end up forming false conclusions that are then just repeated by others and misinformation is spread, which happens all too often these days at speed courtesy of the internet and a worrying lack of scientific literacy.

Well, it's a good thing, then, that every single other post on ASR conforms to the scientific principles and follows scientific process to the letter!

Let's give Dominik a chance, as he's already doing a better job of running a listening blind test than most, and appears to be willing to make changes and corrections as issues are raised.

 

[jaakkopasanen]

@jaakkopasanen, IMHO some more unit tests will provide more confidence in the tool's results, specially considering that you have implemented all probability functions from scratch. You know, TDD .

Happy to review and accept a PR implementing the tests

 

[xaviescacs]

Happy to review and accept a PR implementing the tests

If I were fluent in js and react I would be certainly glad to contribute. I'm proficient in R and more or less capable in Python, but my js is at survival level. Let me know if I can help you in any way, it's a nice project.

Edit: Well, the test are not that difficult to implement. I'll give it a thought.

 

[dominikz]

This thread is meant to once and for all settle the age-old question on whether different non-broken DACs have a "sound" or if all DACs sound the same!
...just kidding - those debates will probably never end.

Let me start by providing the link to the ABX test.
Note: There are 16 trials in the test.

[EDIT 2021-12-29] For those wanting to use foobar2000 (in WASAPI exclusive mode) with ABX comparator plugin (16 trials suggested), here are direct links to the audio test files:
1. Topping E50 ABX sample file
2. FiiO Taishan D03K ABX sample file
If you do a test please report your results (copy/paste of ABX comparator result output) by posting in this thread or via a private message. Thanks!

Details of the test are provided below.

Introduction

Anyway, since I recently got a Topping E50 DAC and E1DA Cosmos ADC (both SOTA converters) I thought it might be interesting to prepare a controlled blind test between a SOTA DAC and one that is a relatively low-performing unit by todays standards (but not so much that one would call it "broken"). The ideas is to see if there are any audible differences in such an extreme comparison.
Many people might expect that this would be an easy test (given the price and spec difference), so let's start with that before trying anything else

Title states "Phase 1" because I plan to do a more difficult follow-up (E50 vs Babyface DAC) if results of this test show that several people can reliably tell the difference in this 'simple' test.

DACs under test

Meet the contenders:

• Topping E50
  • Price: 269$
  • SINAD: ~120dB
  • Full measurements and review link
• FiiO Taishan D03K
  • Price: 20$
  • SINAD: ~91dB
  • Full measurements link

View attachment 174684

Test equipment, SW and test track

The test track is the original 44,1kHz/24bit digital master wav file of the song "Farewell to Arms" (link to full song, available on various streaming services).
The song was mainly selected because I have the distribution rights and master files for it (shameless self-promotion alert! ).

The ADC used to record both DACs was the E1DA Cosmos ADC (full AP measurements available here) which, similarly to Topping E50, achieves measurement-equipment-grade conversion performance (SINAD ~120dB).

Head'n'HiFi Objective2 (O2) headphone amp in unity (1x) gain setting was used as an impedance buffer when recording the D03K - more on this below.

With both E50 and D03K the optical input was used to feed the DAC, and the source was the RME Babyface Silver Edition (1st gen) soundcard.

PreSonus Studio One 5 Professional was used to record the files. The project was configured to use the same 44,1kHz sample rate as the source track, and 32bit bit-depth to avoid any chance of loss of data.

ASIO4All driver was used so that the RME soundcard and E1DA Cosmos ADC USB devices can be used together as a single device for measurements and recording. 44,1kHz sample rate was used in all cases, to be consistent with the original recording and to avoid any resampling.

The online ABX test is constructed with the amazing abxtests.com web-based tool by @jaakkopasanen (see related thread). Thanks to @jaakkopasanen for building and providing this resource to the community!

Test file recording and preparation

In principle the concept is that of a simple loopback test - i.e. the output of both DACs was (separately) recorded by the same ADC, and resulting files level matched to better than 0,1dB accuracy.

Tone generator plugin (set to generate a -1dBFS 1kHz tone) was added to the track before the source file and used to calibrate the levels of both DACs when recording, and later to fine-tune the levels of the resulting recordings.

The recording chain was:

1) RME Babyface optical out -> Topping E50 optical in then balanced L/R out -> E1DA Cosmos ADC (set to 4,5V sensitivity) balanced L/R in

With this setup -1dBFS sine output results in -2,1dBFS input (-1,1dB loss in level).

Here's a short snip of the performance in this setup (used to reference with independent measurements and to make sure that the recording setup is correct):
View attachment 174697
View attachment 174698
We see THD+N is very low (at -115dB vs fundamental at 1kHz) and pretty close to maximum performance. Around 2dB is lost due to selected measurement input level, and additional ~3dB should be lost due to both ADC and DAC having similar SNRs and the noise summing in loopback.

2) RME Babyface optical out -> FiiO Taishan D03K optical in, then unbalanced RCA L/R out -> Head'n'Hifi O2 in 1x/unity gain (impedance buffer) -> E1DA Cosmos ADC (set to 1,7V sensitivity) unbalanced L/R input

With this setup -1dBFS sine output results in -3dBFS input (-2dB loss in level).

Note: E1DA Cosmos ADC has no input buffer and at 1,7V sensitivity has a very low input impedance (around 450 Ohms in unbalanced mode). If connected directly to D03K output this low input impedance is loading down the DAC's output, decreasing the level and increasing the distortion significantly. By using the O2 as buffer (1x gain with maximum volume pot position results in unity gain between input and output) its high input impedance and low output impedance allows for optimum impedance matching between the two devices. Since performance of O2 is much better than that of D03K there is no loss in signal transfer quality due to this.

Here's a short snip of the performance in this setup (used to reference with independent measurements and to make sure that the recording setup is correct):
View attachment 174699
View attachment 174700
Since D03K is not a very high performing unit (much worse performing than the rest of the measurement chain), this is fully in line with previous measurements, as well as the manufacturer's specification.

From the above diagrams we see that the total loopback recording chain 1kHz SINAD is:

• ~115 dB with Topping E50
• ~90 dB with FiiO Taishan D03K

Lastly let's compare the measured performance of the two DACs on FR and THD:
View attachment 174702
E50 FR is perfectly flat, while D03K has a ~0,3dB loss at 20Hz, and a ~0,7dB peak at 17,5kHz.
View attachment 174703
THD of E50 is more uniform across frequencies and >28dB better than that of D03K.

As we saw before, there is an overall ~0,9dB level difference in the raw recorded files, due to different analogue output levels of the two DACs and fixed sensitivity settings of the ADC.
To further fine-match the volume, Presonus Mixtool plugin (allowing gain adjustments to 0,01 dB precision) was used to tune the 1kHz tone level to maximum precision. Resulting files are therefore matched to significantly better than required 0,1dB (at 1kHz at least, note slight FR differences above).

Discussion and conclusion

As you can see, care was taken to achieve maximum performance from both DACs when preparing the test and to achieve very good level matching in all stages. This, combined with the blinded ABX test approach, ensures a controlled subjective comparison of audible differences between the two DACs.

Note however that an ABX test by itself will not tell us which DAC sounds 'better' (preference) - it will just tell us if there's any audible difference between them at all. This is still an important first step before it makes sense to investigate preference at all.

Though I consider this to be a difficult test to most people, I expect that those with well-preserved high frequency hearing (>10kHz) might still be able to hear a difference between the two DACs due to differences in their frequency response. Note that FiiO Taishan D03K has a slight sub-LF loss and a small top-octave peak that should result in a slightly brighter sound than the Topping E50, which could then be used to identify the "X".

I will wait for a week or so before posting preliminary result summary (unless a lot of people complete the test sooner and we have some significant results quickly). Anyone completing the test is of course free to post their results and impressions here at any time

If no listeners are able to reliably differentiate these two DACs I will not proceed to prepare the more difficult 'phase 2' test.
However if there are several listeners that can differentiate between these two DACs reliably, I will prepare the a similar comparison between the RME Babyface 1st gen DAC (flat FR, SINAD >100dB), representing average but solid DAC performance, and Topping E50 (flat FR, SINAD ~120dB) representing SOTA DAC performance.

Lastly, let me link again to the ABX test.
Note: There are 16 trials in the test.

[EDIT 2021-12-29] For those wanting to use foobar2000 (in WASAPI exclusive mode) with ABX comparator plugin (16 trials suggested), here are direct links to the audio test files:
1. Topping E50 ABX sample file
2. FiiO Taishan D03K ABX sample file
If you do a test please report your results (copy/paste of ABX comparator result output) by posting in this thread or via a private message. Thanks!

Enjoy!

So let me give a short overview of results we have so far.

These are the results of participants that took the online test via abxtests.com - we had a total of 22 completed attempts.
Note that here I'm saying 'attempts' instead of 'participants' - this is because a few participants reported they took the test more than once.

Number of correct answers (out of 16)	p-value P(X>=x)	Number of attempts (with the same correct answer count)	Comments
1	99,998%	0
2	99,974%	0
3	99,791%	0
4	98,936%	1
5	96,159%	2
6	89,494%	7
7	77,275%	4
8	59,819%	3	Note: In one attempt in this group "A" was selected every time, I suspect it was a test run of a sort and not a 'real' test attempt.
9	40,181%	3
10	22,725%	1
11	10,506%	0
12	3,841%	0
13	1,064%	0
14	0,209%	0
15	0,026%	0
16	0,002%	1

Note: p-value P(X>=x) has been calculated with this online calculator (n=16, p=0.5, q=0.5, K=<number of correct trials>) and cross-checked here. Please let me know if anyone spots any mistakes.

As we can see, out of the 22 finished (online) test attempts so far, we have one where all provided answers were correct (16/16, p-value = 0,002%). No other attempts achieved p-value lower than 1% (or even 5%). The second-best attempt got 10/16, for a p-value of 22,725%.

In addition to those included above, we had one participant who reported that they didn't finish the test due to not being able to hear a difference. I'm another one that fits into that category - I couldn't hear any difference between "A" and "B" so couldn't complete the test.

One participant reported also attempting the test in foobar2000 ABX comparator, but not being able to do better that guessing there either (ABX comparator results log were not posted, though).

As others have stated, we should be careful when interpreting results of this test (and similar ones). This is because there are still variables that are not controlled - such as individual system differences between participants, things like different operating system audio settings, possibly different browser behaviour, driver configuration, potential use of various audio enhancements, differences in audio equipment and its calibration, etc... So it seems unlikely to me that we can easily make many generalized conclusions from this test.
However, it is IMO interesting to note that while most had difficulty identifying "X" correctly within the constraints of this test, we seem to have had one attempt where a participant was able to reliably select the correct answer in all 16 trials. It would be interesting to hear what this participant used to anchor to when doing the test - e.g. was it listening for the slight increase in 'brightness' we expected from FiiO D03K vs Topping E50 (as predicted from frequency response measurement) or something else?

Please let me repeat again that my intention is not to argue that 'all DACs are the same' - IMO they are not.
E.g. FiiO D03K has some frequency response variations, not impressive SINAD, only 1,5V RMS maximum output and is sensitive to low load impedance. Therefore it is absolutely imaginable that this DAC's limitations may become audible in certain setups - e.g. those with less-than-ideal gain staging and/or when driving a very low impedance analogue inputs.
On the other hand, as long as the test setup is optimized to achieve good performance out of each DAC and some basic listening test controls are applied, it can IMO be surprising how close to transparency some of these budget DACs can be - even if compared to objectively much better performing units as here.

In the end, I do hope this was an interesting exercise to those included. Hopefully one that also illustrates the importance of precise level matching and blind listening when doing comparisons of audio equipment.

 

[Pdxwayne]

I decided to check the file differences using Deltawave program. I see that the initial peak value between files has a different of about 0.5db. So, I decided to listen to only 28 to 29 seconds of the files when doing abx.

I did a total of 4 set of 16 trials. I got 40/64. Per https://stattrek.com/online-calculator/binomial.aspx, p value for 40/64 is 0.0299. It would seems if I looked at the files close enough, I could find something that give me a significant p value.


I could have done another set with 64 trials, but too lazy to do that now. The different is not big enough for me to concern about anyway. Either DAC will work just fine for me, when playing the test song. Although, I would be very curious about comparing files with lots of low bass notes between 20 to 40 hz.


Here are my abx results for today:

 

[solderdude]

Of course you might start to wonder now if the used ADC did not impart its sound and make them sound more the same than if they were used directly.

 

[Blumlein 88]

Of course you might start to wonder now if the used ADC did not impart its sound and make them sound more the same than if they were used directly. View attachment 176112

I'd say not likely. Wasn't the Cosmos ADC in use here?

 

[somebodyelse]

Of course you might start to wonder now if the used ADC did not impart its sound and make them sound more the same than if they were used directly. View attachment 176112

I suspect that's why @KSTR asked for the original as well as the two recordings in #24 and #60

 

[KSTR]

I suspect that's why @KSTR asked for the original as well as the two recordings in #24 and #60

Yep. The high frequency characteristics/artifacts of the tested DACs will not be fully captured when recording at the same sample rate (nor reproduced by our DACs at home), which may or may not skew the results a bit. It's not a deal breaker and the Cosmos ADC is excellent quality.

 

[solderdude]

I'd say not likely. Wasn't the Cosmos ADC in use here?

@Pdxwayne and I have a history of challenging each other... did you notice the winking, smiling devil ? but with a hint of truth in it.

 

[dominikz]

I decided to check the file differences using Deltawave program. I see that the initial peak value between files has a different of about 0.5db. So, I decided to listen to only 28 to 29 seconds of the files when doing abx.

I did a total of 4 set of 16 trials. I got 40/64. Per https://stattrek.com/online-calculator/binomial.aspx, p value for 40/64 is 0.0299. It would seems if I looked at the files close enough, I could find something that give me a significant p value.


I could have done another set with 64 trials, but too lazy to do that now. The different is not big enough for me to concern about anyway. Either DAC will work just fine for me, when playing the test song. Although, I would be very curious about comparing files with lots of low bass notes between 20 to 40 hz.


Here are my abx results for today:
View attachment 176107

View attachment 176108

View attachment 176109

View attachment 176111

Wow, thanks for the effort - that is a lot of listening! I calculate the same p-value as you for the total number of trials (2.997% - i.e better than the 5% condition but worse that the more strict 1% condition).

Of course you might start to wonder now if the used ADC did not impart its sound and make them sound more the same than if they were used directly. View attachment 176112

Sure.
However the recorded files seem to at least show the expected FR difference, here's what we get when importing the left channel of each file into REW:

And if we calculate the spectrum difference we get:

We see the HF rise, and the slight LF loss in the FiiO D03K vs Topping E50.

I suspect that's why @KSTR asked for the original as well as the two recordings in #24 and #60

Yep. The high frequency characteristics/artifacts of the tested DACs will not be fully captured when recording at the same sample rate (nor reproduced by our DACs at home), which may or may not skew the results a bit. It's not a deal breaker and the Cosmos ADC is excellent quality.

Here is the link to the original file.
Note that polarity of this file is inverted vs the other two - this is because E1DA Cosmos ADC is recording with inverted polarity when using the original firmware. There is also some phase wrapping in the recorded files vs the original due to use of a linear phase filter in the ADC.

 

[solderdude]

I was just playing with Pdxwayne's mind.
He knows why and that it was a harmless thing.
He knows I like he is inquisitive and wants to explore his own hearing abilities. I am a great proponent for this.

 

[Blumlein 88]

Do corrective eq on the file played thru the FiiO and see if you still hear the difference.

 

[Volikovvv]

Decided to share my results with you.

ABX

Listening tests in the browser

abxtests.com

So i think there is a difference for me? Listened through D1se-A30pro-Hifiman Edition XS. One of them just sound a little bit brighter, just a touch, and i think it is cheaper Fiio, Topping a little bit warmer sounding. This is how i distinguish these two records.

 

[Pdxwayne]

Decided to share my results with you.
View attachment 177531

ABX

Listening tests in the browser

abxtests.com

So i think there is a difference for me? Listened through D1se-A30pro-Hifiman Edition XS. One of them just sound a little bit brighter, just a touch, and i think it is cheaper Fiio, Topping a little bit warmer sounding. This is how i distinguish these two records.

Nice! Should I assume you can still hear up to 20khz?