DAC ABX Test Phase 1: Does a SOTA DAC sound the same as a budget DAC if proper controls are put in place? Spoiler: Probably yes. :)

[voodooless]

I think for the reason that you can select a certain part of the downloaded files and repeat it/switch it in Foobar abx. This is the most effective way to tell the small sound differences.

Some would say the difference ought to be so glaringly obvious, none of these special features should be necessary

 

[pma]

Some would say the difference ought to be so glaringly obvious, none of these special features should be necessary

This is a question what we call a ”difference”. This might lead to a never ending debate. Then, music sample choice makes difference, again. If we stick with “glaringly obvious”, then no progress in DACs and amps is needed and we may stay in 1958 .

 

[voodooless]

This is a question what we call a ”difference”.

Simple definition: difference is something detectable in a proper double blind test. Now, what constitutes a proper test is much more up for debate I guess.

If we stick with “glaringly obvious”, then no progress in DACs and amps is needed and we may stay in 1958 .

Well, maybe not 1958, but early ‘90s? Possibly…

But the point it: so many claim magical qualities vs a simple stupid DAC of a few €$£¥, that surely no special features would be needed to keep them apart. So what happened.. are these stories not true after all? … Don’t worry, not expecting an answer on this

This thread is an excuse machine waiting to happen, myself included because the thing won’t work on my phone

 

[KSTR]

I can understand that. Would you perhaps be willing to argument why?
E.g. if you see some flaw in the methodology perhaps there are things I can improve in future attempts - I'd be very grateful for any help!
I'll also be happy to post the source files, but perhaps it makes sense to first allow some more time for people to complete the online version.
So far we have 6 people who finished the test, and one that reported starting the test but not finishing.

First of all, I'm fully OK with that, take your time and proceed the way you've planned.

My approach to an ABX like that is to have "full control over everything" and that includes having the original and the two recordings available as WAV-files. The reason is to create enough "motivation bias" to actually do the ABX proper. For this I do a detailed analysis of the data first. I could work around the access issue by digital loopback to losslessly re-record the recordings but it would be easier and safer to find them uploaded somehwere. I don't use streaming and windows system sound, I'm using SW players or DAW's for playback, offline.

For improvements, you may want to use a higher sample rate (like 2x or 4x) for the captures to avoid any top-end characteristic of the ADC (and our playback DACs) playing a role, plus this would not remove the IMD/HD components above fs/2 as well as the images above fs/2. This is fine tuning, of course, and not guaranteed to have real benefit in many cases.

An in general it would be best to have the sources and the recorder in hard clock-sync to avoid any clock drift/offset issues (which can even give false clues in ABX) but this is often impossible, actually I see it is strictly not possible here as the ADC has no means of syncing. When I'm doing test like this and the DUTs have Coax or Optical inputs, I can set up my two ADI-2 Pro FS units to have the DUT in sync and be able to record at 2x or 4x (requires 2 seperate computers as ASIO supports only one sample rate at at time).

 

[dominikz]

Some would say the difference ought to be so glaringly obvious, none of these special features should be necessary

This is a question what we call a ”difference”. This might lead to a never ending debate. Then, music sample choice makes difference, again. If we stick with “glaringly obvious”, then no progress in DACs and amps is needed and we may stay in 1958 .

In a way I agree with both arguments.

On the one hand, we often read claims of night-and-day audible differences between well-performing DACs in casual listening comparisons. This seems unlikely, especially if few people can distinguish devices that have large performance differences when proper controls are put into place. One really can't help but assume level mismatch, non-blind conditions or some other systematic error in many of those cases.

On the other hand, this does NOT mean that SOTA performance in DACs is useless. IMHO this would be a pretty narrow view that doesn't take into account some valid use-cases.
E.g. I bought the E50 for a few specific uses where I actually need its better performance (vs what I get from the otherwise solid DAC in my RME Babyface soundcard). When listening to music (or any other audio content), if I level match both DACs I can't really hear any difference between RME and E50 DACs. But I still have a practical use for the additional performance of the E50 when doing audio measurements, and for specific audio recording/production purposes.

So for me the point of this test is NOT to imply that buying objectively better DACs is a waste of people's money. However I do hope it can help:
1) qualify the rough extent of differences to expect in many cases when comparing DACs,
2) underline the importance of controlled testing to avoid biases and systematic sources of error, and
3) battle FOMO for people with lower budgets for audio purchases I personally find this last point important, as I believe non-technical people who perhaps can't afford quality audio gear can sometimes feel disheartened and demotivated when reading/hearing much of the usual audiophile wisdom.

First of all, I'm fully OK with that, take your time and proceed the way you've planned.

My approach to an ABX like that is to have "full control over everything" and that includes having the original and the two recordings available as WAV-files. The reason is to create enough "motivation bias" to actually do the ABX proper. For this I do a detailed analysis of the data first. I could work around the access issue by digital loopback to losslessly re-record the recordings but it would be easier and safer to find them uploaded somehwere. I don't use streaming and windows system sound, I'm using SW players or DAW's for playback, offline.

Fully understood and all valid points, IMHO.

For improvements, you may want to use a higher sample rate (like 2x or 4x) for the captures to avoid any top-end characteristic of the ADC playing a role, plus this would not remove the IMD/HD components above fs/2 as well as the images above fs/2. This is fine tuning, of course, and not guaranteed to have real benefit in many cases.

Good point, thanks! In this case I was trying to keep the same (source file) sample rate in all stages of the playback and recording chain to avoid any resampling artefacts, but I may look into this for future comparisons (if any).

An in general it would be best to have the sources and the recorder in hard clock-sync to avoid any clock drift/offset issues (which can give false clues in ABX) but this is often impossible, actually I see it is strictly not possible here as the ADC has no means of syncing. When I'm doing test like this and the DUTs have Coax or Optical inputs, I can set up my two ADI-2 Pro FS units to have the DUT in sync and be able to record at 2x or 4x (requires 2 seperate computers as ASIO supports only one sample rate at at time).

Again, fully agree with this.
Unfortunately, and as you also noticed, the E1DA Cosmos ADC only has USB output - would really love if there was e.g. S/PDIF output as well, with it I'd be able to sync both the DAC and ADC to the same clock via my RME soundcard as you suggested.
There are I2S headers on the Cosmos PCB however, but I've not yet been motivated enough to try and make use of those!

 

[xaviescacs]

In the tool, were you can choose your experience... a musician is supposed to have a worst ear than an audiophile? Come on... I will try when my new RME DAC arrives, in a week or so.

 

[dominikz]

In the tool, were you can choose your experience... a musician is supposed to have a worst ear than an audiophile? Come on... I will try when my new RME DAC arrives, in a week or so.

To be honest it never even crossed my mind someone might see it that way! There was definitely no such intention from my side when creating the test.

To be clear: the "Experience" list was NOT meant to grade critical hearing capability of listed groups in any order; the idea was just to have it as additional data in case we have a large enough sample in the end to do a more detailed statistical analysis of the results.

 

[Pdxwayne]

This is my second try using my phone and headphones. It is bad.

I noticed something odd with the p value. It should not be that low.

Also, I am pretty sure I heard a difference between files, but I got very bad results instead. Another point to doubt the capabilities of the web site's algorithm.

It would be great if you can share the files so that people can do this using foobar2000 abx also.

 

[xaviescacs]

To be honest it never even crossed my mind someone might see it that way! There was definitely no such intention from my side when creating the test.

To be clear: the "Experience" list was NOT meant to grade critical hearing capability of listed groups in any order; the idea was just to have it as additional data in case we have a large enough sample in the end to do a more detailed statistical analysis of the results.

Are you telling me that this list is the result of a random sampling? Come on... Now seriously, I think it should have been a multiple choice question. I'm in 6 of this categories and I'm not sure what to choose.

 

[Chrispy]

Yeah, just checking out the "experience" thing.....curious what the differences are for the type of listener between audio enthusiast and audiophile ....the former simply doesn't want to be identified as the latter?

 

[xaviescacs]

But again, to be able to draw a little profile, the answer should be multiple choice. Because a musician audio enthusiast it's quite different from a musician audio scientist, as there is no guarantee, rather than the natural order of the lost, that users will choose the most relevant category.

 

[pkane]

Yeah, just checking out the "experience" thing.....curious what the differences are for the type of listener between audio enthusiast and audiophile ....the former simply doesn't want to be identified as the latter?

I'm an audio enthusiast that identifies as an audiophile

 

[dominikz]

Are you telling me that this list is the result of a random sampling? Come on...

Not random obviously - just not graded. The list in my test is based on the categories from the 'demo' yaml template file available on ABX tool github page - I just added a few extra categories to try and cover more variants.

Yeah, just checking out the "experience" thing.....curious what the differences are for the type of listener between audio enthusiast and audiophile ....the former simply doesn't want to be identified as the latter?

Since participants are anyway self-identifying when doing the test I just wanted to give them more options. In hindsight it might have been smarter to omit the list completely

Multiple-choice doesn't seem to be supported by the tool at the moment, unfortunately, from ABX tool github page:

form is optional and can be used as a survey for collecting demographics information from the participants. You should avoid collecting personally identifiable information. Form can have any number of input fields, these are listed with the bullet point syntax. Each input field needs to know a name and an inputType. Supported types are text, number and select (dropdown). List the set of available options for select type input with the same bullet point syntax.

In the end this data will probably not be meaningful at all since I doubt there will be enough participants to try and find any kind of correlations.

 

[dominikz]

This is my second try using my phone and headphones. It is bad.

I noticed something odd with the p value. It should not be that low.

Also, I am pretty sure I heard a difference between files, but I got very bad results instead. Another point to doubt the capabilities of the web site's algorithm.

It would be great if you can share the files so that people can do this using foobar2000 abx also.

View attachment 175161

Thanks for sharing your results and comments!

Your p-value is actually relatively low (6,7%) which may indicate better-than-chance performance (even if more trials are wrong than right).

TBH at the moment I really see no good reason to doubt the tool.

The simple ABX demo provided by the author of the tool worked exactly as expected in my attempts, audio measurements I did seem to indicate the tool doesn't impact test file sound quality, and the preliminary results of the current DAC test seem to be in line with expectations (most participants' performance is no better than guessing, but not all).

It would be great if you can share the files so that people can do this using foobar2000 abx also.

Since it has been requested by several people so far I will do so - but I'd like to wait a few more days to give this online tool a chance

 

[xaviescacs]

In the end this data will probably not be meaningful at all since I doubt there will be enough participants to try and find any kind of correlations.

A few dozens would be enough to have some fun. Let's not despair.

Multiple-choice doesn't seem to be supported by the tool at the moment, unfortunately, from ABX tool github page:

Any js developer out there to give @jaakkopasanen a helping hand?

Overall,I find this is really nice, both the shape and the goal.

 

[Chrispy]

I'm an audio enthusiast that identifies as an audiophile

I'm more the other way around due the general bad name audiophiles have from all the silly stuff many swear by. It's been soiled.

ps I did start the test and on my laptop speakers heard no difference so just stopped....might be more interesting if I connect my computer otherwise, which I normally don't....or stream it but don't see an option to stream it.

 

[Pdxwayne]

Thanks for sharing your results and comments!

Your p-value is actually relatively low (6,7%) which may indicate better-than-chance performance (even if more trials are wrong than right).

TBH at the moment I really see no good reason to doubt the tool.

The simple ABX demo provided by the author of the tool worked exactly as expected in my attempts, audio measurements I did seem to indicate the tool doesn't impact test file sound quality, and the preliminary results of the current DAC test seem to be in line with expectations (most participant's performance is no better than guessing, but not all).


Since it has been requested by several people so far I will do so - but I'd like to wait a few more days to give this online tool a chance

Have you tried putting very obviously different files to check the online tool's accuracy in tracking selected results? I am curious about the maturity of the tool and if there is any potential algorithm issues. Thanks!

 

[dominikz]

Have you tried putting very obviously different files to check the online tool's accuracy in tracking selected results? I am curious about the maturity of the tool and if there is any potential algorithm issues. Thanks!

Indeed I have. First I completed the pretty obvious ABX demo and got the expected results.

Then I also tested the tool behaviour while I was measuring the streamed audio quality with my own 1kHz test tones, here's the result of that:

I.e. behaviour of the ABX tool in all my test trials was consistent and exactly as expected - I found no such issues.

Perhaps I should note that I too cannot hear any difference in my E50/D03K ABX test (my HF hearing ends somewhere between 16-17kHz).

 

[SIY]

It's a blind listening test. Why provide graphs? the THD and IMD metrics are completely insufficient metrics for describing the perceptual effects of nonlinear distortion. (1)

The problem with ABX tests is volume matching. You can't do that by ear. You need a calibrated Class 1 SPL Meter. Let's say a Larson Davis 831C-with a 378A04-microphone. Noise floor: 5 db(A). These things are very expensive. If I have to do a wild guess > $5000. This is a low estimate.
The meter needs to be regularly calibrated by the manufacturer or someone who is licensed to do that.

i quote Amirm "Without level matching, listening test results are unreliable" (2)

Ya ba dibba dibba dibba dibba dibba dibba dum O and it needs to be very quiet in your listening room when you're doing the test (it's 35 Db here right now and it's the middle of the night. The perks of living in a city). I like to see environmental noise level of < = 25 Db. I have one closed headphone. I only use that thing for recording sessions sometimes. It's an Audio Technica ATH-M50X. Sound isolation isn't that great. Sound quality is not that good either without EQ. but i am used to that and i know what to expect and correct accordingly. I also have several active monitors to verify the mix. I also listen in my car and a number of bluetooth speakers. If it sounds good on everything. It will probably be a good mix. I hope.

(1) Perception & Thresholds of Nonlinear Distortion using Complex Signals

https://hifisonix.com/wordpress/wp-content/uploads/2017/11/Perceptual-Levels-of-distortion.pdf

(2) https://www.audiosciencereview.com/forum/index.php?threads/understanding-audio-measurements.2351/

An SPL meter is not only expensive, but is a lousy tool for this purpose. Measure the electrical signal, it’s more accurate and far cheaper. Or use a software tool like Adobe or Goldwave.

 

[pma]

The problem with ABX tests is volume matching. You can't do that by ear. You need a calibrated Class 1 SPL Meter. Let's say a Larson Davis 831C-with a 378A04-microphone. Noise floor: 5 db(A). These things are very expensive. If I have to do a wild guess > $5000. This is a low estimate

Everyone knows this and almost everyone can use an AC multimeter and a sine test tone. These amateurish forum debates are hopeless, turning around in circles with same neverending arguments.