• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

DAC ABX shootout - unable to distinguish between 10$ and 15k$

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,386
Likes
24,749
Location
Alfred, NY
I know its probably me, but I cannot fathom an apple dongle sounding the same as any Chord or Topping DAC.
Yet... it does. The performance of that little thing is astonishingly good. Not state of the art measurement, but damn excellent measurements. When I did a comparison to an RME ADI-2 Pro FS, nope, could hear nothing different. This really is the golden age of electronics.

1645659848113.png
 
Last edited:

Zensō

Major Contributor
Joined
Mar 11, 2020
Messages
2,740
Likes
6,740
Location
California
Yet... it does. The performance of that little thing is astonishingly good. Not state of the art measurement, but damn excellent measurements. When I did a comparison to an RME ADI-2 Pro FS, nope, could hear nothing different. This really is the golden age of electronics.

View attachment 188756
It is quite an amazing piece of tech, so good that it's hard for people to fathom. I have an ADI-2 DAC on my desk, but if I didn't need its various features and I/O I'd be just as happy with an Apple dongle.
 
Last edited:

Grooved

Addicted to Fun and Learning
Joined
Feb 26, 2021
Messages
679
Likes
441
I don't have a lot of faith in that online 2-way timing test. It seems like something else is going on that makes this test very simple over some setups. I was able to pass this test on my first attempt at 1ms after working my way up from 50ms using only my Dell Latitude work laptop with the built-in speakers and Window's default drivers at 40% volume level.

Maybe the lower delays simply cannot be properly reproduced with some equipment and this manifests as a clearly audible difference not exclusively related to only the timing differences?

I kind of wonder if the difference is due to single driver of your headphones vs multi speakers of your MacBook.

I did a quick search about Macbook:
......
Both the 14-inch MacBook Pro and 16-inch MacBook Pro feature a six-speaker sound system with 80% more bass. The speakers include two, larger tweeters and four force-canceling woofers that Apple says can go half an octave (or "so la ti do") deeper than before.Oct 18, 2021
.......

@Grooved also mentioned using stereo speakers was very easy.

In that case, it seems to indicate you might have better luck hearing a difference between your test chains using stereo speakers instead of headphones....

Distorsion may also change transients, so maybe a really bad equipment could increase the chance of getting it right, but I now think using stereo speakers or laptop with woofer and tweeters make each instrument being separated, which is not the case in headphones. With speakers, Kick Drum goes in one driver (woofer) and Hi Hat goes in the other driver (tweeter)

Regarding the stereo effect, I already found it easier to find difference between AAC and FLAC on stereo speakers than on headphones, so I wouldn't be surprised if it was the case here too, but the test is mono so it should have less impact.
I will do a test the only one speaker to see if it changes something.
 
Last edited:

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,386
Likes
24,749
Location
Alfred, NY
It is quite an amazing piece of tech, so good that it's hard for people to fathom. I have an ADI-2 DAC on my desk, but if I didn't need it's various features and I/O I'd be just as happy with an Apple dongle.
That's my feeling as well. The RME has some amazing features, has a great ADC, and can drive just about anything. I love it. But if all I needed were a basic DAC, the dongle would do fine.
 

Gene LeClair

Member
Joined
Jan 18, 2022
Messages
45
Likes
69
I can't say that the use of an iPhone or iPad having a downside, I am more concerned (for me) with the source of the music files themselves.
I am not a streaming person and do NOT have my pc hooked into my stereo. All digital files (FLAC) are played via a pi4.
I know its probably me, but I cannot fathom an apple dongle sounding the same as any Chord or Topping DAC.
So my mind goes right to the source files as the likely culprit.
You should do a double blinded test of source files with your equipment and post the results. It would be an interesting test to read.
 
OP
E

Echoes

Member
Joined
Feb 21, 2022
Messages
21
Likes
145
Hello all and thank you again for active participation:

This Friday evening I came in much better prepared: four songs clipped @ 30 second samples, same source (iPhone 13Pro using VLC), two systems to compare (Chord Dave vs Apple Lightning Dongle), volume matched using a small portable scope + double check using ears, and a simplified test: The test subject does not know which system is which, though he can switch between source No.1 and Source No.2. Sources play the same track, almost in synch. Subject can switch between the two sources as many times and as quickly as he wants and only needs to identify which system is which. My friend has scored 3/4 (got overly confident), I have managed to nail 4/4 tracks. So there is a difference! Have latter spend around two hours listening to the music and got very confident nailing down each DAC signature sounds. Tried the same tests using Chord Dave without the upscaller with similar results. Speaking of marginal returns, the both devices produced music quite well.

I would not be confident that I would distinguish then in a double blind ABX testing, but I shall definitely try in the future, starting with sequence A, B, X repeated by A, B, X. I suspect my confidence would go down if the listening sessions would be longer, lets say 1-2 minute of each system instead of available instant switch.

Testing is very much like work rather than enjoying the music, so I am sorry I did not perform 100+ hours of analysis, excel sheets filled with data etc. for you.

Speaking of.. sound level matching. Proven crucial. I used a portable scope, yet easiest was to set it using sine-wave, not pink-white noise or music. Using noise files the voltages were constantly drifting, I know the USB scope is more of a toy and not a piece of art, yet it is way more sensitive to voltage than my Voltmeter. Can you share your experiences or drop a link to a good resource on level matching?

Moreover, one more question on Tube amps. I have a Bottlehead Crack OTL tube amp, which I level matched to A90 headphone amp, using a scope + 1 kHz sine wave. Crack OTL sounded very much more quiet, then I volume matched using my ears and got 115 mV Vpp on the A90 and 155 mV Vpp on the Bottlehead crack. The Tube amp voltage also seemed to wobble a lot from its main axis. Anybody could comment on that? Screenshots attached.
 

Attachments

  • B_CRACK.png
    B_CRACK.png
    796.9 KB · Views: 65
  • A90.png
    A90.png
    935.2 KB · Views: 61

stanroz

Member
Joined
Nov 30, 2021
Messages
54
Likes
9
A fascinating thread!

one area which should not be ignored is the role of the subconscious. In these double blind tests It’s important to have test subjects report not only what they hear but how they feel.

In my own comparison testing (not blind though) I found that very often I feel the difference faster than hear it…. Of course I could be fooling myself …. which is why these double blind tests are important.

Also I found that sometimes I can tell very little difference rapidly switching back and forth, but when playing for longer periods, the difference (between headphones, etc) is more profound. For instance, one playback will make me less tired than another, etc.

I don’t have the link but I read a report once based on double blind studies on lossy vs lossless playback… the report concluded that the difference was FELT more than HEARD and that there was some evidence that lossy introduced more negativity into one’s emotional response.
 
OP
E

Echoes

Member
Joined
Feb 21, 2022
Messages
21
Likes
145
A fascinating thread!

one area which should not be ignored is the role of the subconscious. In these double blind tests It’s important to have test subjects report not only what they hear but how they feel.

In my own comparison testing (not blind though) I found that very often I feel the difference faster than hear it…. Of course I could be fooling myself …. which is why these double blind tests are important.

Also I found that sometimes I can tell very little difference rapidly switching back and forth, but when playing for longer periods, the difference (between headphones, etc) is more profound. For instance, one playback will make me less tired than another, etc.

I don’t have the link but I read a report once based on double blind studies on lossy vs lossless playback… the report concluded that the difference was FELT more than HEARD and that there was some evidence that lossy introduced more negativity into one’s emotional response.
Hello Stanroz,

Just to point out, that in this comparison test both me and another "subject" were able to accurately say which one of the systems was playing. So I would say both the difference and sound signature could be determined on this one. ABX no doubt would be harder.
 

billyjoebob

Senior Member
Joined
Oct 9, 2021
Messages
307
Likes
118
A fascinating thread!

one area which should not be ignored is the role of the subconscious. In these double blind tests It’s important to have test subjects report not only what they hear but how they feel.

In my own comparison testing (not blind though) I found that very often I feel the difference faster than hear it…. Of course I could be fooling myself …. which is why these double blind tests are important.

Also I found that sometimes I can tell very little difference rapidly switching back and forth, but when playing for longer periods, the difference (between headphones, etc) is more profound. For instance, one playback will make me less tired than another, etc.

I don’t have the link but I read a report once based on double blind studies on lossy vs lossless playback… the report concluded that the difference was FELT more than HEARD and that there was some evidence that lossy introduced more negativity into one’s emotional response.
Listening fatigue is very real and not often considered.
It may very well be the primary reason to NOT like a product.
 
Last edited:

Grooved

Addicted to Fun and Learning
Joined
Feb 26, 2021
Messages
679
Likes
441
Hello all and thank you again for active participation:

This Friday evening I came in much better prepared: four songs clipped @ 30 second samples, same source (iPhone 13Pro using VLC), two systems to compare (Chord Dave vs Apple Lightning Dongle), volume matched using a small portable scope + double check using ears, and a simplified test: The test subject does not know which system is which, though he can switch between source No.1 and Source No.2. Sources play the same track, almost in synch. Subject can switch between the two sources as many times and as quickly as he wants and only needs to identify which system is which. My friend has scored 3/4 (got overly confident), I have managed to nail 4/4 tracks. So there is a difference! Have latter spend around two hours listening to the music and got very confident nailing down each DAC signature sounds. Tried the same tests using Chord Dave without the upscaller with similar results. Speaking of marginal returns, the both devices produced music quite well.

I would not be confident that I would distinguish then in a double blind ABX testing, but I shall definitely try in the future, starting with sequence A, B, X repeated by A, B, X. I suspect my confidence would go down if the listening sessions would be longer, lets say 1-2 minute of each system instead of available instant switch.

Testing is very much like work rather than enjoying the music, so I am sorry I did not perform 100+ hours of analysis, excel sheets filled with data etc. for you.

Speaking of.. sound level matching. Proven crucial. I used a portable scope, yet easiest was to set it using sine-wave, not pink-white noise or music. Using noise files the voltages were constantly drifting, I know the USB scope is more of a toy and not a piece of art, yet it is way more sensitive to voltage than my Voltmeter. Can you share your experiences or drop a link to a good resource on level matching?

Moreover, one more question on Tube amps. I have a Bottlehead Crack OTL tube amp, which I level matched to A90 headphone amp, using a scope + 1 kHz sine wave. Crack OTL sounded very much more quiet, then I volume matched using my ears and got 115 mV Vpp on the A90 and 155 mV Vpp on the Bottlehead crack. The Tube amp voltage also seemed to wobble a lot from its main axis. Anybody could comment on that? Screenshots attached.
Interesting @Echoes

EDIT : what follows was a mistake I made, it was for another thread with ABX test ;)

One thing I forgot to ask you sooner regarding your previous test: can you do a test with Deltawave to compare each one of your previous recordings with the original file?
If I'm not wrong (unless I saw it on another thread), you compared both files between each other, but not each one with the original file
I ask that because using Cosmos ADC, you certainly did have no clock sync between your DAC and the ADC

Try using these settings: (and maybe with "clock drift" enabled

DW basic settings.PNG


and
DW advanced settings.PNG
 
Last edited:

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
Hello all and thank you again for active participation:

This Friday evening I came in much better prepared: four songs clipped @ 30 second samples, same source (iPhone 13Pro using VLC), two systems to compare (Chord Dave vs Apple Lightning Dongle), volume matched using a small portable scope + double check using ears, and a simplified test: The test subject does not know which system is which, though he can switch between source No.1 and Source No.2. Sources play the same track, almost in synch. Subject can switch between the two sources as many times and as quickly as he wants and only needs to identify which system is which. My friend has scored 3/4 (got overly confident), I have managed to nail 4/4 tracks. So there is a difference! Have latter spend around two hours listening to the music and got very confident nailing down each DAC signature sounds. Tried the same tests using Chord Dave without the upscaller with similar results. Speaking of marginal returns, the both devices produced music quite well.

I would not be confident that I would distinguish then in a double blind ABX testing, but I shall definitely try in the future, starting with sequence A, B, X repeated by A, B, X. I suspect my confidence would go down if the listening sessions would be longer, lets say 1-2 minute of each system instead of available instant switch.

Testing is very much like work rather than enjoying the music, so I am sorry I did not perform 100+ hours of analysis, excel sheets filled with data etc. for you.

Speaking of.. sound level matching. Proven crucial. I used a portable scope, yet easiest was to set it using sine-wave, not pink-white noise or music. Using noise files the voltages were constantly drifting, I know the USB scope is more of a toy and not a piece of art, yet it is way more sensitive to voltage than my Voltmeter. Can you share your experiences or drop a link to a good resource on level matching?

Moreover, one more question on Tube amps. I have a Bottlehead Crack OTL tube amp, which I level matched to A90 headphone amp, using a scope + 1 kHz sine wave. Crack OTL sounded very much more quiet, then I volume matched using my ears and got 115 mV Vpp on the A90 and 155 mV Vpp on the Bottlehead crack. The Tube amp voltage also seemed to wobble a lot from its main axis. Anybody could comment on that? Screenshots attached.
Nice! Now that you hear a difference, I think you will get many questions from many members.
; )

Here are mine:
1. When you do voltage matching, did you use the same exact chain as your listening test chain?
2. How many times per song did you test to determine your 4 songs out of 4 songs results?
3. What are the differences you sensed/heard between the chain?

Thanks!
 

anotherhobby

Addicted to Fun and Learning
Forum Donor
Joined
Dec 17, 2021
Messages
623
Likes
1,319
Hello all and thank you again for active participation:

This Friday evening I came in much better prepared: four songs clipped @ 30 second samples, same source (iPhone 13Pro using VLC), two systems to compare (Chord Dave vs Apple Lightning Dongle), volume matched using a small portable scope + double check using ears, and a simplified test: The test subject does not know which system is which, though he can switch between source No.1 and Source No.2. Sources play the same track, almost in synch. Subject can switch between the two sources as many times and as quickly as he wants and only needs to identify which system is which. My friend has scored 3/4 (got overly confident), I have managed to nail 4/4 tracks. So there is a difference! Have latter spend around two hours listening to the music and got very confident nailing down each DAC signature sounds. Tried the same tests using Chord Dave without the upscaller with similar results. Speaking of marginal returns, the both devices produced music quite well.

I would not be confident that I would distinguish then in a double blind ABX testing, but I shall definitely try in the future, starting with sequence A, B, X repeated by A, B, X. I suspect my confidence would go down if the listening sessions would be longer, lets say 1-2 minute of each system instead of available instant switch.

Testing is very much like work rather than enjoying the music, so I am sorry I did not perform 100+ hours of analysis, excel sheets filled with data etc. for you.

Speaking of.. sound level matching. Proven crucial. I used a portable scope, yet easiest was to set it using sine-wave, not pink-white noise or music. Using noise files the voltages were constantly drifting, I know the USB scope is more of a toy and not a piece of art, yet it is way more sensitive to voltage than my Voltmeter. Can you share your experiences or drop a link to a good resource on level matching?

Moreover, one more question on Tube amps. I have a Bottlehead Crack OTL tube amp, which I level matched to A90 headphone amp, using a scope + 1 kHz sine wave. Crack OTL sounded very much more quiet, then I volume matched using my ears and got 115 mV Vpp on the A90 and 155 mV Vpp on the Bottlehead crack. The Tube amp voltage also seemed to wobble a lot from its main axis. Anybody could comment on that? Screenshots attached.
Very interesting! I'm also especially curious on how many individual tests (guesses) were conducted and the exact number passed for each person.
 

Roland68

Major Contributor
Joined
Jan 31, 2020
Messages
1,363
Likes
1,127
Location
Cologne, Germany
Listening fatigue is very real and not often considered.
It may very well be the primary reason to NOT like a product.
I can only confirm that, with both extremes.
I had already compared devices/chains where it was just fun to listen to the pieces of music. So much so that I kept forgetting that I wanted to compare the devices and listened to the songs to the end, often several times in a row. And although the test lasted several hours instead of an hour at most, it was very relaxing.

On the other hand, a year ago I had a DAC with me that, with one of my standard test pieces of music, I would have passed any ABX test with another device.
Actually, the device was my favorite in my test and I had hopes that it would be as good as the other devices, or even better.
But this particular device gave me a headache within 30 seconds of playing a certain song. Music off, headache gone, music on and headache back within 30 seconds.
I repeated this for 5 days within 2 weeks with the same result. I even got 2 replacement devices to rule out a defect, but it was the same. The serial numbers were only about 10 or 20 devices apart.
I've had devices be annoying, sound uncomfortable, or be really bad before, but I've never had a headache.
Otherwise, the device sounded normal and very good, even with other songs, really good.
 
OP
E

Echoes

Member
Joined
Feb 21, 2022
Messages
21
Likes
145
Nice! Now that you hear a difference, I think you will get many questions from many members.
; )

Here are mine:
1. When you do voltage matching, did you use the same exact chain as your listening test chain?
2. How many times per song did you test to determine your 4 songs out of 4 songs results?
3. What are the differences you sensed/heard between the chain?

Thanks!
Hi,

1) Yes, load (headphones) connnected to the amps too.
2) That was done in a following manner: 10 swiches between different sources per song, subject knows that each time its an different source. Subject needs to determine which source is which.
3) Most important difference with my ears was the instrument/vocals articulation. Sound was softer, but not mushy in the Chord Dave. In comparison Apple adapter was more punchy, yet quieter track details were leas noticable and destrinquishable. I really need up up my game in describing sound..
 
OP
E

Echoes

Member
Joined
Feb 21, 2022
Messages
21
Likes
145
Interesting @Echoes

One thing I forgot to ask you sooner regarding your previous test: can you do a test with Deltawave to compare each one of your previous recordings with the original file?
If I'm not wrong (unless I saw it on another thread), you compared both files between each other, but not each one with the original file
I ask that because using Cosmos ADC, you certainly did have no clock sync between your DAC and the ADC

Try using these settings: (and maybe with "clock drift" enabled

View attachment 189506

and
View attachment 189507
Please help me on that.
What I did is take a ripped cd, .flac format, but 30 seconds of it using audacity and exported to VAW. What exactly should I be comparing to what?
 

billyjoebob

Senior Member
Joined
Oct 9, 2021
Messages
307
Likes
118
I can only confirm that, with both extremes.
I had already compared devices/chains where it was just fun to listen to the pieces of music. So much so that I kept forgetting that I wanted to compare the devices and listened to the songs to the end, often several times in a row. And although the test lasted several hours instead of an hour at most, it was very relaxing.

On the other hand, a year ago I had a DAC with me that, with one of my standard test pieces of music, I would have passed any ABX test with another device.
Actually, the device was my favorite in my test and I had hopes that it would be as good as the other devices, or even better.
But this particular device gave me a headache within 30 seconds of playing a certain song. Music off, headache gone, music on and headache back within 30 seconds.
I repeated this for 5 days within 2 weeks with the same result. I even got 2 replacement devices to rule out a defect, but it was the same. The serial numbers were only about 10 or 20 devices apart.
I've had devices be annoying, sound uncomfortable, or be really bad before, but I've never had a headache.
Otherwise, the device sounded normal and very good, even with other songs, really good.
I had a Topping D50S that sounded great, but also gave me a headache after 1/2hr of listening.
I'm sure this is not everyone's experience, but i had to sell it..
So while (some) say all dac's sound the same, fatigue can't be argued.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,524
Likes
37,057
Hello all and thank you again for active participation:

This Friday evening I came in much better prepared: four songs clipped @ 30 second samples, same source (iPhone 13Pro using VLC), two systems to compare (Chord Dave vs Apple Lightning Dongle), volume matched using a small portable scope + double check using ears, and a simplified test: The test subject does not know which system is which, though he can switch between source No.1 and Source No.2. Sources play the same track, almost in synch. Subject can switch between the two sources as many times and as quickly as he wants and only needs to identify which system is which. My friend has scored 3/4 (got overly confident), I have managed to nail 4/4 tracks. So there is a difference! Have latter spend around two hours listening to the music and got very confident nailing down each DAC signature sounds. Tried the same tests using Chord Dave without the upscaller with similar results. Speaking of marginal returns, the both devices produced music quite well.

I would not be confident that I would distinguish then in a double blind ABX testing, but I shall definitely try in the future, starting with sequence A, B, X repeated by A, B, X. I suspect my confidence would go down if the listening sessions would be longer, lets say 1-2 minute of each system instead of available instant switch.

Testing is very much like work rather than enjoying the music, so I am sorry I did not perform 100+ hours of analysis, excel sheets filled with data etc. for you.

Speaking of.. sound level matching. Proven crucial. I used a portable scope, yet easiest was to set it using sine-wave, not pink-white noise or music. Using noise files the voltages were constantly drifting, I know the USB scope is more of a toy and not a piece of art, yet it is way more sensitive to voltage than my Voltmeter. Can you share your experiences or drop a link to a good resource on level matching?

Moreover, one more question on Tube amps. I have a Bottlehead Crack OTL tube amp, which I level matched to A90 headphone amp, using a scope + 1 kHz sine wave. Crack OTL sounded very much more quiet, then I volume matched using my ears and got 115 mV Vpp on the A90 and 155 mV Vpp on the Bottlehead crack. The Tube amp voltage also seemed to wobble a lot from its main axis. Anybody could comment on that? Screenshots attached.
Level matching is simple. Get a multimeter that reads AC voltage. Send a 1 khz sine wave through, and measure at the speaker terminals if using speakers, or at the output jack if using headphones. Match voltage to within 1.12% or certainly get it to better than 2%. I suggest a 1 khz signal at -20 db. A scope is not the easiest way to do this. If your matching was 115 mv vs 155 mv that is about 2.6 db difference and will easily be heard as different. Matching is desired to within .1 db to not corrupt results. Again this is less than 2% difference between the two values. So between 153 and 157 mv would be a good enough match for 155 mv.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,524
Likes
37,057
Interesting @Echoes

One thing I forgot to ask you sooner regarding your previous test: can you do a test with Deltawave to compare each one of your previous recordings with the original file?
If I'm not wrong (unless I saw it on another thread), you compared both files between each other, but not each one with the original file
I ask that because using Cosmos ADC, you certainly did have no clock sync between your DAC and the ADC

Try using these settings: (and maybe with "clock drift" enabled

View attachment 189506

and
View attachment 189507
Actually you can compare files that weren't locked by the clock. That is one of the big benefits of Deltawave.
 
Top Bottom