• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

DAC ABX shootout - unable to distinguish between 10$ and 15k$

Axo1989

Major Contributor
Joined
Jan 9, 2022
Messages
2,811
Likes
2,808
Location
Sydney
I’ve done mine. I’m guessing the dog ate yours?
The contrast between good faith postings here from @radix or @anotherhobby or @mdsimon2 and others, versus your adversarial quips should have been enough to tell me not to engage. My mistake.

I'll simply repeat that discussing basic scientific method and controlled testing informally in response to the OP here or in other threads like @dominikz recent ABX thread is entirely reasonable.

Asking some questions about the OP's recent experience and methods isn't necessarily "a strong defensive tendency" it may be simple curiosity. Framing it as a priori defensive is adversarial.
 

pkane

Master Contributor
Forum Donor
Joined
Aug 18, 2017
Messages
5,632
Likes
10,205
Location
North-East
The contrast between good faith postings here from @radix or @anotherhobby or @mdsimon2 and others, versus your adversarial quips should have been enough to tell me not to engage. My mistake.

I'll simply repeat that discussing basic scientific method and controlled testing informally in response to the OP here or in other threads like @dominikz recent ABX thread is entirely reasonable.

Asking some questions about the OP's recent experience and methods isn't necessarily "a strong defensive tendency" it may be simple curiosity. Framing it as a priori defensive is adversarial.

Agreed. Let’s start over.
 

Mnyb

Major Contributor
Forum Donor
Joined
Aug 14, 2019
Messages
2,641
Likes
3,617
Location
Sweden, Västerås
Hope op is well and and not scared away by so many heated responses on this imho not very controversial topic .
Seem like the kind of member the forum needs :) .

The having fun and learning stuff you know ?
 

anotherhobby

Addicted to Fun and Learning
Forum Donor
Joined
Dec 17, 2021
Messages
623
Likes
1,319
I'm fully expecting that when the OP comes back and explains his testing methods that they will be pretty solid, especially since his wife has a scientific background and participated. I'm not even remotely surprised by the result — I'm just really interested in how the test was performed, and I think it's important.
 
OP
E

Echoes

Member
Joined
Feb 21, 2022
Messages
21
Likes
145
Since we have helps, can we skip the switch? I always have a suspicion that the switch could do something to the signals.....

Blindfold the Op, wife manually move the headphones plug to different headphones amp, would that work?
This all started with helping hands switching between sources and the switch box. Once we we not able to tell the difference that switchbox is within the circuit, we moved to a 4-to-1 setup to continue our investigation. On Friday evening I will try to document and upload some pictures/info on the results for your evaluation. Perhaps somebody could provide the listening track suggestions to further remove bias?
 
OP
E

Echoes

Member
Joined
Feb 21, 2022
Messages
21
Likes
145
  1. What audio did you use to level match? pink noise? At what level at the mic?
  2. Did you do ABCDX, where ABCD are random order then randomly pick one for X, then ask "which one is X?"
  3. Or did you do randomized pairwise comparisons?
  4. How long were each sample played?
  5. How many different samples did your have, and were they randomized order?
  6. Could you list the tracks you used, I wonder how broad an audio gamut they span.

Thanks.

BTW, I recently started using this 1:4 (or 4:1) switchbox from Nobosound. With a little two-sided tape to the desk, it works very well.


1) Pink noise, however, to level match it, we used audacity recording input graph and were setting it to ‑10dB. This Friday we will perform adjustments using a scope on the headphone output voltage, as it has been pointed out that is more precise. Again, from the listening perspective, it seemed to be matched pretty decent.
2-3) ABCDX random order using a random number generator (excel =RANDBETWEEN(1,4).); Personal unpleasant experience is after you play the same system setup over and over again, when being told (or not knowing that) switching has been performed and start hearing "differences" - this is when the dirty stuff hits the pan and you start loosing confidence.
4) We agreed on a listening time no more than 2 minutes. Personal experience - more playtime - less confidence as you completely loose reference points. I have tried taking notes, my friend used "objectifying sound onto a certain object" method - whatever that means, but none of that seemed to help.
5-6) Due to lack of time we only performed 1 song testing - using Fleetwood Mac - Dreams. I am not confident this is the best track for selection, but for a first test it did the job.
 
OP
E

Echoes

Member
Joined
Feb 21, 2022
Messages
21
Likes
145
Thank you all for participating in this thread. I know some of the topics are quite heated, and I need to remind everybody that is is purely My own experience with my setup and personal limitations that does not apply to your ears or your systems. Throughout the discussions I have already gathered useful info on further testing how to level match, perhaps will get some suggestions on the refence tracks - this will only help to dive deeper into the research however unpleasant/eye opening the findings may be.

My biggest takeaway from this experience so far is that once you pass a certain threshold in distortions (both noise and equalization) .. enjoying the music has become more important than the kit. I will keep you updated where it takes me, thanks!
 
Last edited:
OP
E

Echoes

Member
Joined
Feb 21, 2022
Messages
21
Likes
145
BTW, I recently started using this 1:4 (or 4:1) switchbox from Nobosound. With a little two-sided tape to the desk, it works very well.

Seems to be a nice little kit, I guess the inner wiring is similar. Can you share your experience on do you hear the difference between it being in the signal path or bypassed? Testing blindly I could not tell with mine.
 

radix

Major Contributor
Joined
Aug 1, 2021
Messages
1,365
Likes
1,295
1) Pink noise, however, to level match it, we used audacity recording input graph and were setting it to ‑10dB. This Friday we will perform adjustments using a scope on the headphone output voltage, as it has been pointed out that is more precise. Again, from the listening perspective, it seemed to be matched pretty decent.
2-3) ABCDX random order using a random number generator (excel =RANDBETWEEN(1,4).); Personal unpleasant experience is after you play the same system setup over and over again, when being told (or not knowing that) switching has been performed and start hearing "differences" - this is when the dirty stuff hits the pan and you start loosing confidence.
4) We agreed on a listening time no more than 2 minutes. Personal experience - more playtime - less confidence as you completely loose reference points. I have tried taking notes, my friend used "objectifying sound onto a certain object" method - whatever that means, but none of that seemed to help.
5-6) Due to lack of time we only performed 1 song testing - using Fleetwood Mac - Dreams. I am not confident this is the best track for selection, but for a first test it did the job.

@Echos,

Thank you for the info. It will be interesting to see if doing the more accurate scope level matching affects the outcome.

So you have 10 minutes per test? ABCD @ 2min each, plus X @ 2 min? I don't know the literature on this, but personally I don't think I could keep a reliable sound memory for that long. For example, if X is "A", could you really remember it over all that time?

You might try sets of pairwise ABX and doing shorter listening windows. For example, do 12X, 13X, 14X, 23X, 24X, 34X in random orders and mixing up A and B (e.g. 12X or 21X). It's a lot more tests, but it might give different results. And you need to do enough so each pair gets about the same number of trials. Or just do a bunch of 12X and see if those results correlate to your ABCDX tests.

I'm just vicariously running the experiment with you, and it's a lot easier to spend your time doing a bunch of tests then mine :)

Thank you for your efforts.
 

radix

Major Contributor
Joined
Aug 1, 2021
Messages
1,365
Likes
1,295
Can you share your experience on do you hear the difference between it being in the signal path or bypassed? Testing blindly I could not tell with mine.
I don't have ABX blind tests for it in the signal path. But I don't hear any difference. My hearing conks out around 12 kHz nowadays, so maybe someone with younger ears could tell the difference?

In the beginning, I thought i heard crosstalk when switched to a different headphone, then I realized I was wearing the open back Sundara and was hearing another pair just through the open back. I've since repeated that test, cranking my ADI-2 DAC to full volume and I heard nothing on a non-selected channel.
 

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
1) Pink noise, however, to level match it, we used audacity recording input graph and were setting it to ‑10dB. This Friday we will perform adjustments using a scope on the headphone output voltage, as it has been pointed out that is more precise. Again, from the listening perspective, it seemed to be matched pretty decent.
2-3) ABCDX random order using a random number generator (excel =RANDBETWEEN(1,4).); Personal unpleasant experience is after you play the same system setup over and over again, when being told (or not knowing that) switching has been performed and start hearing "differences" - this is when the dirty stuff hits the pan and you start loosing confidence.
4) We agreed on a listening time no more than 2 minutes. Personal experience - more playtime - less confidence as you completely loose reference points. I have tried taking notes, my friend used "objectifying sound onto a certain object" method - whatever that means, but none of that seemed to help.
5-6) Due to lack of time we only performed 1 song testing - using Fleetwood Mac - Dreams. I am not confident this is the best track for selection, but for a first test it did the job.
I think maybe ABCDX is too much to ask for first time.

Do simple ABX to start. For example, start with the most expensive vs the cheapest, if you haven't already done that.

If hd650 bass clarity and bass extension is anything like the 560s, I would say it is not a good choice to hear minor differences.
Maybe try some online blind tests first to see if you can do well with the headphones?

For example, sound level tests like the one at:

Do distortion tests as described in this thread:

Let us know how you do.
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,383
Likes
24,749
Location
Alfred, NY
Perhaps somebody could provide the listening track suggestions to further remove bias?
The tracks chosen won't affect bias- you're doing controls for that, for which BIG thumbs up. The objection had to do with test sensitivity, i.e., could you be missing differences that are there because your chosen test track doesn't reveal them. So you might want to try using a few different tracks with very different recordings and demands.

My prediction is that it won't matter, you'll continue to get null results, and no matter how many different tracks you use, true believers in "DAC sound" (outside of frequency response) won't accept your results. But what you're trying to do (and again, big thumbs up for defining that very well) is to see if YOU can hear these differences, so the whining that it doesn't answer other questions is meaningless.

And again, serious kudos for being adventurous, open-minded, and willing to put your perceptions to the test. That's more than I can say about 99% of so-called audiophiles.
 
OP
E

Echoes

Member
Joined
Feb 21, 2022
Messages
21
Likes
145
The tracks chosen won't affect bias- you're doing controls for that, for which BIG thumbs up. The objection had to do with test sensitivity, i.e., could you be missing differences that are there because your chosen test track doesn't reveal them. So you might want to try using a few different tracks with very different recordings and demands.

My prediction is that it won't matter, you'll continue to get null results, and no matter how many different tracks you use, true believers in "DAC sound" (outside of frequency response) won't accept your results. But what you're trying to do (and again, big thumbs up for defining that very well) is to see if YOU can hear these differences, so the whining that it doesn't answer other questions is meaningless.

And again, serious kudos for being adventurous, open-minded, and willing to put your perceptions to the test. That's more than I can say about 99% of so-called audiophiles.
Really appreciate the support.
 

antcollinet

Master Contributor
Joined
Sep 4, 2021
Messages
7,409
Likes
12,294
Location
UK/Cheshire
But we do not know that is true until we know the methodology. If the methodology did not test a differentiating feature, then that invalidates the conclusion of the ABX test. Or if the test structure made differentiation difficult or impossible, I would say that invalidates the conclusion of no differentiation.

Maybe the only played drum & bass, not female vocal. Or maybe they played 10 minute segments for ABCD so by the time the X came around, of course one could not clearly attribute it to A, B, or C. Or maybe they did ABX choosing pairwise, but did not equally cover all pairs.

So, I do not accept the premise that a flawed test that indicates no differentiation still implies no differentiation in a test without those flaws.

BTW, I am not saying that this experiment did any of those things. I do not know. Thus my question.

Marc
You make some good points. I had in my mind the controls of double blind/unsighted, and level matched. If those are not done properly then I stand by my statement that they cannot cause a result of no difference

However as you point out there could be some test conditions that will mask a difference, eg distortion created downstream of the devices under test which might mask distortion being tested for, so I withdraw my more general statement.
 

anotherhobby

Addicted to Fun and Learning
Forum Donor
Joined
Dec 17, 2021
Messages
623
Likes
1,319
1) Pink noise, however, to level match it, we used audacity recording input graph and were setting it to ‑10dB. This Friday we will perform adjustments using a scope on the headphone output voltage, as it has been pointed out that is more precise. Again, from the listening perspective, it seemed to be matched pretty decent.
2-3) ABCDX random order using a random number generator (excel =RANDBETWEEN(1,4).); Personal unpleasant experience is after you play the same system setup over and over again, when being told (or not knowing that) switching has been performed and start hearing "differences" - this is when the dirty stuff hits the pan and you start loosing confidence.
4) We agreed on a listening time no more than 2 minutes. Personal experience - more playtime - less confidence as you completely loose reference points. I have tried taking notes, my friend used "objectifying sound onto a certain object" method - whatever that means, but none of that seemed to help.
5-6) Due to lack of time we only performed 1 song testing - using Fleetwood Mac - Dreams. I am not confident this is the best track for selection, but for a first test it did the job.

My thoughts repeat some of the advice already given. I would consider performing a higher number of tests with a smaller number of devices. A 4-way double blind test does seem exhausting to me, which will make it harder to focus. I'd narrow your testing down to the two devices you care the most about, or the biggest gap in tech. Do more tests on less devices to gain more confidence in your result. If you can't hear a difference there, I don't know that I'd go thru the effort of more testing other than to satisfy my own curiosities (which is what I guess you are doing anyway).

As for music, I agree with the school of thought of picking something you enjoy that you are very familiar with. I'd make sure it has plenty of dynamic range and is very well recorded. I don't expect any perceivable differences, but if you're really trying to suss out if you can hear a difference, you want to stack the deck in such a way that if there was a difference you'd have every opportunity to catch it, otherwise you might still question your result later.

Lastly, when I first tried DBT, I really wanted to detect a difference. I believed there was one, and that I could hear it, because my sighted tests seemed obvious to me. It became frustrating quick as I started losing confidence and couldn't tell the differences I thought were there. I had to step away and come back with a different mentality of just being curious about the whole thing instead of trying to "pass a test" or prove some sort of thing to myself. Do what you can to keep relaxed, and not detecting the differences (that might not be there) is not the same as not passing the test.
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,383
Likes
24,749
Location
Alfred, NY
My thoughts repeat some of the advice already given. I would consider performing a higher number of tests with a smaller number of devices. A 4-way double blind test does seem exhausting to me, which will make it harder to focus. I'd narrow your testing down to the two devices you care the most about, or the biggest gap in tech. Do more tests on less devices to gain more confidence in your result. If you can't hear a difference there, I don't know that I'd go thru the effort of more testing other than to satisfy my own curiosities (which is what I guess you are doing anyway).

As for music, I agree with the school of thought of picking something you enjoy that you are very familiar with. I'd make sure it has plenty of dynamic range and is very well recorded. I don't expect any perceivable differences, but if you're really trying to suss out if you can hear a difference, you want to stack the deck in such a way that if there was a difference you'd have every opportunity to catch it, otherwise you might still question your result later.

Lastly, when I first tried DBT, I really wanted to detect a difference. I believed there was one, and that I could hear it, because my sighted tests seemed obvious to me. It became frustrating quick as I started losing confidence and couldn't tell the differences I thought were there. I had to step away and come back with a different mentality of just being curious about the whole thing instead of trying to "pass a test" or prove some sort of thing to myself. Do what you can to keep relaxed, and not detecting the differences (that might not be there) is not the same as not passing the test.
Yes, any good experiment is not a pass or fail- whatever the results are, you learned something.
 

Pdxwayne

Major Contributor
Joined
Sep 15, 2020
Messages
3,219
Likes
1,172
@Echoes, one tip for you.

If you already have a good ADC, simply capture the playback for each chain into your computer.

Then, use @pkane 's deltawave app to do null comparisons between captured files. App available for free at https://deltaw.org/.

Look at the "pk metric tab". If you see something that is between 0 to -50db, then you may have a section of the song on which you can focus to listen for a difference.

So instead of listening for 2 minutes straight each time, you listen to a short section only to keep your short term memory more intact.
 
Top Bottom