DAC ABX shootout - unable to distinguish between 10$ and 15k$

[DonR]

I can't decide whether to respond a) incorrect, or b) why not triple? Science is hard.

Because double eliminates bias from creeping into the test. Triple doesn't further reduce something that has already been eliminated.

 

[Axo1989]

The reason for the apparent contradiction is that the controls in a properly implemented blind test are designed to prevent any biases from influencing the result. Biases can alter the subjects perception so that they hear a difference even when non exists.

If the controls are NOT properly implemented, they are more likely to perceive a difference where not exists. However poorly implemented controls will never hide a difference that does exist.

So that means if someone does not implement the controls properly but STILL fails to detect a difference in the test it really doesn't matter much if the test set up was flawed. A flawed test setup won't prevent you from hearing a difference that genuinely exists.

So in the negative result, the test setup doen't need to be questioned to the same extent - or at all: If I can't hear a difference even in a fully sighted test, then I can't hear it. Test controls are not needed.

Uh huh.

 

[anotherhobby]

The reason for the apparent contradiction is that the controls in a properly implemented blind test are designed to prevent any biases from influencing the result. Biases can alter the subjects perception so that they hear a difference even when non exists.

If the controls are NOT properly implemented, they are more likely to perceive a difference where not exists. However poorly implemented controls will never hide a difference that does exist.

So that means if someone does not implement the controls properly but STILL fails to detect a difference in the test it really doesn't matter much if the test set up was flawed. A flawed test setup won't prevent you from hearing a difference that genuinely exists.

So in the negative result, the test setup doen't need to be questioned to the same extent - or at all. If I can't hear a difference in a fully sighted test, then I can't hear it. Test controls are not needed.

I don't agree that this is a good reason to not question testing methodologies when the result confirms your hypothesis.

 

[antcollinet]

I don't agree that this is a good reason to not question testing methodologies when the result confirms your hypothesis.

No - it is not being ignored when it confirms a hypothesis. It is being ignored when it cant have had an influence the result.

 

[Mnyb]

Yes it’s a bit lazy.

But it is the expected result ? Not “according to ASR” but as things works ? So yes people including me did not ask to many pointed follow ups ?

I understand why the questions seems to get a bit harder when someone finds an unicorn ? Blind tests digital coax cables and finds they sound different ?
If someone finds the unexpected that counter what we knows , the questions comes fast and a search for experimental errors starts . Because it usually is an experiment error .

But as anotherhobby sais . You can have a false confirmation even if the conclusion is correct ?

 

[Axo1989]

Because double eliminates bias from creeping into the test. Triple doesn't further reduce something that has already been eliminated.

Oh crap, I'd decided not to post that, because pedantry, but it must have been there in the reply box. I'll have to stand by it now though. Your post is nonsense (using the term idiomatically, not literally). Single, double and triple blind are all valid methodologically, and usage depends on a number of factors (what we are testing, what we need to control, etc). Triple is absolutely necessary in certain test/analysis scenarios.

 

[antcollinet]

Uh huh.

 

[pkane]

I don't agree that this is a good reason to not question testing methodologies when the result confirms your hypothesis.

1. ASR is not about doing science or confirming hypotheses -- it's about reporting objective measurements of audio devices

2. Even if we were all doing science here, how many Physicists do you think would jump to validate an experiment that confirms Newton's F=ma these days? Not a lot, if any. How many would question an experiment that demonstrated that F=ma^2, for example? Most, if not all.

 

[radix]

No - it is not being ignored when it confirms a hypothesis. It is being ignored when it cant have had an influence the result.

But we do not know that is true until we know the methodology. If the methodology did not test a differentiating feature, then that invalidates the conclusion of the ABX test. Or if the test structure made differentiation difficult or impossible, I would say that invalidates the conclusion of no differentiation.

Maybe the only played drum & bass, not female vocal. Or maybe they played 10 minute segments for ABCD so by the time the X came around, of course one could not clearly attribute it to A, B, or C. Or maybe they did ABX choosing pairwise, but did not equally cover all pairs.

So, I do not accept the premise that a flawed test that indicates no differentiation still implies no differentiation in a test without those flaws.

BTW, I am not saying that this experiment did any of those things. I do not know. Thus my question.

Marc

 

[Axo1989]

Assuming good faith in the OP, and that I get your previous drift, it's correct that type III errors aren't usually serious (where you still have the correct result) but it isn't correct to say they can't occur. We are also assuming we aren't seeing a type I or II error. But as the post above indicates, we don't know that yet.

 

[Falcon91Wolvrn03]

perhaps you will find my online DAC ABX test interesting as well. That one compared an objectively flawed low-budget DAC (FiiO D03K) with one that is SOTA performing (Topping E50) and so far the large majority of participants seem to be struggling to detect any difference.

If I understand the result correctly, did it show that most attempts couldn't tell a difference, but one out of 72 could? That a difference was heard by someone with very good hearing???

 

[pkane]

But we do not know that is true until we know the methodology. If the methodology did not test a differentiating feature, then that invalidates the conclusion of the ABX test. Or if the test structure made differentiation difficult or impossible, I would say that invalidates the conclusion of no differentiation.

Maybe the only played drum & bass, not female vocal. Or maybe they played 10 minute segments for ABCD so by the time the X came around, of course one could not clearly attribute it to A, B, or C. Or maybe they did ABX choosing pairwise, but did not equally cover all pairs.

So, I do not accept the premise that a flawed test that indicates no differentiation still implies no differentiation in a test without those flaws.

What the OP reported was a test of specific devices, using his own ears and his own system and music content that didn't sound different to him during that one particular test. For some reason there is a strong defensive tendency here to try to refute the results because they don't apply to everyone, and to every device under the sun, used in every possible system, with every possible piece of music.

The OP didn't set out to prove that nobody can hear the differences, nor did he set out to prove that these differences can't be heard with all content, and under all possible circumstances. So, let's treat this result as a single data point that it is. One that doesn't generalize to the whole world population and doesn't prove any generalizable hypothesis. Just a data point.

 

[Pdxwayne]

What the OP reported was a test of specific devices, using his own ears and his own system and music content that didn't sound different to him during that one particular test. For some reason there is a strong defensive tendency here to try to refute the results because they don't apply to everyone, and to every device under the sun, used in every possible system, with every possible piece of music.

The OP didn't set out to prove that nobody can hear the differences, nor did he set out to prove that these differences can't be heard with all content, and under all possible circumstances. So, let's treat this result as a single data point that it is. One that doesn't generalize to the whole world population and doesn't prove any generalizable hypothesis. Just a data point.

So, you are not curious that using a different headphones might get him different results? Or a certain song is better for testing? You just want the results as is?

 

[pkane]

So, you are not curious that using a different headphones might get him different results? You just want the results as is?

To be blunt, I care about my own listening test results, and assume that others do the same. If the result satisfied OP’s curiosity, then I’m perfectly fine with the result he got.

 

[DonR]

Oh crap, I'd decided not to post that, because pedantry, but it must have been there in the reply box. I'll have to stand by it now though. Your post is nonsense (using the term idiomatically, not literally). Single, double and triple blind are all valid methodologically, and usage depends on a number of factors (what we are testing, what we need to control, etc). Triple is absolutely necessary in certain test/analysis scenarios.

Uh huh.

 

[Axo1989]

1. ASR is not about doing science or confirming hypotheses -- it's about reporting objective measurements of audio devices

Absolutely. That's why we are posting here on Audio Engineering Reports, and not ever there with the crazies at Audio Science Review.

 

[JSmith]

BTW, I recently started using this 1:4 (or 4:1) switchbox from Nobosound.

That looks like a helpful little device and a damn sight cheaper than this baby;

ABX Switch Comparator

Due to high demand we have produced another run of our ABX Switch Comparator. This unit allows seamless switching of up to two preamplifiers, two power amplifiers, and three sets of speakers either with or without subwoofers. The AVA ABX Switch can be used to evaluate cables and component parts...

avahifi.com

JSmith

 

[pkane]

Absolutely. That's why we are posting here on Audio Engineering Reports, and not ever there with the crazies at Audio Science Review.

Go ahead and point me to some scientific research that’s been conducted and published on ASR.

 

[Pdxwayne]

To be blunt, I care about my own listening test results, and assume that others do the same. If the result satisfied OP’s curiosity, then I’m perfectly fine with the result he got.

If OP is satisfied, why he wanted to do another round?

Aren't we supposed to help OP getting the best he can get? We shouldn't assume all OP will use is just one headphones forever.

Also, there are other questions.
For example, does OP like rocking out sometimes? Would he sense a difference between his 4 different chains using "rocking out" volume?
What type of songs OP likes? Did he already try all sorts of songs and all made no difference?

Why not asking more questions to help OP getting the best he can get?

 

[Axo1989]

Uh huh.