DAC audible differences are finally proven with ABX?

[Gorgonzola]

The test results are on page 83 here

Which pairs of amps fit your description?



But we can never test 'any and all conditions'. This is why statistics are used.
And btw, no one says amps sound the same under 'any and all conditions', in the first place.

The amp pairs where differences heard have a low(ish) chance of being random are:

1. Counterpoint vs. NAD
2. Futterman vs. Pioneer
3. Futterman vs. Hafler

Granted, the Futterman OTL amps are a very different design from the solid state amps compared. So sure, this where one may expect differences -- and there they are.

Re. the latter points: statistics only apply to the tests used, they are can't be reliable across different test protocols. As for "any and all conditions", of course not which is my point.

 

[krabapple]

The amp pairs where differences heard have a low(ish) chance of being random are:

1. Counterpoint vs. NAD

p=0.056 (0.06, to two decimal places). 'Lowish' but really, borderline.

1. Futterman vs. Pioneer
2. Futterman vs. Hafler

p>0.15 , well above 'standard' cutoff of 0.05

 

[pablolie]

How was the expertise established? There is no objective data on that. For that, we would need to know a truly audible artifact that was detected at very low level. All we have here is an all up set of results. Unless it is proven that audible differences exist, we can't jump to the conclusion that the author has above average discrimination.

I was being cynical I personally don't believe it at all.

 

[AndyBell]

What makes me hesitant to accept this test is:

1. The exceptionally high success rate when the op admits the differences were very subtle.
2. The reluctance to allow the test to be tested - the test equipment isn’t available for testing and recordings of the DAC outputs aren’t made available.

As it stands we just have to take the op’s word for it. And from a scientific pov that’s not good.

I have a background in drug trials and I’ve seen test results published and then withdrawn/discredited because no one was able to repeat them or the test protocol was found wanting.

As it stands, the op’s results deserve attention but are currently unfalsifiable because they can’t be scrutinised or repeated.

That doesn’t make them invalid but I can’t help feeling that if I had produced this test I’d be eager to get it ratified and the results confirmed.

As it stands some are likely to accept it at face value and may make expensive purchases based on it. So this isn’t a mere academic exercise. It affects people.

 

[amirm]

FYI, Erin responded to my request and is just as well worried about loaning me his AVA comparator as well.

So at this point, we are at the end of the road in trying to understand these results.

 

[amirm]

I went to AS site to see if there is more detail there (there was not) but ran into this from @firedog:

"At the ASR thread, they are being polite because it's clear Josh is a serious guy and did a serious test. But they will look for any possible reason to say the test is flawed.

As expected, if you get the results they say are impossible, the test must be flawed. You will never win there, no matter what test you do and what the conditions are.

I bet if Josh had gotten their expected result, they'd trumpet his test as proof."

You best not generalize such things. For me personally, it would be beneficial to point to a listening test where differences were found when I get the routine question of "why do you test another DAC because they all sound the same."

But facts are facts and this test, despite the reported effort that has gone into it, has generated questionable results. Our inability to probe further and get responses to our questions puts more doubt on the work. So regardless of whether I "like" the results, I want to make sure the data is reliable.

I mean look at this study of reconstruction filters in peer reviewed paper from Stuart, et. al.: “The audibility of typical digital audio Filters in a high-fidelity playback system,” Convention Paper, Presented at the 137th AES Convention 2014

While the results had statistical power (dashed line is p = 0.05), respondents did not remotely achieve near 100% outcome identifying differences between filters. Best case percentage was just 65% compared to near 100% we have in this report.

I personally have defended the results of my passing of such tests numerous times across many years, agreeing to more tests, discussion, etc. Yet we have reached the end of this road in a couple of days with no additional insight.

I find it puzzling and disappointing and the author wants to continue to use the box for other listening tests while seemingly not worried that it could be generating false results.

 

[kemmler3D]

1. recordings of the DAC outputs aren’t made available.

As it stands we just have to take the op’s word for it. And from a scientific pov that’s not good.

Yes, even if we have no doubt in OP's honesty and report, not having the recordings means we can't really hunt for the "why" here, which is honestly the only really interesting part.

Accepting this result with no further info means all I know is some guy out there has really exceptional hearing (congenitally, in terms of technique, or both...). That's very nice for him, but...

Whoop de doo, that doesn't advance the state of the art, DAC makers can't use that knowledge, Amir can't devise a better measurement suite, we can't make better choices in buying DACs... we just have to hand this guy the gold medal and call it a day?

It's like... "Hey, I just ran a 3-minute mile. Here are my lap times."

"WHAT? HOW??"

"Ah, sorry, gotta get back to work now, bye"

"AHHHHHH!"


All to say I hope OP finds the time to assist in the "how" questions at some point.

Again, assuming all is kosher, my worldview has shifted this much: All DACs sound alike except to this one dude, he needs to shop around and the rest of us can keep looking at measurements.

I went to AS site to see if there is more detail there (there was not) but ran into this from @firedog:

"At the ASR thread, they are being polite because it's clear Josh is a serious guy and did a serious test. But they will look for any possible reason to say the test is flawed.

As expected, if you get the results they say are impossible, the test must be flawed. You will never win there, no matter what test you do and what the conditions are.

I bet if Josh had gotten their expected result, they'd trumpet his test as proof."

You best not generalize such things. For me personally, it would be beneficial to point to a listening test where differences were found when I get the routine question of "why do you test another DAC because they all sound the same."

But facts are facts and this test, despite the reported effort that has gone into it, has generated questionable results. Our inability to probe further and get responses to our questions puts more doubt on the work. So regardless of whether I "like" the results, I want to make sure the data is reliable.

I mean look at this study of reconstruction filters in peer reviewed paper from Stuart, et. al.: “The audibility of typical digital audio Filters in a high-fidelity playback system,” Convention Paper, Presented at the 137th AES Convention 2014

While the results had statistical power (dashed line is p = 0.05), respondents did not remotely achieve near 100% outcome identifying differences between filters. Best case percentage was just 75% compared to near 100% we have in this report.

I personally have defended the results of my passing of such tests numerous times across many years, agreeing to more tests, discussion, etc. Yet we have reached the end of this road in a couple of days with no additional insight.

I find it puzzling and disappointing and the author wants to continue to use the box for other listening tests while seemingly not worried that it could be generating false results.

Right, firedog's objection doesn't cut that deep...

If one more experiment comes along that says protons still haven't decayed in the lab, nobody will check their methods too carefully.

If someone reports observing proton decay they should expect to have their dental fillings X-rayed individually, lab torn down and rebuilt twice, etc. If the result is confirmed, we throw a parade.

Same principle applies here.

I think that attitude is also a bit unfair. I am not hoping someone overturns OP's result. I'm hoping it turns out to be real and we learn something new about audio equipment. But until all details have been confirmed, (and possible causes identified and discussed) I don't know anything new.

I don't come to ASR hoping nothing changes and audio stays the same forever, I look forward to news that shows a way to actually raise the bar. After all, how could I justify buying new gear if it all sounds the same as last year's model?

 

[mhardy6647]

The reported results aren't impossible -- they're just extremely unlikely, which absolutely mandates some looking into.
It's a bit like UFOs -- I mean UAPs, in that respect.

 

[krabapple]

'Trust but verify' operates in this case. The OP has given us no reason to mistrust him (i.e., question his sincerity), but his results are so extraordinary, such distant outliers compared to all the 'priors', for all the reasons stated so far, that verification is simply *required*. That's absolutely the way it would and should be if this was a scientific report in, say, Nature. See: cold fusion.

tl;dr: tough noogies, 'firedog'.

 

[pjug]

FYI, Erin responded to my request and is just as well worried about loaning me his AVA comparator as well.

So at this point, we are at the end of the road in trying to understand these results.

Did Erin express any interest in measuring the comparator himself?

 

[Cbdb2]

There's no proof of that even existing, so... Why assume something like that? The output matters. The measurements capture the output. Why should we care about output stages, capacitors, op amps or whatever else if the resulting performance is essentially identical? It's a repeating discussion even here on ASR which I do not understand. I have seen no logical argument which supports it. If the end result at the output RCA/XLR terminals is identical, why focus on the parts inside the little black box? The output is the only thing which matters for the actual sound, because that is what we listen to. Everything else is only relevant to other qualities like longevity.

Well this is what the BSaudio site says about there $8k DAC: "Paul had the final say on voicing, ensuring the end product met his sonic vision."

 

[mlsstl]

The goal in science is to have others duplicate your findings. History is full of examples where a seemingly brilliant finding could never be confirmed by anyone else. One person working alone and self reporting the results is meaningless until confirmed by others.

 

[pjug]

@Josh83 Did you take the photos used in your article yourself?
edit: Answered on AS. Chris added the photos not Josh.

 

[Wesseling Audio Lab]

He starts with a story - "Craigslist", blah,blah,blah. "my man in Seattle", blah,blah,blah.

Right from start, I have zero interest and go no further! No confidence in what comes after that

 

[Roland68]

".. The “worst” I could be off was 0.1 dB, and in most cases REW showed the two DACs matched more closely than that .."

I have heard some objectivists on this site claim 0.1dB can be hearable. Maybe they were right... (but I doubt it).

Myself, I don't recall anyone saying that. On the contrary, I've only seen the 0.1 dB value being recommended as the sufficient level of volume matching.

At an output level of 2 volts, 0.1 dB corresponds to 0.023 volts.
That is reasonably precise for a comparison; however, in our comparative tests, we aim for the highest possible accuracy at the third decimal place—that is, at least 0.00X.

 

[Ropeburn]

sonic vision

 

[RandomEar]

He starts with a story - "Craigslist", blah,blah,blah. "my man in Seattle", blah,blah,blah.

Right from start, I have zero interest and go no further! No confidence in what comes after that

I think that is unfortunate. The article is well written overall and though it clearly does cater to the subjective crowd to a degree, the testing itself was thorough. After all, the article was written for a website which needs to generate clicks to make money, so it has to appeal to a broad audience.

Discounting all the effort because of one or two sentences in the introduction seems premature.

 

[killdozzer]

Well this is what the BSaudio site says about there $8k DAC: "Paul had the final say on voicing, ensuring the end product met his sonic vision."

This is sooo funny! So, first it was accurate, but then an old guy with somewhat spent hearing came in and "voiced" it!

 

[pjug]

At an output level of 2 volts, 0.1 dB corresponds to 0.023 volts.
That is reasonably precise for a comparison; however, in our comparative tests, we aim for the highest possible accuracy at the third decimal place—that is, at least 0.00X.

I think the device attenuated in discrete steps, so he was limited by that, but still got better than 0.1 dB matching.

 

[pjug]

I listed the songs in the article: two Van Morrison songs (“Rough God Goes Riding” and “Ancient Highway”), two Steely Dan tracks (“Negative Girl” and “Cousin Dupree”), one Jenny Lewis song (“Head Underwater”), and a transient-rich test signal, “Drum Solo Stereo,” from the Audio Check album available on Qobuz.

Is anyone able to check whether these tracks can have intersample overs? I don't really think it is the problem, but it is one thing that can at least be checked.