DAC audible differences are finally proven with ABX?

[Plcamp]

Assuming the test was properly conducted and the results are valid, how can we dismiss them as trivial and insignificant?
We thought DACs that measure well were audibly indistinguishable. If I turns out they aren't, this to me, seems to be pretty big news

Put another way…Until that assumption is validated, we must dismiss the results as ‘no news here’?

 

[AndyRoo]

I’m baffled that people here are treating this test as some kind of sinister agit-prop triumph for subjectivists that challenges the integrity of the audio science world and needs to be mobilized against.

Apologies if I gave that impression.

I don’t think it’s sinister and needs refuting. I think it’s interesting, statistically very compelling, and so needs proper verification.

I’m open to the idea it could be legit, however unlikely it is that anyone could identify such minor differences with almost unerring accuracy.

 

[solderdude]

Or maybe Cameron (GoldenSound) if he's heard about this. I think someone said he might have this comparator too.

I remembered incorrectly. He has a Skedra ABX box.

this video is 4 years old... doesn't look like he ever used it or did and could not tell a difference.

Then there is this video making claims to hear differences between DACs but this turned out to be differences between reconstruction filters.

This video is already 2 years old and there was a discussion he would repeat the test while being monitored remotely. Never happened of course.

 

[ppierre]

Frankly, he’s just a DAC reviewer publishing subjective opinions, which he backs up by "demonstrating" that he can tell them apart by ear.

https://audiophilestyle.com/ca/reviews/gustard-r30-discrete-r2r-dac-review-r1397/

Before beginning my subjective listening evaluation, I tested whether I could reliably discern the R30 from the Yggrasil LIM and XSP, respectively, by connecting them in pairs to my newly acquired Van Alstine ABX box. ...., I was able to score a perfect 8/8 in repeated ABX trials for both DAC pairs, suggesting that I’m able to reliably differentiate between the R30 from both the “Yggy” LIM and the Matrix XSP.

Those who want to believe him can gorge themselves on purely subjective opinions.

 

[mhardy6647]

All it takes is a loose connector somewhere to detect superluminal neutrinos after all. And yeah it causes a stir as it should.e

"The system" often works.
Not always, but often enough that we have, you know, safe and effective drugs, cell phones, controlled nuclear fission, radio and TV, space travel... those sorts of things.

 

[mhardy6647]

Contrary to the rather unfortunate thread title that DAC audible differences are finally proven with ABX, Josh was deliberately (and wisely) not claiming that - only that he had heard the differences with an ABX DBT. That's not the same thing, and there is no burden of proof on him to prove anything. You are free to ignore what he said if you wish.

So... someone with appropriate access might want to change the thread title.

And, no, I am not being facetious -- I am trying very hard to behave myself for a change, not because this thread or its content is a big deal, but understanding the scientific method is an increasingly big deal in the current cultural climate.

 

[pma]

Frankly, he’s just a DAC reviewer publishing subjective opinions, which he backs up by "demonstrating" that he can tell them apart by ear.

https://audiophilestyle.com/ca/reviews/gustard-r30-discrete-r2r-dac-review-r1397/

Quite typical, a “soft” subjective approach. Thus, we shall never get recorded files. This would be out of the scenario. The thread runs in circles.

 

[krabapple]

I don’t think what I wrote was condemning the discussion as negative or toxic, I just don’t feel this test has been convincing proved completely invalid yet. Lots of intense spitballing though!

Look at all those qualifier words!

 

[krabapple]

tl;dr: tough noogies, firedog. You're wrong.

But if you want to read:

Lots of fake amateur armchair science being proposed here.
Most work in science doesn’t challenge the established science, it backs up what’s established.

Are you suggesting 'science' should re-invent the wheel as part of every experiment?

Let’s say I present a minor paper to a reputable scientific journal. No established consensus is challenged, just some additional slightly different scientific perspective shown, say additional data not previously known.

A proper peer review of my paper would include examining the procedure, the use of statistics, and the citations in the paper, etc. etc. To say that experimental results that don’t challenge an established consensus don’t and aren’t examined, but are just accepted, is false.

That’s why the perspective here is incorrect.

Josh setup a serious test for himself and tried to run it in a serious way. The best he could do. He made no broad claims for his results and didn’t say they were exhaustive or a last scientific word.
Of course his test can be examined and improvements proposed.

It was a 'serious' test, in that he sincerely tried to bullet-proof his experiment.
It of course would not be accepted as is, in a decent peer reviewed journal.
Because of its extremely 'consensus challenging' findings, reviewers would pick every nit we've picked here. And probably say it needs to be a proctored DBT.
At best they'd send it back for revision.

But that's ok, we aren't a scientific journal. We are, however, 'peer reviewers' to Josh.

Some of us are even 'real scientists' and have experience with real submission of work to real peer reviewed journals.

 

[AndyRoo]

Josh setup a serious test for himself and tried to run it in a serious way. The best he could do

He published the results. That alone means he felt they were valid and thus invites scrutiny.

That’s not a criticism, either.

But subsequently resisting the inevitable scrutiny is not a good way to be taken seriously by a community that takes evidence and proof extremely seriously.

 

[krabapple]

Contrary to the rather unfortunate thread title that DAC audible differences are finally proven with ABX, Josh was deliberately (and wisely) not claiming that - only that he had heard the differences with an ABX DBT.

The Defense:
Your Honor, I heard a pretty amazing series of DAC difference in my ABX tests, and reported it to audiophiles on the Internet.

However, I'm not saying DACs may be discerned as audibly different.


The Prosecution:
Your Honor, I move to dub that line of argument 'hilariously legalistic'.

 

[krabapple]

Other than via an ABX DBT, how else would he (or anyone else) prove that there are audible differences between DACs?

The one way would be to measure the devices and find that the magnitude of the objective difference is within the known parameters of audibility.

 

[krabapple]

One could use an ABX ABT, but the point is that he isn't attempting to prove the differences to anyone.
Because he knew that all this would happen.

Does that read like audio science to you?

 

[Sernyl]

Exactly one person did this specific test…

… what does it matter..?

I was wondering the value of such an individual test, without any "external" control ...and other ears.
How can it be scientifically reproduceable ?
But I wonder more why there are so many posts here...

 

[krabapple]

I was wondering the value of such an individual test, without any "external" control ...and other ears.
How can it be scientifically reproduceable ?

Re-do it with more rigorous controls (positive and negative controls) and a proctor.
You can even have the subject be just Josh again.

That would be a 'more' scientific test of Josh's ability to hear the DAC differences he says he heard.

It's not likely publishable (n =1) but it would be scientific.

 

[Sernyl]

How many listeners should there be in such a test ?
How many locations ?
LOL

 

[hardisj]

Hey, folks. Been a while. I promised myself years ago no more forum posts because I always wind up spending way too much time posting and not enough getting other things done. But I think nearly 4 years is enough self-timeout time. Hopefully I can remember how to use the bbcode correctly.

Amir emailed me about this thread and asked about borrowing my ABX comparator to review/measure. We had a bit of discussion via email about this. I did give him my permission to provide my responses here but think it's worthwhile to share them myself because - as a user of this unit - I can provide my own experience.

With that said, get your comfy pillow under your butt and get ready for a long post...

Email 1:

Hi, Amir. It was good to meet you at AXPONA. I hope all is well.

I’ll start with the bad news first. I do indeed have the Van Alstine ABX unit and I understand why you’d be interested in testing this. But I have no desire to ship it anywhere. As you know, these are very hard to find and I’m not willing to risk something getting lost in shipment. Even though I know it’s unlikely to happen, I'm not willing to take that risk.

I’ll give you my two cents with respect to the ABX unit and Josh‘s article. Bear with me because this is going to be long-winded, but I have a point. I promise.

I’ve had my comparator for about three years now. I don’t use it often, but I consider it an essential part of my electronics reviews. In fact, I am currently using it to do some blind comparisons between a clone McIntosh amp and a real McIntosh amp.

While I have yet to use it to compare DACs, I have used it to compare preamps, amplifiers and speakers. What I can tell you is that when it comes to electronics the only time I’ve ever been able to reliably tell a difference is with an amplifier that has high output impedance.

I used it when I reviewed one of the newer WiiM amplifiers and compared it to the previous version without PFFB. I could hear a difference, but the data gave me insight as to why. I also used it to compare the NAD C3050 against one of Dayton’s hybrid tube amplifiers. There was definitely more “heft“ in the midrange with the Dayton. Looking at the frequency response with the speakers connected at the amplifier, it was pretty easy to tell why this was the case. The Dayton had had a boosted upper midbass and lower midrange.

The most audibly different test case, however, was when I compared a McIntosh MC 275 tube amplifier to a Prima Luna tube amplifier (I forget the model number). Both of these tests compared a low output impedance versus a high(er) output impedance amplifier. In short, the fact that I heard a difference - even in blind ABX testing - made perfect sense.

Worth noting is that the latter comparison - between the two tube amplifiers - was the hardest to perform in a meaningful way because the Prima Luna amplifier had such high output impedance that the frequency response shaped it as much as 3 dB in some regions, depending on the speaker was driving. While the McIntosh, with its incredibly low output impedance, was not low dependent, and thus the frequency response was unchanged. This made it practically impossible to do true “level matching”. I discussed this in a video I made last fall about why amplifiers*can* sound different.

In cases where I heard a difference, I was able to track down the difference with some basic measurements. However, outside of these cases where there is a perfectly reasonable explanation for the differences that were legitimately heard in ABX testing, I have yet to hear a difference in well engineered electronics.

I say all that to say this:
I am highly suspicious of Josh’s ABX test results. The article is very well written. And it seems like he took all the necessary steps to level match and set the test up the right way. But his result, with him getting practically 100% over 112 tests, seems extremely unlikely assuming everything is in order and indeed, the only variable is simply the DUT.

I don’t want it to sound like I am bashing Josh or his test/article. In fact, it’s one of the more thorough pieces I’ve read in years. It’s possible everything was done perfectly and the results wouldn’t change no matter how many times you ran the test. However, I think his results should’ve been questioned before being published. To get that kind of accuracy, regardless of how much ear training* or experience you have seems highly improbable. And that is putting it mildly.

* like Josh, I also use Sound Gym for my ear training.

It seems to me the more likely answer is that there’s an issue in the setup somewhere. I don’t know what that would be because as far as I can tell everything looks like it was done correctly. But without testing his exact unit or at least having the actual measurements from each of the DUTs outputs, we can only speculate that they’re is a set of issue.

And that leads me to this: testing my unit would not guarantee his is not flawed, if that’s the goal. Sure, if they are all a bit off spec then you could surmise that from testing mine. But mine is also the latest version which was manufactured in December 2022. We don’t know what version Josh’s is, do we? Let’s say mine shows a clean bill of health. That doesn’t mean that his isn’t faulty in some way.

As I said above, I think the best bet is for Josh to publish all of his data showing the measurements from each of the units and testing rounds. If nothing else, at least simple frequency response will tell him/community if there is any deviation there or output levels that might cause something. It’s also worth noting that doing this testing could be done to simply try to understand why differences were hurt. Not necessarily to Hoke holes in the methodology. I genuinely see this as a great opportunity for learning. Other learning that there is a flaw in the test set up and/or learning why there were differences heard.

Funny story, though, if I am not mistaken I was actually going buy the same box from the same gentleman in Washington. If that is the case, then his unit was an older model. But the current owner can test that by simply powering the unit on and reading the display on boot up.

— As an aside, I reached out a couple months ago to see if you would be interested in testing my McIntosh C55 stereo preamp before I sold it to a friend of mine. I sent you an email and a text but maybe I just have the wrong contact info for you. I’ll keep this one for future needs. —

TL;DR
I’m not interested in shipping my unit. It’s too risky for me.
Testing my unit would not guarantee the performance of Josh’s.
I think Josh‘s results are too good to be true and further analysis is needed.
I personally have no vested interest in his results other than the scientist in me wants to know they are indeed correct.

Take care,
Erin

PS: please forgive any grammatical errors. I’m using Siri and she still hasn’t figured out my southern accent.

PPS: You are more than welcome to copy and quote me on the forum if it saves you from having to summarize this email.

Email 2:

It's unfortunate he's not willing to do any further investigation.

I do have the QA402. I can provide some measurements of my ABX but I have no idea when that will be. I'm so busy with life that I have only completed two reviews in the last 5 months. I'm working on one today.

I am using version 3, dated 12/19/21. So it's at least that new. And it has the audible relay clacks. I even reached out to AVA and asked if these could be swapped for solid state relays. I have provided a screenshot of this email and quoted it below:
Unfortunately, the relay noise is unavoidable in the ABX box, and there is not much that can be done about it. I don't believe solid state relays are an option as I have never seen or heard of any that are a "drop in replacement" for a standard electro-mechanical relay. It is possible that the noise created by the relays is being amplified mechanically. If this is the case, you might try to isolate and/or insulate. The two things that come to mind immediately are: 1) try isolating the ABX box from what it sits on so no mechanical vibrations can pass from the ABX to some other entity. A fluffy towel under the ABX should tell you if this is going to make any difference or not. 2) Removing the cover and try covering the opened ABX with a towel. If this helps, perhaps insulation of some sort could be glued to the inside of the top cover Good luck

In other words, according to Dan - the designer of the ABX Comparator - there is NO VERSION that DOESN'T have audible relay clicks. Dan is the designer. He would know.
Maybe Josh's has been modified or maybe he happened to stuff a towel or something to quieten the relays in his. But it does not come with silent relays as per the designer.

I've also provided some photos - taken from my phone - for reference. Feel free to share these on the forum if you'd like.

- Erin

I think that covers it.

Thanks,
Erin

 

[pma]

Ad relay clicks, even if it was a SSR based, regardless that the smallest DC at the abx box input would lead to acoustically audible click from speakers or headphones when switching between the channels. So it goes.

 

[krabapple]

How many listeners should there be in such a test ?
How many locations ?
LOL

What answers would you consider to be NOT lolworthy?

 

[krabapple]

PS: please forgive any grammatical errors. I’m using Siri and she still hasn’t figured out my southern accent.

Siri did pretty well! The only glitches I noted were 'hurt' for 'heard', and "hoke' for "poke'

It’s also worth noting that doing this testing could be done to simply try to understand why differences were hurt. Not necessarily to Hoke holes in the methodology.