DAC audible differences are finally proven with ABX?

[kemmler3D]

Fair point - I'll amend my post accordingly.

To be clear, I don't think we can rule out dishonesty. I just think other explanations are plausible.

 

[Mort]

Let's all move along. Nothing to see here. Let's talk about tariffs or the aesthetics of buckeye amps.

 

[oleg87]

Amir might hate me for this time consuming suggestion…

…but it would be great if every DAC test had, along with its measurement results, a time aligned, level matched file of standard recordings?

A recording of what though? Or would it be to assuage the measurement-skeptical? In the general case, it's going to tell you exactly what the standard test panel tells you, as ultimately both are just recordings of the DAC, which the test panel exercises more thoroughly than a short musical extract is likely to.

 

[RexrothPigeon]

Let's all move along. Nothing to see here. Let's talk about tariffs or the aesthetics of buckeye amps.

just move on if you're no longer interested in the thread and let amir and @RickS mod the place

 

[PristineSound]

To be clear, I don't think we can rule out dishonesty. I just think other explanations are plausible.

Absolutely, dishonesty and forgery is all in the cards. I'm from New York, trust is earned by verification and proven reputation.

Having said that, innocent until proven guilty should still be the law of the land, well at least in my part of the world that is. Because that is something I would appreciate if I was in Josh's shoes.

Anyway, we shall find out soon enough and I am excited. BTW I nominate @NTTY to be an official part of the ASR staff, what rock star.

Oh and something great came out of this. An open source ABX device is being developed by @mcdn with the help of other technical experts.

 

[JustAnandaDourEyedDude]

The truth about the study is buried beneath the combined distortion floor of the testing protocol, the equipment, and the incompleteness of audible transparency of the verification and reporting. Noise occasionally creeps into the exchange of signals between the two sides.

 

[MattHooper]

Trying to convince those that don't even remotely comprehend scientific method is a lost cause.
If we all abided by the law of universal naive belief in wonders that some people here confuse with "scientific", we'd decommissioned power plants immediately a few years ago when lk-99 was invented.

Exactly. Hundreds of posts that would change nothing.

These seem to be referencing my posts here. (if I’m wrong, please correct me)

If so, that would be wildly off the mark. I’m very familiar with the scientific method - I have been defending it for years both on audio forums and elsewhere. (which is why in all my posts on ASR you won’t find anything that contradicts science, and plenty, which defends it). I wrote a bit about that here.

I was simply noting among members here the varied amount of analysis as to what may have gone wrong with Josh’s test, as well as what moves would be acceptable in order to verify and accept the results. I think a thread like this naturally raises interesting basic questions about science/blind testing and what justifies assent to the conclusion of any blind tests.

It includes the question of how justified any individual audiophile is in accepting the results of their own blind tests, when it makes sense and when it doesn’t.

And in the case of reporting blind tests, what justifies acceptance of the results by others.
These questions are bound into the nature of this thread.

So think of it from the perspective of audiophiles watching this thread who are less familiar with blind testing…and who might have similar questions.

Having done various blind tests over the years (including in the audio and video realm) and presented them online, I’ve seen varying reactions in terms of what people accepted. That includes when I have presented the results to communities that included EEs and people advocating for blind testing. Some folks provisionally accepted the results, others still rejected them.

That goes for this forum as well. For instance when I presented the results of my blind test between my tube and solid state preamps on ASR, there was a mix of people who remained sceptical and questioning of the method, and others who accepted the results. This is one reason I referenced the nature of scientific epistemology, where a certain level of personal judgement can end up playing a part, even among scientists.

So the questions I’ve asked or not due to my lack of understanding of science - as I said, I have my own views on answering the questions . I am interested in seeing the answers from other ASR members - how people might navigate some of these possibly gray areas (or if they don’t see any gray areas).

 

[Mort]

just move on if you're no longer interested in the thread and let amir and @RickS mod the place

You must be fun at parties. And your sarcasometer needs calibrating.

 

[amirm]

Amir might hate me for this time consuming suggestion…

…but it would be great if every DAC test had, along with its measurement results, a time aligned, level matched file of standard recordings?

Hate would not be the word for it. This is a lot of work and will cost a lot to put somewhere for people to access.

 

[solderdude]

Not to mention the used music would need to be 'revealing' and not have any copyrights (or be very short under fair use)

 

[MattHooper]

I believe that through his long association with the forum and his excellent contributions, @MattHooper has earned the right to civil and polite responses, and I say this despite my differing views on this specific topic.
I also want to answer his question.
If I do an experiment at home that fully confirms science as we know it, no one will bother to dismantle my test, because it went exactly as expected. If my experiment subverts known science, it could certainly be an exceptional result that raises human knowledge a notch, but precisely because of its exceptionality, I expect it to be tested in every possible way to confirm the result, because once confirmed, it forever changes a paradigm that was taken for granted. I imagine Josh knows this and should have foreseen it by agreeing to come here and contribute to the discussion.

Thanks. My own view generally conforms to what you’ve said. As I mentioned in my other post I tend to navigate this space via
“extraordinary claims require extraordinary evidence.” The tricky part for the audiophile who is not an electronics expert or engineer is recognizing extraordinary claims. I.e. highly implausible claims based on currently accepted science and engineering. And the problem is many audiophiles just don’t have the knowledge to recognize they are in the presence of an extraordinary claim. (which isn’t helped by the subjective-oriented audio sphere that makes extra extraordinary claims look like ordinary claims).

Since I’m not a scientist or EE myself, I’ve just tried to pay attention to the discussions and information provided by the experts, so that I at least have a grasp of the landscape of what would be an extraordinary claim in audio or not. Which really comes down to basic familiarity with the standard battleground high end audio controversies.

Easy: If the results of a blind test attempted at home by an individual contradict established science, then it's foolish to overturn our understanding of established science, which means the only reasonable course of action is to bring deep skepticism to bear on those results.

Generally agree.

Though I think it’s interesting to ask when somebody might rightly characterize something as contradicting established science. I’ve seen more than a few dubious leaps to those conclusions on ASR (certain folks have hallucinated that even I have stated things that contradict science, when it was nothing of this sort).

For instance, if Josh had only blind tested two DACs and had this result, with that have automatically been taken as contradicting science? Or with the possibility to be opened that he was able to hear a difference between a couple of DACs? It seems the sheer number of DACs tested with positive blind test results is mostly responsible for the immediate levels of skepticism.

One of the reasons why did my own blind tests of some CDPs and a DAC is it that I was on an audio discussion site in which some engineers were exceedingly sceptical about claims to distinguish between CD players/DACs. I understood the skepticism which is why I did my blind test and reported the results for critique. Did my results challenge existing science? I don’t think so.
Nor did some of the other members. But some still rejected the results as not budging what they held to be very improbable.
Individuals seem to have different lines here.

In my video I explained that there is a big difference between convincing yourself and the public. As you well see the bar is much higher for latter.

The issue here is that the minimum bar was not achieved for expert approval. We needed full analysis of the comparator and use of controls for the same -- what was stated at the start of this thread.

Ultimately not everyone will accept. So we aspire for good population not dismissing it out of hand.

Thanks for the reply!

Jesus Mat, a little of reading will answer the questions, but if you don't have time to read, a simple AI inquiry will do "To meet high scientific rigor in a blind audio test, you technically need a minimum of two distinct operational roles (an administrator/proctor and a listener), but a robust study design typically relies on an independent team of 3 to 4 separate personnel roles to achieve full double- or triple-blinding".
Which means if you are the investigator and subject the result are worthless. Good science is a lot of work.

Whoah…settle down pardner

Again, I’m familiar with blind testing; my aim was to understand the views of other members in answering these questions. And how they navigate some of the gray areas.

For instance, putting aside that another A.I. finds your A.I. answer to be a bit extreme
;-)…

If we take the criteria you just listed, then the blind tests/foobar results presented by Amir earlier would seem to fail that criteria.

And yet Amir clearly had some confidence in the results. And has seemed to express confidence in similar personal tests.

Should Amir put no stock whatsoever in those results and consider them totally worthless? Or is there some more nuance here?

As people keep saying… science isn’t easy
;-)

Its the lack of enough verifiable evidence... not the effort, not the results, not Josh

In case you were misconstruing that part of my reply: my reference to the complicated nature of Josh’s test, as well as the resulting technical commentary here, was not a suggestion this gave the test credence: it was in response to pma suggesting that I was the one complicating things - a snide remark that ignored that… in fact blind testing is pretty complicated, and the commentary here has been complicated. (including his own replies, telling people how complicated the issues are to untangle and do properly).

I agree with your commentary overall of a blind testing. Thank you.

When you think you have done everything correctly then you can/should accept the results.
Still does not mean everything was done correctly. That requires verification.

That’s a bit of the interesting tension IMO - when it is justified for the individual to accept the results versus others who are presented the results. You would think in principle it should be the same criteria. And in science in principal, it is supposed to be. But in practice, we are ultimately left to our own judgements about the epistemic weight one should give the evidence, the lines are not so clear. (That’s why there are still disagreements amongst scientists).

Maybe some people don’t like these gray areas. Prefer certainties especially to be able to say “it’s easy!….”

But especially if we’re going to recommend audiophiles tribe, blind testing for themselves, I think navigating some of these squishy areas - including when we have to acknowledge them - is relevant.

Cheers. And thanks all for the answers to my pesky questions!

 

[MattHooper]

Let's all move along. Nothing to see here. Let's talk about tariffs or the aesthetics of buckeye amps.

“ let’s shut down activity in a thread because I’m no longer interested in it.”

As opposed to

“ I’m not interested in this thread so I won’t bother visiting it anymore.”

I always find such posts to be pretty weird.
I guess some people just have to make sure we know they are bored.

 

[RexrothPigeon]

You must be fun at parties. And your sarcasometer needs calibrating.

I am very fun at parties, and the surefire way to spot sarcasm is that it's funny

 

[Mort]

I am very fun at parties, and the surefire way to spot sarcasm is that it's funny

Sigh. Ignored.

 

[mps]

Hate would not be the word for it. This is a lot of work and will cost a lot to put somewhere for people to access.

I agree that doing it for every DAC review would be a heavy lift, but would doing it once in the spirit of the 80/20 rule be doable/worthwhile?

 

[solderdude]

Anxiously awaiting Amir's test report and the plan from there on.

 

[NTTY]

Hi everyone,

As I’ve mentioned before, I like to involve my wife in my audio testing. She is always a remarkably willing participant. That said, she’s the equally remarkable Chief Financial Officer of our household, and she monitors the treasury
When the recent $3,000 transaction cleared, she delivered her feedback from her desk to the kitchen (I was cooking, yes)… It was a perfectly balanced, highly detailed and high-resolution "What the hell is this?!"

Immediately questioned, and explaining that it was "for audio science" didn't quite cut it, especially when she realized the money went toward funding measurements rather than a physical item for our home
Blaming @pablolie for setting the price floor with his initial $3k offer didn't help my case either!

To smooth things over, I prepared the little table below for my beloved CFO, with the actual real cash out and forecasted Year End spend (yeah, I learned her language, I had to):

Line item​	Amount​
VA Box​	$2 750.00​
Shipping​	$200.00​
@amirm (new owner)​	-$1 200.00​
@fivepast8​	-$400.00​
@WilliamfromParis​	-$245.00​
Anonymous donor​	-$50.00​
Anonymous donor​	-$50.00​
@pablolie​	-$825.00​
@gzost​	-$125.00**​
Total (NTTY) spend​	$0.00*​

*Updated to reflect latest contributions
**Contributed the same amount to Amir and me

In the spirit of the transparency we’ve all been calling for from the first page of this thread, I wanted to share these numbers with you too.

While I initially fronted nearly $3,000 to secure the box, my actual out-of-pocket cost is now down to $0!!!

I owe a massive thank you to @amirm (the true new owner of the ABX box), @fivepast8 , @WilliamfromParis, @gzost and two anonymous donors for stepping up to co-finance this. Special thanks to @pablolie who updated his contribution to make mine down to zero! I certainly will reinvest the money to bring some more reviews to the community!

On that note, I see a lot of people putting indirect pressure on Amir to run incredibly time-consuming battery of tests (recording, sharing, switching multiple DACs, etc.). Honestly, I don’t think that’s fair. My sole motivation for buying this box was to see if the switching hardware itself was the key to the blind-test results. Going far beyond that risks turning this into a textbook case of Brandolini’s law, something I explicitly said I wanted to avoid back on my first message in this thread, page 15.

Ultimately, I trust Amir to draw the line on what is actually relevant. But beyond my wife-CFO's concerns (now resolved), I feel uncomfortable seeing him inundated with demands from the sidelines.

For those who are highly eager to see more extensive testing done to advance audio science, I trust you will also be equally eager to step up and contribute one way or another, if you haven't yet.

Enjoy the weekend!

 

[PristineSound]

“ let’s shut down activity in a thread because I’m no longer interested in it.”

As opposed to

“ I’m not interested in this thread so I won’t bother visiting it anymore.”

I always find such posts to be pretty weird.
I guess some people just have to make sure we know they are bored.

I find it even more bizarre when someone tell others that they are ignoring them rather than actually ignoring them.

 

[Talisman]

Hi everyone,

As I’ve mentioned before, I like to involve my wife in my audio testing.

I hope she does them from the kitchen; everyone knows that certain tiny details are only noticed by wives from the kitchen.
(Sorry, I couldn't resist.)

 

[PristineSound]

For those who are highly eager to see more extensive testing done to advance audio science, I trust you will also be equally eager to step up and contribute one way or another, if you haven't yet.

I have made this offer in the name of science and as my small contribution to the community, it is incredibly risky for me. Not to mention what kind of tariff and import headache I have to deal with, or what kind of sales tax I will be on the hook for (and with no LLC nevertheless) or what kind of shit show hassle it will be on April 15th for me.

I hope my CFO, don't kill me, maybe it would have been a better idea to get clearance from her first, while I am in the kitchen with my apron on of course.