DAC audible differences are finally proven with ABX?

[MattHooper]

There is a standard HR corporate line is that we don't pay for effort but results.

OK, I’ll try again, as I’m quite interested in your answer:

If somebody follows the protocol you suggest in your video about blind testing and gets positive results for identifying between the components, should they accept the results?

If so… why and if not, why not?

Are there some blind tests in which the audiophile testing at home should accept positive results? And other tests in which they should only accept one result… a negative result?

 

[Salt]

If you don’t like essays, here’s the short form:

For any particular claim, or blind test, how much is enough in order to accept the results?

And for individuals doing blind tests at home:

is it ever justified to accept the results of a blind test we individuals attempt at home? And on what grounds?

This would be reasonable and relevant questions. Don’t you agree? I’m interested in your thoughts.

Cheers

Thanks for reply.
For the first Bolt: don't understand what ' how much' is defined.
As for foobar, it is 16 trials.
For the second Bolt: all blind tests, performed to accepted means of performance, should be acceptable, not?
Just proof that.

 

[Talisman]

I believe that through his long association with the forum and his excellent contributions, @MattHooper has earned the right to civil and polite responses, and I say this despite my differing views on this specific topic.
I also want to answer his question.
If I do an experiment at home that fully confirms science as we know it, no one will bother to dismantle my test, because it went exactly as expected. If my experiment subverts known science, it could certainly be an exceptional result that raises human knowledge a notch, but precisely because of its exceptionality, I expect it to be tested in every possible way to confirm the result, because once confirmed, it forever changes a paradigm that was taken for granted. I imagine Josh knows this and should have foreseen it by agreeing to come here and contribute to the discussion.

 

[MattHooper]

Thanks for reply.

No, thank you. I understand why some are put off my longer posts.

For the first Bolt: don't understand what ' how much' is defined.
As for foobar, it is 16 trials.
For the second Bolt: all blind tests, performed to accepted means of performance, should be acceptable, not?

By that criteria, Josh would’ve been justified in accepting the results of his own ABX tests wouldn’t he?

(justified… doesn’t mean they are correct. People are giving the test reasonable scrutiny here).

I scored 17 out of 18 guesses correct in my own blind test of a CDP vs DAC (which followed the criteria set out by Amir). That would satisfy the criteria as well?

Just proof that.

Not sure I am clear about that. The tests themselves are supposed to be the proof aren’t they? Or do you mean something else?

 

[Salt]

No, thank you. I understand why some are put off my longer posts.



By that criteria, Josh would’ve been justified in accepting the results of his own ABX tests wouldn’t he?

(justified… doesn’t mean they are correct. People are giving the test reasonable scrutiny here).

I scored 17 out of 18 guesses correct in my own blind test of a CDP vs DAC (which followed the criteria set out by Amir). That would satisfy the criteria as well?



Not sure I am clear about that. The tests themselves are supposed to be the proof aren’t they? Or do you mean something else?

The tests are there for proof.
But not a singular one. You need to run several ones to come to a conclusion.
Math with statistics can hurt ...

 

[kemmler3D]

And that is the issue hovering over all of this:

How much is enough?

For any particular claim, or blind test, how much is enough in order to accept the results?

We are trying to maintain a scientific mindset, but of course it’s the case that there’s the important distinction between “the evidence is conclusive” and “the evidence is sufficient to rationally accept the conclusion for now.” Science almost never gives us the former in the absolute sense. There is no point at which science normally says “all possible objections have been eliminated.”

A lot of science consists of peers trying to find weaknesses in each other’s work.
You’ll have scientist who find the evidence from a certain study to be persuasive, others who think the evidence is suggestive but insufficient, others who hold that there’s likely a methodological flaw that substantially undermines the conclusion.

So individual scientists and communities are stuck making judgments about how much evidential weight should be assigned to the available evidence. They can be rational judgments, but there’s still often an element of subjectivity as to where one grants assent to the conclusions.

And you can see that playing out in this thread. There’s a variety of suggestions as to what it would take to justify accepting the results of Josh’s test right down to sending the ABX box, all his cabling and all the DACs to ASR.

You’ve got folks like pablolie who has said essentially he rejects the results out of hand, not even interested in exploring the results.

There’s PMA’s response to Josh’s test: “I am not much interested in his results, even if they were 100/100 success. The reason is, as I have mentioned many times, that his test conditions cannot be independently verified and repeated.”

Some seem to think measuring the ABX box will be enlightening, others seem to think it still doesn’t guarantee it will allow us to rule for or against Josh’s results.

It doesn’t seem surprising an audiophile might conclude: “ OK I followed these recommended steps to do a blind test and I got my results, I’m going to decide how much creedence to put in the results. And I also understand that if I present my results to others, there’s always going to be someone who might accept the results and others who reject the results as undemonstrated, because somebody can always come up with something more that could’ve gone wrong, or more stringent requirements in order for them to accept the results. So I’m going to draw my own line.” And that the line one draws is going to be different from the lines drawn by other people here.

And on the theme of an individual audiophile doing blind testing:

Before I did my last blind test between my preamplifiers I watched Amir’s excellent video on how to do blind testing, which included instructions for people to try blind testing at home. (It’s also my impression that the protocols Amir is willing to accept or suggest in that video are allowed to be somewhat looser than the level of scrutiny being brought Josh’s test).

@amirm , You mentioned in the “how to repeat the tests” portion about how one can do blind randomized, switching between two different audio items, and you conclude:

Compare the two and see how many you got right, and “ if you didn’t get nine out of 10 right, I’d say you just learned something important about audio that you didn’t know.”

But what if you did get 9 out of 10 right? Did you also just learn something about audio? Presumably positive results count as well.

Let’s say somebody had blind swapped between two different AC cables and they got nine out of 10 guesses right. Did they just learn something about audio?

We would agree that on technical grounds those results would be very implausible, and would more likely point to a flaw in the test.

But this would mean that the individual would need that type of wider context knowledge to know how much epistemic weight to give their results. Successful blind test results are not enough. (I would use the heuristic “extraordinary claims require extraordinary evidence” to navigate this space, but of course that takes some knowledge of what amounts to an extraordinary claim in audio engineering and psychoacoustics).

And then you start to get the gray areas…. Differences between amplifiers, especially tube amplifiers, DACs….

Did I for instance learn that (years ago) in using the type of protocol you suggested in your video, where my results for identifying between CDPs and a DAC were well beyond chance…that I should accept my results? Did I learn the truth that they really sounded different?

If I present the methods here and there’s nothing that stands out to others indicating a flaw people can put their finger on, folks can still reject the results because there may be some hidden flaw I wasn’t aware of and is not reported properly. (which is a properly scientific mindset…. It’s the reason for other parties trying to replicating results),

That being the case is it ever justified to accept the results of a blind test we individuals attempt at home?

(this is very far from a slippery-slope throwing up of hands and saying “ therefore how can any scrutiny ever settle any claim?” I have my own views of how to answer some of these questions, but I’m looking to see what other members here have to say).

The real question is not whether you accept Josh's results or not, or whether anyone's results should be accepted, but how did those results actually come about?

When the test shows no difference and you don't expect one, the answer is easy.

In the reverse scenario we need more info. This is true for anyone that gets 9/10 or 19/20 or whatever. Oh, I heard something - let's run it down.

We're 1400 comments in and not much closer to accomplishing that.

To put it another way, it is not really incumbent upon me to say whether I believe the results or not or to put my faith anywhere. Josh's credibility and trustworthiness and motivations are not really my business. But I am definitely interested in the "why". I would like to know how Josh got his results with as much detail as we can muster. I am fully prepared to speculate with the best of them, but that's not very useful.

 

[krabapple]

We started from 'trust, but verify' when we learned of an utterly *amazing* result, reported with apparent sincerity.

We aren't being allowed to verify.

Should we continue to trust?

 

[tmtomh]

The real question is not whether you accept Josh's results or not, or whether anyone's results should be accepted, but how did those results actually come about?

When the test shows no difference and you don't expect one, the answer is easy.

In the reverse scenario we need more info. This is true for anyone that gets 9/10 or 19/20 or whatever. Oh, I heard something - let's run it down.

We're 1400 comments in and not much closer to accomplishing that.

Indeed - and this is the issue. I think we all know what's going on here:

• @Josh83 conducted the test and not only did he get a result that is unexpected, but the result was quite extreme - 15/16 or 16/16 on every run (if memory serves).
• Some are deeply skeptical in a "suspicion" kind of way.
• Others are skeptical in more of a "let's look into this further and see if we can figure out the cause or at least exhaust most of the possibilities" kind of way.
• Others - a little bit here but mainly over at AudiophileStyle and in the broader audio internet community - take the above skepticism as a supposedly hypocritical anti-scientific attitude because ASR folks are supposedly "unwilling to accept the science when it doesn't reinforce their beliefs." (I do not agree with this; I am just trying to summarize)
• And to complicate matters further, Josh's refusal to send Amir the unit, record and share the actual test output of the DACs, and (apparently semi-humorously but also semi-seriously) ask an exorbitant price for someone to purchase the unit, reinforces the suspicions of those who harbor suspicions as to his veracity or level of good faith. And conversely, those same escalating expressions of frustration and suspicion directed towards Josh tend to harden and reinforce what I'll call the "anti-ASR" folks' view that ASR members are just gatekeeping objectivist orthodoxy and not truly open to scientific results.

The underlying problem, as I see it, is that such an extraordinary result as Josh has gotten here simply will not be taken at face value by a lot of ASR members unless it is validated with a degree of rigor and thoroughness one would expect for a peer-reviewed scientific paper. There's nothing wrong with that - it's not @amirm's or anyone else's fault that Josh has not provided sufficient evidence and documentation to objectively validate his results - we can't relax the standard for determining objective scientific facts just because it might be socially awkward or acrimonious to keep insisting on such standards. Likewise, it's not Josh's fault that he never set out to provide results that could get published in a peer-reviewed scientific journal - he's allowed to run an ABX test and write about the results.

I think the best we can do is try to drain some of the blood out of this and hold open the possibilities that either (a) these DACs produce consistent, very small differences in output that most of us cannot hear and that appear to be insignificant for the purposes of musical enjoyment or perceived fidelity during everyday hi-fi listening (even critical listening; or (b) something in the test and equipment that Josh conducted resulted in the "tell" (or tells) that he says he detected, which enabled him to ace the test; or (c) the output of the DACs is for all intents and purposes identical and Josh either lied about the results (which I personally find highly unlikely) or heard differences produced by some aspect of the gear or the test protocol that is not part of what gets recorded (also unlikely in my view). Personally I'd rank (b) as most likely and (a) as second most likely, but hey, what do I know?

 

[RexrothPigeon]

If you don’t like essays, here’s the short form:

For any particular claim, or blind test, how much is enough in order to accept the results?

And for individuals doing blind tests at home:

is it ever justified to accept the results of a blind test we individuals attempt at home? And on what grounds?

This would be reasonable and relevant questions. Don’t you agree? I’m interested in your thoughts.

Cheers

Easy: If the results of a blind test attempted at home by an individual contradict established science, then it's foolish to overturn our understanding of established science, which means the only reasonable course of action is to bring deep skepticism to bear on those results.

 

[pablolie]

Indeed - and this is the issue. I think we all know what's going on here:

• @Josh83 conducted the test and not only did he get a result that is unexpected, but the result was quite extreme - 15/16 or 16/16 on every run (if memory serves).
• Some are deeply skeptical in a "suspicion" kind of way.
• Others are skeptical in more of a "let's look into this further and see if we can figure out the cause or at least exhaust most of the possibilities" kind of way.
• Others - a little bit here but mainly over at AudiophileStyle and in the broader audio internet community - take the above skepticism as a supposedly hypocritical anti-scientific attitude because ASR folks are supposedly "unwilling to accept the science when it doesn't reinforce their beliefs." (I do not agree with this; I am just trying to summarize)
• And to complicate matters further, Josh's refusal to send Amir the unit, record and share the actual test output of the DACs, and (apparently semi-humorously but also semi-seriously) ask an exorbitant price for someone to purchase the unit, reinforces the suspicions of those who harbor suspicions as to his veracity or level of good faith. And conversely, those same escalating expressions of frustration and suspicion directed towards Josh tend to harden and reinforce what I'll call the "anti-ASR" folks' view that ASR members are just gatekeeping objectivist orthodoxy and not truly open to scientific results.

The underlying problem, as I see it, is that such an extraordinary result as Josh has gotten here simply will not be taken at face value by a lot of ASR members unless it is validated with a degree of rigor and thoroughness one would expect for a peer-reviewed scientific paper. There's nothing wrong with that - it's not @amirm's or anyone else's fault that Josh has not provided sufficient evidence and documentation to objectively validate his results - we can't relax the standard for determining objective scientific facts just because it might be socially awkward or acrimonious to keep insisting on such standards. Likewise, it's not Josh's fault that he never set out to provide results that could get published in a peer-reviewed scientific journal - he's allowed to run an ABX test and write about the results.

I think the best we can do is try to drain some of the blood out of this and hold open the possibilities that either (a) these DACs produce consistent, very small differences in output that most of us cannot hear and that appear to be insignificant for the purposes of musical enjoyment or perceived fidelity during everyday hi-fi listening (even critical listening; or (b) something in the test and equipment that Josh conducted resulted in the "tell" (or tells) that he says he detected, which enabled him to ace the test; or (c) the output of the DACs is for all intents and purposes identical and Josh either lied about the results (which I personally find highly unlikely) or heard differences produced by some aspect of the gear or the test protocol that is not part of what gets recorded (also unlikely in my view). Personally I'd rank (b) as most likely and (a) as second most likely, but hey, what do I know?

There are more sightings of the Loch Ness monster or Sasquatch that there are of audibility superpowers such as Josh. Even so, Nessie and Sasquatch face a hard time gaining any credibility.
Being feeble-minded and buying into any improbable claim a-priori is NOT the way of science, anyone that claims that has an iceberg salad occupying their entire brain cavity.

 

[antcollinet]

For any particular claim, or blind test, how much is enough in order to accept the results?

For a test whose results seems to fly in the face of what we know about what the human auitory system can detect, then quite a lot. Certainly you'd think providing recorded test files and availability of the test box to independent measurements would be an absolute bare minimum.

is it ever justified to accept the results of a blind test we individuals attempt at home? And on what grounds?

If someone does a blind test at home then it is up to them how much credence they themselves give it. It is probably not justified for anyone else to accept vanishingly unlikely results without quite a bit of validation.


Certainly if I'd got a result along the lines of Josh's results, I'd be on here asking the great and the good WTF it is that I'm doing wrong. I'd certainly not be announcing it anywhere as some new, and as to now unknown to engineering-and-psychoacoustics achievement.


EDITED to add:

I scored 17 out of 18 guesses correct in my own blind test of a CDP vs DAC (which followed the criteria set out by Amir). That would satisfy the criteria as well?

In that case if you got an expert to review your test method, and no faults were found, then the next step would be to measure the two devices to find the performance deviation that could explain the result. You could - I guess - swap those two activities round in the timeline - depending on the relative ease of each.

Absent either/both of these, then I'd suggest the result is only of interest to yourself. Not because it is uninteresting as a discussion, but because it is impossible to know if you heard real differences, or irrelevant (to the purpose of the test) tells.

 

[Plcamp]

There’s PMA’s response to Josh’s test: “I am not much interested in his results, even if they were 100/100 success. The reason is, as I have mentioned many times, that his test conditions cannot be independently verified and repeated.”

I see no defect in this logic. It really is just that simple IMO.

 

[amirm]

If somebody follows the protocol you suggest in your video about blind testing and gets positive results for identifying between the components, should they accept the results?

In my video I explained that there is a big difference between convincing yourself and the public. As you well see the bar is much higher for latter.

The issue here is that the minimum bar was not achieved for expert approval. We needed full analysis of the comparator and use of controls for the same -- what was stated at the start of this thread.

Ultimately not everyone will accept. So we aspire for good population not dismissing it out of hand.

 

[PristineSound]

I believe that Josh is under no obligation to loan out or sell his ABX. And he does not need to provide any reason why. Even though I am glad that he finally agreed to sell it, and I believe it's for the better for everyone, including himself, despite Josh is selling it under immense pressure.

Also, we are all so spoiled by Amir running ASR all for free, we have to keep in mind that if Josh wanted to profit from his ABX device under current circumstances, as distasteful as it may be, he is allowed (at least in the country that he is under the jurisdiction of).

Josh is also under no obligation to provide any files, but it does hurt him as providing the files doesn't take much of his time and it doesn't really cost him much nor are there any risk to any of his equipment. By not providing the files, he is only hurting his own credibility, but again, he is under no obligation to do anything for anyone.

All of that does justify members raising an eyebrow with his test results. Although, I'm not one to raise my eyebrow at this moment due to the need for further investigation.

At this point, I think all of this discussion around Josh is moot, as we got one of the best in the business who will be doing further investigation; Amir is getting the ABX comparator. And hopefully an open source ABX comparator will be available for the masses to conduct repeatable tests.

 

[antcollinet]

I believe that Josh is under no obligation to loan out or sell his ABX. And he does not need to provide any reason why. Even though I am glad that he finally agreed to sell it, and I believe it's for the better for everyone, including himself, despite Josh is selling it under immense pressure.

Also, we are all so spoiled by Amir running ASR all for free, we have to keep in mind that if Josh wanted to profit from his ABX device under current circumstances, as distasteful as it may be, he is allowed (at least in the country that he is under the jurisdiction of).

Josh is also under no obligation to provide any files, but it does hurt him as providing the files doesn't take much of his time and it doesn't really cost him much nor are there any risk to any of his equipment. By not providing the files, he is only hurting his own credibility, but again, he is under no obligation to do anything for anyone.

All of that does justify members raising an eyebrow with his test results. Although, I'm not one to raise my eyebrow at this moment due to the need for further investigation.

At this point, I think all of this discussion around Josh is moot, as we got one of the best in the business who will be doing further investigation; Amir is getting the ABX comparator. And hopefully an open source ABX comparator will be available for the masses to conduct repeatable tests.

Obligation? Of course not. But why the hell wouldn't he want to. He must have been aware of the controversy his post would create. You'd think he'd have all his ducks lined up just ready to prove its validity.

But no, instead, active resistance.

Smacks of either dishonesty, or, at best, lack of confidence in his own test result.

EDIT - more likely, as pointed out by @kemmler3D, a massive expectation mismatch between camps.

 

[PristineSound]

Obligation? Of course not. But why the hell wouldn't he want to. He must have been aware of the controversy his post would create. You'd think he'd have all his ducks lined up just ready to prove its validity.

But no, instead, active resistance.

Smacks of either dishonesty, or, at best, lack of confidence in his own test result.

I'm in agreement.

 

[Madlop26]

If you don’t like essays, here’s the short form:

For any particular claim, or blind test, how much is enough in order to accept the results?

And for individuals doing blind tests at home:

is it ever justified to accept the results of a blind test we individuals attempt at home? And on what grounds?

This would be reasonable and relevant questions. Don’t you agree? I’m interested in your thoughts.

Cheers

Jesus Mat, a little of reading will answer the questions, but if you don't have time to read, a simple AI inquiry will do "To meet high scientific rigor in a blind audio test, you technically need a minimum of two distinct operational roles (an administrator/proctor and a listener), but a robust study design typically relies on an independent team of 3 to 4 separate personnel roles to achieve full double- or triple-blinding".
Which means if you are the investigator and subject the result are worthless. Good science is a lot of work.

 

[solderdude]

Ha.

If you survey all the analysis of Josh’s results, all the questions and technical discourse and various opinions on what would be required to accept the results … it ain’t me who’s making things complicated.

Josh went through quite a lot of complicated effort to do his test… and it still wasn’t enough, right?

Its the lack of enough verifiable evidence... not the effort, not the results, not Josh

If you don’t like essays, here’s the short form:

For any particular claim, or blind test, how much is enough in order to accept the results?

Controls checked, statistics, repeated, witnessed, everything well documented, verifiable.

And for individuals doing blind tests at home:

is it ever justified to accept the results of a blind test we individuals attempt at home? And on what grounds?

When you think you have done everything correctly then you can/should accept the results.
Still does not mean everything was done correctly. That requires verification.

OK, I’ll try again, as I’m quite interested in your answer:

If somebody follows the protocol you suggest in your video about blind testing and gets positive results for identifying between the components, should they accept the results?

If so… why and if not, why not?

Are there some blind tests in which the audiophile testing at home should accept positive results? And other tests in which they should only accept one result… a negative result?

The correct following of the protocol needs to be verified by people that can check if everything was done correctly for the test results to be accepted as valid.

What's lacking here is the independent verification of the test being done 'properly'.
It might have been done properly, looks to be done thoroughly but the procedure and test results are not verified independently.
The test was not repeated nor witnessed and not enough evidence is presented.

 

[Mnyb]

Indeed - and this is the issue. I think we all know what's going on here:

• @Josh83 conducted the test and not only did he get a result that is unexpected, but the result was quite extreme - 15/16 or 16/16 on every run (if memory serves).
• Some are deeply skeptical in a "suspicion" kind of way.
• Others are skeptical in more of a "let's look into this further and see if we can figure out the cause or at least exhaust most of the possibilities" kind of way.
• Others - a little bit here but mainly over at AudiophileStyle and in the broader audio internet community - take the above skepticism as a supposedly hypocritical anti-scientific attitude because ASR folks are supposedly "unwilling to accept the science when it doesn't reinforce their beliefs." (I do not agree with this; I am just trying to summarize)
• And to complicate matters further, Josh's refusal to send Amir the unit, record and share the actual test output of the DACs, and (apparently semi-humorously but also semi-seriously) ask an exorbitant price for someone to purchase the unit, reinforces the suspicions of those who harbor suspicions as to his veracity or level of good faith. And conversely, those same escalating expressions of frustration and suspicion directed towards Josh tend to harden and reinforce what I'll call the "anti-ASR" folks' view that ASR members are just gatekeeping objectivist orthodoxy and not truly open to scientific results.

The underlying problem, as I see it, is that such an extraordinary result as Josh has gotten here simply will not be taken at face value by a lot of ASR members unless it is validated with a degree of rigor and thoroughness one would expect for a peer-reviewed scientific paper. There's nothing wrong with that - it's not @amirm's or anyone else's fault that Josh has not provided sufficient evidence and documentation to objectively validate his results - we can't relax the standard for determining objective scientific facts just because it might be socially awkward or acrimonious to keep insisting on such standards. Likewise, it's not Josh's fault that he never set out to provide results that could get published in a peer-reviewed scientific journal - he's allowed to run an ABX test and write about the results.

I think the best we can do is try to drain some of the blood out of this and hold open the possibilities that either (a) these DACs produce consistent, very small differences in output that most of us cannot hear and that appear to be insignificant for the purposes of musical enjoyment or perceived fidelity during everyday hi-fi listening (even critical listening; or (b) something in the test and equipment that Josh conducted resulted in the "tell" (or tells) that he says he detected, which enabled him to ace the test; or (c) the output of the DACs is for all intents and purposes identical and Josh either lied about the results (which I personally find highly unlikely) or heard differences produced by some aspect of the gear or the test protocol that is not part of what gets recorded (also unlikely in my view). Personally I'd rank (b) as most likely and (a) as second most likely, but hey, what do I know?

Its simply needs replicating by others.

And explaination on the mechanism , is it for example about very good HF hearing ? That would exclude me

I’m also curious about magnitude for the typical home listening scenario if it pans out ?

People have been at it for 40 years or so ? So the natural mechanism when finding a positive in this area is experimental error ? It’s not strange at all ? And curious minds starts to ask questions and tries to replicate ?

 

[pablolie]

What's lacking here is the independent verification of the test being done 'properly'.
It might have been done properly, looks to be done thoroughly but the procedure and test results are not verified independently.
The test was not repeated nor witnessed and not enough evidence is presented.

Trying to convince those that don't even remotely comprehend scientific method is a lost cause.
If we all abided by the law of universal naive belief in wonders that some people here confuse with "scientific", we'd decommissioned power plants immediately a few years ago when lk-99 was invented.