Just how legit is this person's blind test results?

[nyxnyxnyx]

Purr1n from SBAF posted a thread about his own blind test process ( https://www.superbestaudiofriends.o...tic-vs-magni-with-statistical-analysis.13192/ ).

From the look of it he attempted to recreate Amir's testing condition, so at least that's something. But unfortunately I don't remember/know every criteria that needs to be met in order to create an ideal blind test so I don't know fully know how great his testing procedure was, it would be great if someone here is knowledgeable enough to give an opinion.

The only few things holding me back to believe his results is that it's all just texts, no videos or anything as evidence, and his attitude toward Amir/ASR at times. But I know just how much of an effort it is to setup the whole thing, than having to record a video and do everything else in a pure scientific, unbiased approach so I can understand the lack of physical evidence. Still, if we want to doubt his result, is having a video of him perfectly passing the test the only way we can doubt the test's legitimacy?

 

[Astoneroad]

Yes.

 

[nyxnyxnyx]

FWIW he seems willing enough to continue doing tests, he even got himself this

 

[Astoneroad]

Now all he needs is a tripod for his phone to shoot the vid and repeat the test.

 

[nyxnyxnyx]

Yes.

So all the conditions he listed is sufficient for unbiased, truthful testing?

Edit: well I hope so

 

[MaxwellsEq]

It's a bit hard to follow the methodology, I'm not sure of the operating conditions. Ideally, there would be many more samples, and potentially others taking part.

 

[Keith_W]

From what I understand of the methodology:

- This was a DBT between a Schiit Magni 3+ and Schiit Magni Heresy amp. According to Schiit's webpage both amps look the same apart from paint finish, sell for the same price. Looking at the specs, the obvious differences between the two include IMD (<-103dB vs. <-112dB at 1V, 300 Ohms respectively), SNR (>110dB vs. >115dB at 2V), Output impedance (<0.2 Ohms vs. <0.33 Ohms), input impedance (25k Ohms vs. 50k Ohms), and topology (one has fully discrete components, the other has op-amps with no overall feedback).
- He level matched all the headphone amps at the output using a DMM (Digital Multimeter) to 0.01V RMS.
- His kids did the swapping for him while he was in another room, and he used the same headphones.
- There were no measurements of any kind apart from level matching with said DMM, perhaps he does not own an appropriate setup. However, the manufacturer does post results of both amps using an Audio Precision APX555 here for the Magni 3+ and for the Magni Heretic.
- I might have missed it somewhere, but he does not mention which headphone he was using. (EDIT) He used ZMF Verite headphones.

Result: 100% correct out of 10 attempts. The likelihood of guessing 10 correct answers with a choice of 2 is 1/2^10, or 1/1024. If this is true, then in my opinion this is a valid result.

Feasibility: is it possible for two amps by the same manufacturer with different circuit topologies, different input and output impedances, and different measurements to sound different, such that a trained ear can reliably pick them with 100% accuracy in a DBT? I would think so.

Opinion: IMO his methodology was sound. I would like to have seen more test subjects and some measurements (to verify the manufacturer's own posted measurements), along with presence of unbiased observers, but I can understand the limitations of hobbyists conducting experiments at home so I do not necessarily mark him down for this one. I have ignored all the anti-ASR snark just as I ignore all the snark on ASR. For me this is a valid DBT result and I warmly congratulate purr1n for making the attempt.

 

[nyxnyxnyx]

From what I understand of the methodology:

- This was a DBT between a Schiit Magni 3+ and Schiit Magni Heresy amp. According to Schiit's webpage both amps look the same apart from paint finish, sell for the same price. Looking at the specs, the obvious differences between the two include IMD (<-103dB vs. <-112dB at 1V, 300 Ohms respectively), SNR (>110dB vs. >115dB at 2V), Output impedance (<0.2 Ohms vs. <0.33 Ohms), input impedance (25k Ohms vs. 50k Ohms), and topology (one has fully discrete components, the other has op-amps with no overall feedback).
- He level matched all the headphone amps at the output using a DMM (Digital Multimeter) to 0.01V RMS.
- His kids did the swapping for him while he was in another room, and he used the same headphones.
- There were no measurements of any kind apart from level matching with said DMM, perhaps he does not own an appropriate setup. However, the manufacturer does post results of both amps using an Audio Precision APX555 here for the Magni 3+ and for the Magni Heretic.
- I might have missed it somewhere, but he does not mention which headphone he was using.

Result: 100% correct out of 10 attempts. The likelihood of guessing 10 correct answers with a choice of 2 is 1/2^10, or 1/1024. If this is true, then in my opinion this is a valid result.

Feasibility: is it possible for two amps by the same manufacturer with different circuit topologies, different input and output impedances, and different measurements to sound different, such that a trained ear can reliably pick them with 100% accuracy in a DBT? I would think so.

Opinion: IMO his methodology was sound. I would like to have seen more test subjects and some measurements (to verify the manufacturer's own posted measurements), along with presence of unbiased observers, but I can understand the limitations of hobbyists conducting experiments at home so I do not necessarily mark him down for this one. I have ignored all the anti-ASR snark just as I ignore all the snark on ASR. For me this is a valid DBT result and I warmly congratulate purr1n for making the attempt.

I like how he disses ASR/Amir now and then but he's willing to walk the walk, doing the test and posting his method publicly for discussions. The only thing still missing is concrete proof.

I've never fully believed in either "all amps sound the same" or "all amps sound different" school of thoughts so I think it's plausible someone could distinguish the differences between two amplifiers in certain situations/setups semi-reliably.

Ideal scenario: both him and other reputable figures like Amir or Jude (head-fi) can setup and record their meeting where they'll do the tests to put a new light to the debate.

 

[kemmler3D]

Assuming the result is legit, that's impressive...

I guess if you can make it audible and zero in on IMD somehow... (how?) then a 9dB difference should be easily detectable. Do we know what the program material was? Maybe two tones at 115dB SPL?

Or perhaps the difference in output impedance had a very strong effect on the headphones he used?

Did he give any subjective notes on what differences were heard?

 

[nyxnyxnyx]

Assuming the result is legit, that's impressive...

I guess if you can make it audible and zero in on IMD somehow... (how?) then a 9dB difference should be easily detectable. Do we know what the program material was? Maybe two tones at 115dB SPL?

Or perhaps the difference in output impedance had a very strong effect on the headphones he used?

Did he give any subjective notes on what differences were heard?

Yeah he always write more subjective impressions/differences than the measurements stuff. I haven't tried those two amplifiers so I can't match his experience.

Throughout the years I've had some "theoretically bad, realistically not-so" experiences. For example, low current OTL amplifiers pairing with planar headphones, on paper it sounds underwhelming, and I am aware why it's not a good match, but when listening it was "alright".

So maybe this could be one of those cases? And Purr1in did mention he "practiced" listening to the amplifier to the point he thinks he's used to one of them before beginning the test, this could be one of the things people who did DBT don't usually do?

 

[AdamG]

Trust but Verify. That’s what the video will do. Otherwise there is lots of wiggle room to produce results that best serve parties of interest.

 

[Blumlein 88]

It seems highly improbable for three reasons. First is how well the amps being used measured. Secondly, if he left the room each time and put the phones back on it likely sounded a bit different from inconsistent phone positioning. Thirdly leaving the room, and letting cables be switched would take quite a bit of time and it is known small differences become much harder to detect after a gap of more than a few seconds.

Otherwise it appears legit, but knowing the headphone and material listened to for the test would be nice. Also if he could repeat it about twice with similar results it would be a lot more convincing.

One final question is how it was handled that his kids switching cables interacted with him during the test. In other words was it single or double blind.

 

[SIY]

He stated that it was really easy to hear the difference, despite a protocol that likely reduces the sensitivity of the test. Make of that what you will.

 

[Mr. Widget]

Personalities aside, I hope to see more of these comparisons... whether they align with the "solved" school of thought or refute it. It is refreshing to see.

And, I did get a kick out of the AmirNAD term.

 

[Mr. Widget]

He stated that it was really easy to hear the difference, despite a protocol that likely reduces the sensitivity of the test. Make of that what you will.

True, however he said it took some effort to learn the "sonic signature" (my words) of each amp, but once he had it became very easy to pick them out. If the amps do in fact sound different from one another as so many of us have "thought" we have heard, then bingo he was right.

Alternatively he is a liar.

With no evidence to the contrary, I'll assume he is not lying.

 

[SIY]

True, however he said it took some effort to learn the "sonic signature" (my words) of each amp, but once he had it became very easy to pick them out. If the amps do in fact sound different from one another as so many of us have "thought" we have heard, then bingo he was right.

Alternatively he is a liar.

With no evidence to the contrary, I'll assume he is not lying.

There could be other alternatives. We have no way of knowing.

 

[Mr. Widget]

There could be other alternatives. We have no way of knowing.

Sure... hold on to that dream.

Sorry, I AM kidding. You and I have been round this one before. I am still firmly open to all possibilities.

 

[nyxnyxnyx]

Related to the topic, can anyone verify the effectiveness of that ABX device? I think he's going to use it in his upcoming tests. I think it'd be really great if the sources can be switched immediately without any compromises.

 

[Blumlein 88]

Sure... hold on to that dream.

Sorry, I AM kidding. You and I have been round this one before. I am still firmly open to all possibilities.

Other alternatives than lying, and yet getting an erroneous result. Lying or not lying are not the only options. One alternate possible other explanation being if it was single blind and somehow without anyone's knowledge or intent his kids switching cables tipped him off as to which was which.

 

[oleg87]

Insofar as you can trust the self-reporting of someone with a clear financial interest in his golden ears...