• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Just how legit is this person's blind test results?

nyxnyxnyx

Addicted to Fun and Learning
Joined
May 22, 2019
Messages
506
Likes
475
Purr1n from SBAF posted a thread about his own blind test process ( https://www.superbestaudiofriends.o...tic-vs-magni-with-statistical-analysis.13192/ ).

From the look of it he attempted to recreate Amir's testing condition, so at least that's something. But unfortunately I don't remember/know every criteria that needs to be met in order to create an ideal blind test so I don't know fully know how great his testing procedure was, it would be great if someone here is knowledgeable enough to give an opinion.

The only few things holding me back to believe his results is that it's all just texts, no videos or anything as evidence, and his attitude toward Amir/ASR at times. But I know just how much of an effort it is to setup the whole thing, than having to record a video and do everything else in a pure scientific, unbiased approach so I can understand the lack of physical evidence. Still, if we want to doubt his result, is having a video of him perfectly passing the test the only way we can doubt the test's legitimacy?
 
OP
nyxnyxnyx

nyxnyxnyx

Addicted to Fun and Learning
Joined
May 22, 2019
Messages
506
Likes
475
FWIW he seems willing enough to continue doing tests, he even got himself this
1678391493936.png
 

Astoneroad

Addicted to Fun and Learning
Forum Donor
Joined
Feb 16, 2022
Messages
999
Likes
2,051
Location
a Cave in the desert
Now all he needs is a tripod for his phone to shoot the vid and repeat the test.
 

MaxwellsEq

Major Contributor
Joined
Aug 18, 2020
Messages
1,725
Likes
2,579
It's a bit hard to follow the methodology, I'm not sure of the operating conditions. Ideally, there would be many more samples, and potentially others taking part.
 
  • Like
Reactions: JRS

Keith_W

Major Contributor
Joined
Jun 26, 2016
Messages
2,646
Likes
6,046
Location
Melbourne, Australia
From what I understand of the methodology:

- This was a DBT between a Schiit Magni 3+ and Schiit Magni Heresy amp. According to Schiit's webpage both amps look the same apart from paint finish, sell for the same price. Looking at the specs, the obvious differences between the two include IMD (<-103dB vs. <-112dB at 1V, 300 Ohms respectively), SNR (>110dB vs. >115dB at 2V), Output impedance (<0.2 Ohms vs. <0.33 Ohms), input impedance (25k Ohms vs. 50k Ohms), and topology (one has fully discrete components, the other has op-amps with no overall feedback).
- He level matched all the headphone amps at the output using a DMM (Digital Multimeter) to 0.01V RMS.
- His kids did the swapping for him while he was in another room, and he used the same headphones.
- There were no measurements of any kind apart from level matching with said DMM, perhaps he does not own an appropriate setup. However, the manufacturer does post results of both amps using an Audio Precision APX555 here for the Magni 3+ and for the Magni Heretic.
- I might have missed it somewhere, but he does not mention which headphone he was using. (EDIT) He used ZMF Verite headphones.

Result: 100% correct out of 10 attempts. The likelihood of guessing 10 correct answers with a choice of 2 is 1/2^10, or 1/1024. If this is true, then in my opinion this is a valid result.

Feasibility: is it possible for two amps by the same manufacturer with different circuit topologies, different input and output impedances, and different measurements to sound different, such that a trained ear can reliably pick them with 100% accuracy in a DBT? I would think so.

Opinion: IMO his methodology was sound. I would like to have seen more test subjects and some measurements (to verify the manufacturer's own posted measurements), along with presence of unbiased observers, but I can understand the limitations of hobbyists conducting experiments at home so I do not necessarily mark him down for this one. I have ignored all the anti-ASR snark just as I ignore all the snark on ASR. For me this is a valid DBT result and I warmly congratulate purr1n for making the attempt.
 
Last edited:
OP
nyxnyxnyx

nyxnyxnyx

Addicted to Fun and Learning
Joined
May 22, 2019
Messages
506
Likes
475
From what I understand of the methodology:

- This was a DBT between a Schiit Magni 3+ and Schiit Magni Heresy amp. According to Schiit's webpage both amps look the same apart from paint finish, sell for the same price. Looking at the specs, the obvious differences between the two include IMD (<-103dB vs. <-112dB at 1V, 300 Ohms respectively), SNR (>110dB vs. >115dB at 2V), Output impedance (<0.2 Ohms vs. <0.33 Ohms), input impedance (25k Ohms vs. 50k Ohms), and topology (one has fully discrete components, the other has op-amps with no overall feedback).
- He level matched all the headphone amps at the output using a DMM (Digital Multimeter) to 0.01V RMS.
- His kids did the swapping for him while he was in another room, and he used the same headphones.
- There were no measurements of any kind apart from level matching with said DMM, perhaps he does not own an appropriate setup. However, the manufacturer does post results of both amps using an Audio Precision APX555 here for the Magni 3+ and for the Magni Heretic.
- I might have missed it somewhere, but he does not mention which headphone he was using.

Result: 100% correct out of 10 attempts. The likelihood of guessing 10 correct answers with a choice of 2 is 1/2^10, or 1/1024. If this is true, then in my opinion this is a valid result.

Feasibility: is it possible for two amps by the same manufacturer with different circuit topologies, different input and output impedances, and different measurements to sound different, such that a trained ear can reliably pick them with 100% accuracy in a DBT? I would think so.

Opinion: IMO his methodology was sound. I would like to have seen more test subjects and some measurements (to verify the manufacturer's own posted measurements), along with presence of unbiased observers, but I can understand the limitations of hobbyists conducting experiments at home so I do not necessarily mark him down for this one. I have ignored all the anti-ASR snark just as I ignore all the snark on ASR. For me this is a valid DBT result and I warmly congratulate purr1n for making the attempt.
I like how he disses ASR/Amir now and then but he's willing to walk the walk, doing the test and posting his method publicly for discussions. The only thing still missing is concrete proof.

I've never fully believed in either "all amps sound the same" or "all amps sound different" school of thoughts so I think it's plausible someone could distinguish the differences between two amplifiers in certain situations/setups semi-reliably.

Ideal scenario: both him and other reputable figures like Amir or Jude (head-fi) can setup and record their meeting where they'll do the tests to put a new light to the debate.
 

kemmler3D

Major Contributor
Forum Donor
Joined
Aug 25, 2022
Messages
3,346
Likes
6,832
Location
San Francisco
Assuming the result is legit, that's impressive...

I guess if you can make it audible and zero in on IMD somehow... (how?) then a 9dB difference should be easily detectable. Do we know what the program material was? Maybe two tones at 115dB SPL?

Or perhaps the difference in output impedance had a very strong effect on the headphones he used?

Did he give any subjective notes on what differences were heard?
 
OP
nyxnyxnyx

nyxnyxnyx

Addicted to Fun and Learning
Joined
May 22, 2019
Messages
506
Likes
475
Assuming the result is legit, that's impressive...

I guess if you can make it audible and zero in on IMD somehow... (how?) then a 9dB difference should be easily detectable. Do we know what the program material was? Maybe two tones at 115dB SPL?

Or perhaps the difference in output impedance had a very strong effect on the headphones he used?

Did he give any subjective notes on what differences were heard?
Yeah he always write more subjective impressions/differences than the measurements stuff. I haven't tried those two amplifiers so I can't match his experience.

Throughout the years I've had some "theoretically bad, realistically not-so" experiences. For example, low current OTL amplifiers pairing with planar headphones, on paper it sounds underwhelming, and I am aware why it's not a good match, but when listening it was "alright".

So maybe this could be one of those cases? And Purr1in did mention he "practiced" listening to the amplifier to the point he thinks he's used to one of them before beginning the test, this could be one of the things people who did DBT don't usually do?
 

AdamG

Helping you stretch your audiophile budget…
Moderator
Forum Donor
Joined
Jan 3, 2021
Messages
4,734
Likes
15,617
Location
Reality
Trust but Verify. That’s what the video will do. Otherwise there is lots of wiggle room to produce results that best serve parties of interest.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,717
Likes
37,475
It seems highly improbable for three reasons. First is how well the amps being used measured. Secondly, if he left the room each time and put the phones back on it likely sounded a bit different from inconsistent phone positioning. Thirdly leaving the room, and letting cables be switched would take quite a bit of time and it is known small differences become much harder to detect after a gap of more than a few seconds.

Otherwise it appears legit, but knowing the headphone and material listened to for the test would be nice. Also if he could repeat it about twice with similar results it would be a lot more convincing.

One final question is how it was handled that his kids switching cables interacted with him during the test. In other words was it single or double blind.
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,174
Likes
1,774
Location
SF Bay Area
Personalities aside, I hope to see more of these comparisons... whether they align with the "solved" school of thought or refute it. It is refreshing to see.

And, I did get a kick out of the AmirNAD term. ;)
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,174
Likes
1,774
Location
SF Bay Area
He stated that it was really easy to hear the difference, despite a protocol that likely reduces the sensitivity of the test. Make of that what you will.
True, however he said it took some effort to learn the "sonic signature" (my words) of each amp, but once he had it became very easy to pick them out. If the amps do in fact sound different from one another as so many of us have "thought" we have heard, then bingo he was right.

Alternatively he is a liar.

With no evidence to the contrary, I'll assume he is not lying.
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,490
Likes
25,259
Location
Alfred, NY
True, however he said it took some effort to learn the "sonic signature" (my words) of each amp, but once he had it became very easy to pick them out. If the amps do in fact sound different from one another as so many of us have "thought" we have heard, then bingo he was right.

Alternatively he is a liar.

With no evidence to the contrary, I'll assume he is not lying.
There could be other alternatives. We have no way of knowing.
 

Mr. Widget

Major Contributor
Forum Donor
Joined
Oct 11, 2022
Messages
1,174
Likes
1,774
Location
SF Bay Area
There could be other alternatives. We have no way of knowing.
Sure... hold on to that dream.

Sorry, I AM kidding. You and I have been round this one before. I am still firmly open to all possibilities.
 
OP
nyxnyxnyx

nyxnyxnyx

Addicted to Fun and Learning
Joined
May 22, 2019
Messages
506
Likes
475
Related to the topic, can anyone verify the effectiveness of that ABX device? I think he's going to use it in his upcoming tests. I think it'd be really great if the sources can be switched immediately without any compromises.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,717
Likes
37,475
Sure... hold on to that dream.

Sorry, I AM kidding. You and I have been round this one before. I am still firmly open to all possibilities.
Other alternatives than lying, and yet getting an erroneous result. Lying or not lying are not the only options. One alternate possible other explanation being if it was single blind and somehow without anyone's knowledge or intent his kids switching cables tipped him off as to which was which.
 
Last edited:
Top Bottom