Just how legit is this person's blind test results?

[BDWoody]

Sure... hold on to that dream.

Sorry, I AM kidding. You and I have been round this one before. I am still firmly open to all possibilities.

Questioning his methodology is not the same as questioning his integrity.

 

[Mr. Widget]

One alternate possible other explanation being if it was single blind and somehow without anyone's knowledge or intent his kids switching cables tipped him off as to which was which.

This possibility is very real and the test subject can easily be unaware of it. I participated in a blind comparison where the results were definitely impacted by tone of voice alone.

 

[Mr. Widget]

Questioning his methodology is not the same as questioning his integrity.

Agreed 100% and if one is to assume that the results are not correct, I would prefer to consider a flawed test over a dishonest one.

That said, I will remain in the minority on this thread and give the results the benefit of the doubt, and believe the results are possible. Not definitive, but possible.

 

[JSmith]

his kids switching cables tipped him off as to which was which

It's possible they changed it every time, rather than not changing it for some... would be easier to pick then. Not saying this was the case necessarily, merely an observation of possibility.


JSmith

 

[JRS]

Purr1n from SBAF posted a thread about his own blind test process ( https://www.superbestaudiofriends.o...tic-vs-magni-with-statistical-analysis.13192/ ).

From the look of it he attempted to recreate Amir's testing condition, so at least that's something. But unfortunately I don't remember/know every criteria that needs to be met in order to create an ideal blind test so I don't know fully know how great his testing procedure was, it would be great if someone here is knowledgeable enough to give an opinion.

The only few things holding me back to believe his results is that it's all just texts, no videos or anything as evidence, and his attitude toward Amir/ASR at times. But I know just how much of an effort it is to setup the whole thing, than having to record a video and do everything else in a pure scientific, unbiased approach so I can understand the lack of physical evidence. Still, if we want to doubt his result, is having a video of him perfectly passing the test the only way we can doubt the test's legitimacy?

It's a bit hard to follow the methodology, I'm not sure of the operating conditions. Ideally, there would be many more samples, and potentially others taking part.

Indeed. What would be mist excellent is that he nailed it and all the other guys were random. Course these would have to be honest. Perhaps rewarded for every correct guess?

 

[DVDdoug]

• Magni+ was smoother and less grainy. Heretic had grainier jumpier highs. This was the biggest giveaway.
• Heretic was flatter and boring sounding, but not by much. This was the second giveaway.

This kind of undefined language always makes me skeptical. What does smooth or grainy or jumpy mean? Usually "flatter" refers to frequency response but probably not in this case... He says the highs are jumpy but he doesn't say boosted... And we don't know what's "boring" to him...

If he says he can hear background hiss, or more hiss, in one of the units, or if he was hearing distortion or weak bass or any real-defined characteristics I'd be more inclined to believe the results are real.

 

[JRS]

Assuming the result is legit, that's impressive...

I guess if you can make it audible and zero in on IMD somehow... (how?) then a 9dB difference should be easily detectable. Do we know what the program material was? Maybe two tones at 115dB SPL?

Or perhaps the difference in output impedance had a very strong effect on the headphones he used?

Did he give any subjective notes on what differences were heard?

First thought I had was impedance differences, esp as headphones can be difficult to drive. Know nothing of the products involved, however. But hell yea impedance differences can cause swings of a half decible under reasonable circumstances and the test isn't gamed. Nor have I believed that a decible is the least difference that everyone can resolve. There are golden ears out there, but usually not the ones who believe they are.

 

[Keith_W]

This kind of undefined language always makes me skeptical. What does smooth or grainy or jumpy mean? Usually "flatter" refers to frequency response but probably not in this case... He says the highs are jumpy but he doesn't say boosted... And we don't know what's "boring" to him...

Why should it make you skeptical? Whenever I participate in a DBT, which as a reminder is whether you can reliably discern between A and B using completely subjective criteria, you MUST define what subjective qualities you are looking for. Failure to do this will result in a 50-50 guess. I am most sensitive to frequency response differences, so the first thing I listen for is whether there is a difference in A and B in what I think I can hear. I might use different terms to describe it, e.g. "more emphasize treble at about 2000Hz", but there is nothing wrong with using words like "sibilant" or "glossy" or whatever less precisely defined terms as long as you hear it and you are able to reliably pick it in a DBT. This guy was able to get 10/10 correct so while his descriptions may not mean much to you, he has done enough to prove there is an audible difference between the two headphone amps.

Now, if purr1n wanted to go next level, he should try to correlate what he subjectively heard with measurable phenomena. Perhaps "jumpy" refers to a step response. Perhaps "grainy" refers to distortion or treble shelving. Perhaps the differences were due to impedance interactions with the headphone. I don't know either, but I am not going to begrudge him for not providing the answer, although these are valid questions to ask. Remember that he is operating within the limits of a hobbyist at home with limited resources at his disposal, so we should be mindful of that and be fair to him.

 

[jae]

There's no telling what DMM he used- depending on the meter and what frequency was chosen for matching it could lead to inaccuracies (especially if measured at the commonly measured 1 khz, where a lower frequency may need to be used).

There's also no measurements for whichever new device he's purchased for abx testing and how it responds under various loads, nor the DUTs. We are assuming the DUT performance is consistent with manufacturer published results or 3rd party measurements of an identical product, which I think is a fair assumption, but also adds uncertainty via design revisions or simple malfunctioning if he is unable to test them to within spec.

This is not even getting to the actual switching/testing methodology and 'trustability' and so on. I'm not too familiar with all the rivalries going on but it seems from his words he either has it out for Amir specifically or ASR in general, it is plain that there is significant bias here and thus any reasonable person can conclude he has a large incentive to find favourable results, or perhaps even fabricate them if he was of that character. What's more is that, it is not as if Amir has a unique or eclectic view on measurements or audibility, virtually all of Amir's claims are consistent with established science and conventions in audio engineering and other interdisciplinary fields to do with audio or hearing, so in essence to contest the audibility of said measurements is to contest over a century of rigour and the logical conclusions of an uncountable number of educated experts doing experiments in controlled settings. The burden is on the claimant to reproduce their findings with comparable diligence; vague, unorthodox, and informal tests aren't really enough to get anyone excited or questioning long-established conventions even if there is a flicker of a possibility the claims have merit.

I'm sure photographic or preferably unedited video proof would lend at least some credence to his claims to some people. Being able to hear differences as low as -70 to -80 dB down with certain signals/spectral content or certain types of distortions is perhaps plausible and maybe even demonstrable in very specific instances in controlled environments although we are on the absolute tail end of audibility here for the most part with a young, top percentile listener with perfect hearing. But at -100 dB or more? And typical music spectra? At typical listening levels? And in an environment with a typical residential noise floor? That will take a lot to prove, if these devices and the rest of the chain are in fact performing in spec and don't have anything particularly wrong with them. All of the "obvious" subjective reports/differences made between the trails could easily be found via analysis of the output spectra if they did in fact exist.

 

[Blumlein 88]

There's no telling what DMM he used- depending on the meter and what frequency was chosen for matching it could lead to inaccuracies (especially if measured at the commonly measured 1 khz, where a lower frequency may need to be used).

There's also no measurements for whichever new device he's purchased for abx testing and how it responds under various loads, nor the DUTs. We are assuming the DUT performance is consistent with manufacturer published results or 3rd party measurements of an identical product, which I think is a fair assumption, but also adds uncertainty via design revisions or simple malfunctioning if he is unable to test them to within spec.

This is not even getting to the actual switching/testing methodology and 'trustability' and so on. I'm not to familiar with all the rivalries going on but it seems from his words he either has it out for Amir specifically or ASR in general, it is plain that there is significant bias here and thus any reasonable person can conclude he has a large incentive to find favourable results, or perhaps even fabricate them if he was of that character. What's more is that, it is not as if Amir has a unique or eclectic view on measurements or audibility, virtually all of Amir's claims are consistent with established science and conventions in audio engineering and other interdisciplinary fields to do with audio, so in essence to contest the audibility of said measurements is to contest over a century of rigour and the logical conclusions of an uncountable number of experts doing experiments in controlled settings. The burden is on the claimant to reproduce their findings with comparable diligence; vague, unorthodox, and informal tests aren't really enough to get anyone excited or questioning long-established conventions even if there is a flicker of a possibility the claims have merit.

I'm sure photographic or preferably unedited video proof would lend at least some credence to his claims to some people. Being able to hear differences as low as -70 to -80 dB down with certain signals/spectral content or certain types of distortions is perhaps plausible and maybe even demonstrable in very specific instances in controlled environments although we are on the absolute tail end of audibility here for the most part with a top percentile listener. But at -100 dB or more? And typical music spectra? At typical listening levels? And in the environment with typical residential noise floor? That will take a lot to prove if these devices and the rest of the chain are in fact performing in spec and don't have anything particularly wrong with them. All of the "obvious" subjective reports/differences made between the trails could easily be found via analysis if they did in fact exist.

He used a Fluke 73 III and 1 khz. Should be fine. Even if off at 1 khz it will read the same for both sources.

I don't think he has used the Van Alstine ABX box yet.

ABX Switch Comparator

Due to high demand we have produced another run of our ABX Switch Comparator. This unit allows seamless switching of up to two preamplifiers, two power amplifiers, and three sets of speakers either with or without subwoofers. The AVA ABX Switch can be used to evaluate cables and component parts...

avahifi.com

 

[kemmler3D]

I guess what we can say is he passed the test he ran for himself, assuming it's not a scam. I don't truly suspect it is.

However, we're missing a few details about the test, ultimately we aren't sure what was actually heard or exactly what test was passed until we know a little more. Still, it's interesting!

 

[JRS]

We should extend an invitation to do it in real time (hell with tape) and livestream it hear on ASR. After we provided the conditions for unassailable results. Hell I'd throw 50 bucks in a communal pot he could collect if he 10/10'ed it. If he loses, he publically retracts his claims based on earlier tests.

 

[Deleted member 2110]

Other alternatives than lying, and yet getting an erroneous result. Lying or not lying are not the only options. One alternate possible other explanation being if it was single blind and somehow without anyone's knowledge or intent his kids switching cables tipped him off as to which was which.

From the link: “Double blind test: I got my kids to change up the wires into the amps headphone output randomly while I wasn't in the room and then I asked them to leave the room in another direction of which I entered the room.”

 

[jae]

We should extend an invitation to do it in real time (hell with tape) and livestream it hear on ASR. After we provided the conditions for unassailable results. Hell I'd throw 50 bucks in a communal pot he could collect if he 10/10'ed it. If he loses, he publically retracts his claims based on earlier tests.

I think there has more or less been an open invitation for such a blind test to anyone (or at least a prominent figure/personality) who was keen on trying an objective blind test to prove their claims to do so. The situation with goldensound comes to mind (who also has an AP and contributes subjective opinions to his objective measurements with no rhyme or reason), which never came into fruition even though a large sum was proposed if he could pass it.

The fact of the matter is that some adequate organised/moderated blind test is unlikely to happen. People are fine with attempting or appearing to attempt abx tests in the comfort of their own home behind the internet (maybe even giving an genuine/honest effort to do so) but when their claims are scrutinised they either aren't able to reproduce adequate conditions and give up, or lose interest. Even if he were to in fact pass an abx, there is a much higher probability that there would be something measurably anomalous with the DUTs in the conditions they were operated in that was able to be found retroactively, rather than him being able to detect differences that low down. Clicking on the guy's profile he has nearly 20,000+ posts and large number of reviews and opinions about gear for the better part of a decade, he appears as an integral part of the website and just by that virtue alone he is very unlikely to walk back anything since it would cause a massive schism in their userbase and effectively kill that website, since it would mean all previous opinions about audibility and sound quality differences would be fabrications.

According to his profile he is also 54. The chance of him having normal audiometric thresholds or not having some kind of hidden hearing loss as a male at that age, let alone one that considers themselves an audiophile that listens to music all day is probably already on the side of "unlikely". If so, that would further preclude him from being able to hear genuine differences in music at -80 dB let alone -100 dB or the proposed differences beyond that. Proof of a normal audiogram conducted by a professional audiologist and perhaps even a speech in noise test should be the absolute minimum criteria for someone to provide before making claims of differences in audibility at these extreme levels, before even entertaining the process of blind testing.

 

[MaxwellsEq]

It's possible they changed it every time, rather than not changing it for some... would be easier to pick then. Not saying this was the case necessarily, merely an observation of possibility.


JSmith

This is a good point. Ideally the children flipped a coin or some other randomising approach to decide whether they would change anything or leave alone. There may easily have been a run of 10 tests only using one device!

 

[Blumlein 88]

From the link: “Double blind test: I got my kids to change up the wires into the amps headphone output randomly while I wasn't in the room and then I asked them to leave the room in another direction of which I entered the room.”

Yes, but did they see each other, or speak to each other. You'd possibly be very surprised how little it takes to unintentionally cue someone. It appears he did try to make it double blind. Like his use of listening material and headphone involved a few more details would have been encouraging.

 

[Sokel]

At least they use tests now,that's something to encourage and try to make it easy,not even harder than it already is.
(a video would be nice thought)

 

[Deleted member 48726]

This kind of undefined language always makes me skeptical. What does smooth or grainy or jumpy mean? Usually "flatter" refers to frequency response but probably not in this case... He says the highs are jumpy but he doesn't say boosted... And we don't know what's "boring" to him...

If he says he can hear background hiss, or more hiss, in one of the units, or if he was hearing distortion or weak bass or any real-defined characteristics I'd be more inclined to believe the results are real.

We don't have to define his descriptions IR to the exact frequencies in question. That's subjective how we would describe the differences we hear. We only need to know if he heard differences, which it seemingly looks like he did.

 

[solderdude]

Now... if only he could repeat the test in front of an Amir... how great would that be ?

Marvin makes it look as it would be like taking candy from a baby.
Can Amir really kiss his $ 1000.- goodbuy (going to a charity of Marvin's choice ?)

 

[Astoneroad]

Now... if only he could repeat the test in front of an Amir... how great would that be ?

If only he could repeat it in front of the world, simply by making a vid of the entire procedure. If true, I'd kick in toward the $1000 donation. I'm guessing that there are other ASR members that would do the same to have their paradigm shifted by real world scientific method and results.