Just how legit is this person's blind test results?

[solderdude]

Note: we are talking about headphone amps in this test (with really easy load) NOT about tube amps or speaker amps that with actual (and not easy) loads measure so substantially different that it exceeds at least one of the 'lenient' properties often by miles.

Here's 2 quotes from Marvin in the same article:

Amir is hostile to the fact that people can actually have the ability to hear differences with amps that have well over 102+ SINAD

It's time to put an end to the AmirNAD dogma nonsense and show that SINAD doesn't make much difference past ~55 to 75db.

So on one hand Marvin claims Amir is hostile over the fact that people can have an ability to hear differences that have well over 102+ SINAD
But Marvin himself states that SINAD doesn't make much difference past ~55 to 75db.

How can this be ?
One makes a claim that 75dB SINAD is enough but also claims that 102+ dB can still show differences.

Well... technically Marvin is correct IF we assume SINAD is the only value that shows 'fidelity quality'.
As we all know that isn't the case nor the premise. It is just something people like to slap the dog with.

I agree with the sentiment that distortion below -70dB is not detectable with music.
Note that I did not say SINAD. That is just a number referenced to 2V or 4V (balanced) with a 1kHz stimulus and in a specified (resistive) load.

How can we hear differences between 55 SINAD and 102+ SINAD amps is the question.

That can be because of:
Level differences, frequency response differences, load differences, channel imbalance differences, polarity differences, distortion (harmonic spread) differences, noise levels, output resistance differences (in a lot but not all cases), test setup errors.

The data above from @Galliardist shows that frequency response and load differences and distortion levels and harmonic spread as well as noise levels and output resistance as well as (claimed) level differences are highly unlikely to the culprit.

So what can it be ?
Remains: Channel imbalance. I see no evidence it was checked for. We only see a multimeter showing numbers but this can only be from 1 channel. The other channel could well differ a bit. Audible ?
Polarity differences ? As far as I know both amp designs are non inverting so unlikely.
Test setup errors: Impossible to tell, we have to take his word for it.
The 'unknown property' or a 'not measured property ?: This is always used as an excuse and even today it has not been shown to exist in lab conditions.
Bias/test objective:

It's time to put an end to the AmirNAD dogma nonsense and show that SINAD doesn't make much difference past ~55 to 75db.

O.K. so he proved that when all is equal (which it seems to be if the test was performed correctly) then he has proven Amir right and the SINAD chart is telling as both amps are way past 75 SINAD and all other aspects are the same.

Marvin proved himself wrong.

What Marvin failed to do is investigate why this happened and to investigate further the test has to be repeated (which it probably will) but with an ABX box.
Then, when channel imbalance is checked for and enough attempts are made and there is still an 'easy to hear' difference (can that be recorded ?) yet no technical explanation is found other than a very small difference in distortion (which would be preferable) then it makes sense to investigate further.

Will this happen ? Of course not. People have been looking for 'correlation' between (basic) measurements and perception for decades.
Some claim to have evidence but is kept 'trade secret' or is not verified by others that look for verification.
Some claim there is no evidence but may not have researched all angles or did not have access to 'the better hearing' people.

Scientists seem to be 'deaf' and don't want to hear (obvious) differences because plots tell it can't be so.
Audiophiles seem to hear things and are not interested in proving this to those horrible 'science' guys despite it being very easy to do.

 

[delta76]

Note: we are talking about headphone amps in this test (with really easy load) NOT about tube amps or speaker amps that with actual (and not easy) loads measure so substantially different that it exceeds at least one of the 'lenient' properties often by miles.

Here's 2 quotes from Marvin in the same article:





So on one hand Marvin claims Amir is hostile over the fact that people can have an ability to hear differences that have well over 102+ SINAD
But Marvin himself states that SINAD doesn't make much difference past ~55 to 75db.

How can this be ?
One makes a claim that 75dB SINAD is enough but also claims that 102+ dB can still show differences.

Well... technically Marvin is correct IF we assume SINAD is the only value that shows 'fidelity quality'.
As we all know that isn't the case nor the premise. It is just something people like to slap the dog with.

I agree with the sentiment that distortion below -70dB is not detectable with music.
Note that I did not say SINAD. That is just a number referenced to 2V or 4V (balanced) with a 1kHz stimulus and in a specified (resistive) load.

How can we hear differences between 55 SINAD and 102+ SINAD amps is the question.

That can be because of:
Level differences, frequency response differences, load differences, channel imbalance differences, polarity differences, distortion (harmonic spread) differences, noise levels, output resistance differences (in a lot but not all cases), test setup errors.

The data above from @Galliardist shows that frequency response and load differences and distortion levels and harmonic spread as well as noise levels and output resistance as well as (claimed) level differences are highly unlikely to the culprit.

So what can it be ?
Remains: Channel imbalance. I see no evidence it was checked for. We only see a multimeter showing numbers but this can only be from 1 channel. The other channel could well differ a bit. Audible ?
Polarity differences ? As far as I know both amp designs are non inverting so unlikely.
Test setup errors: Impossible to tell, we have to take his word for it.
The 'unknown property' or a 'not measured property ?: This is always used as an excuse and even today it has not been shown to exist in lab conditions.
Bias/test objective:

O.K. so he proved that when all is equal (which it seems to be if the test was performed correctly) then he has proven Amir right and the SINAD chart is telling as both amps are way past 75 SINAD and all other aspects are the same.

Marvin proved himself wrong.

What Marvin failed to do is investigate why this happened and to investigate further the test has to be repeated (which it probably will) but with an ABX box.
Then, when channel imbalance is checked for and enough attempts are made and there is still an 'easy to hear' difference (can that be recorded ?) yet no technical explanation is found other than a very small difference in distortion (which would be preferable) then it makes sense to investigate further.

Will this happen ? Of course not. People have been looking for 'correlation' between (basic) measurements and perception for decades.
Some claim to have evidence but is kept 'trade secret' or is not verified by others that look for verification.
Some claim there is no evidence but may not have researched all angles or did not have access to 'the better hearing' people.

Scientists seem to be 'deaf' and don't want to hear (obvious) differences because plots tell it can't be so.
Audiophiles seem to hear things and are not interested in proving this to those horrible 'science' guys despite it being very easy to do.

I only see him as hostile in this case.

Amir did not "invent" the 85dB threshold himself. he based it on well established researches. and each component has different threshold SINAD for audibility (and Amir did not even invent the term SINAD. he only popularized it, yet subjectivists assumed it's his "invention")

 

[nyxnyxnyx]

Since Marv got his hands on that ABX device, I really think that we will see a part 3 of his test, maybe he'll even include some of the observations/accusations from our comments here and answer them. I hope he'll record a video or something this time.

I believe that he never had to be 'half science half audiophile' to improve his standing in the community, or to cater to some kind of crowd. Like folks in SBAF they like him for being a grand contributor, and for his personality (I guess?), by doing this, I don't see how it truly benefits him.

 

[solderdude]

When it would turn out he cannot discern any differences anymore when all controls are properly in place and checked he would have a 'problem' right there.
So one can rest assured. His part 3 will confirm part 1 and 2 which is Amir is a turd and he can (easily) discern between the 2 amps.
I don't see it could go any different.
It is ridiculously easy to ensure a test falls your way.

 

[nyxnyxnyx]

When it would turn out he cannot discern any differences anymore when all controls are properly in place and checked he would have a 'problem' right there.
So one can rest assured. His part 3 will confirm part 1 and 2 which is Amir is a turd and he can (easily) discern between the 2 amps.
I don't see it could go any different.
It is ridiculously easy to ensure a test falls your way.

I agree with you, but this argument is for when we're assuming he's 'sabotaging his own test on purpose' right?
what would be the argument for when he's 'taking test seriously with no mischief but the testing condition is flawed (due to mistakes or a lack of knowledge)'?

 

[solderdude]

It's the premise that the amps sound different and him setting out to prove it with a test that is touted by ASR to be the only valid one (level matched blind).

I would not call it 'sabotaging' his own test but rather 'rigging' the test on purpose or there was something else going on making amps that, given measured performance, are so well below any possibly audible levels, in such an easy load under far from strained conditions, that the actual signal on the output of both amps is incredibly similar would still differ so much that it is audible.

Just because one states the test was carefully setup and a voltmeter was used doesn't mean the test is well controlled.

If it were that easy he could show it in front of a witness as well. One that also listened to both amps. As the test is now and the reporting around it, it proves nothing and is just a verification to SBAF readers that they are not mad and these amps are 'easy' to tell apart.... even in a blind test.

It took 2 years between his first attempt and second attempt so around 2025 part trois ?
How often will Marvin be using his ABX tester anyway... he hates AB testing.

Don't get me wrong I would love to see him pull this off witnessed. Then the same amps fully tested and 2 recordings made through them and dragged through Paul's software.
When there is an audible difference there will be a (susbstantial) measurable difference as well.
When it is audible there must be a difference in nulling as well.
When it is audible the test is repeatable.

In the end... the answer to this question:
Just how legit is this person's blind test results?
should be followed by another question:
Do you trust Marvin (Purr1n) ?

When you trust Marvin to do the test properly you will likely find the test 'legit'.
When you do not trust Marvin to do the test properly you will likely find the test 'suspect'.

The question would be similar to asking the following question on SBAF:
Just how legit are ASR test results?

When they trust Amir to do the test properly they will likely find the test 'legit'.
When they do not trust Amir to do the test properly they will likely find the test 'pure nonsensical and absurd'.

 

[JayGilb]

Since Marv got his hands on that ABX device, I really think that we will see a part 3 of his test, maybe he'll even include some of the observations/accusations from our comments here and answer them. I hope he'll record a video or something this time.

I believe that he never had to be 'half science half audiophile' to improve his standing in the community, or to cater to some kind of crowd. Like folks in SBAF they like him for being a grand contributor, and for his personality (I guess?), by doing this, I don't see how it truly benefits him.

It's not about audio with Marv, it's all about him, nothing else. It just happens to be a site about audio.
If it was Super Best Food Friends, he would be talking about his ability to detect certain flavors over others.

 

[Galliardist]

I only see him as hostile in this case.

Amir did not "invent" the 85dB threshold himself. he based it on well established researches. and each component has different threshold SINAD for audibility (and Amir did not even invent the term SINAD. he only popularized it, yet subjectivists assumed it's his "invention")

It's nothing to do with Amir anyway.
The article here on audibility was written by forum member @flipflop and the lenient thresholds based on the work of NwAvGuy, a well known blogger from a decade or so ago who is no longer active under that name.

There is an assumption made by some that Amir is NwAvGuy but that is not proven and I understand the comments on the subject from Amir to be a denial.

As for well established researches, NwAvGuy gave no sources or references, so we can no more assume well established researches for 85 than we can assume that Purr1n's test is valid.

Yes, Amir did promote the thread, but does that validate the results or make them some kind of ASR dogma? Not really. In a post in the thread Amir warns us that the lenient threshold is only a guideline. And the original article also acknowledges the lack of references.

The SBAF crew know and refer to that gap in the ASR argument. If we are evidence based, we need a better argument than "what a blogger wrote", no matter NwAvGuy is seen as a hero for what he did.

A lot of this thread is about us, not them. And we should have all the relevant measurements and references for them in a thread like this by the end of page three, not a subset of them on page 18, and be starting on literature (not stopping because it's all behind a paywall).

I'm as guilty as anyone of course. I'd like to thank @symphara for posting the thresholds and kicking me up the backside to dig in, but I'm on his ignore list...

 

[SIY]

The question would be similar to asking the following question on SBAF:
Just how legit are ASR test results?

When you trust Amir to do the test properly you will likely find the test 'legit'.
When you do not trust Amir to do the test properly you will likely find the test 'pure nonsensical and absurd'.

The difference being that Amir outlines the test procedure in enough detail to allow replications AND will provide AP project files with the raw data. This is not the case here.

Re: replications, I've now tested three things that he posted here (the actual samples he tested), then sent me, and our measurements line up nearly perfectly.

 

[JayGilb]

The difference being that Amir outlines the test procedure in enough detail to allow replications AND will provide AP project files with the raw data. This is not the case here.

Re: replications, I've now tested three things that he posted here (the actual samples he tested), then sent me, and our measurements line up nearly perfectly.

Thanks for your test confirmations and replications. That's how science works.

 

[delta76]

The question would be similar to asking the following question on SBAF:
Just how legit are ASR test results?

When you trust Amir to do the test properly you will likely find the test 'legit'.
When you do not trust Amir to do the test properly you will likely find the test 'pure nonsensical and absurd'.

- He discloses his test methods/procedures clearly
- It has been proven many times that his test results are consistent with measurements from other sources(differences are within margin of errors)
- He has no financial interests on those tests - and he discloses any interests (if he received the sample from manufacturers, or if they have anything to do with his business). he's also well off (if not wealthy) so buying him off would be quite expensive

Building a reputation as a trustworthy reviewer is very difficult, and Amir managed to do that.

 

[ahofer]

In the end it's peer review.

Perhaps some ASR members should get together and replicate the SBAF test? The amps aren't expensive, and we could *invite* SBAFers to take part.

I'm not too interested in discussing motives or potential dishonesty. There are enough unconscious biases to justify verification of any single experiment by any human or group of humans.

 

[SIY]

Perhaps some ASR members should get together and replicate the SBAF test?

It cannot be replicated.

 

[solderdude]

The difference being that Amir outlines the test procedure in enough detail to allow replications AND will provide AP project files with the raw data. This is not the case here.

Re: replications, I've now tested three things that he posted here (the actual samples he tested), then sent me, and our measurements line up nearly perfectly.

Yes I know.

My point is that when that sme question was asked at SBAF about an ASR test there would be negative comments all around.
The question would be 'similar' and so would the responses, just not on technical grounds but because of the viewpoints of those forum dwellers.

 

[ahofer]

It cannot be replicated.

You know what I mean! Properly blind test those two amps and try to replicate (or not) his results.

 

[julian_hughes]

In the end it's peer review.

Perhaps some ASR members should get together and replicate the SBAF test? The amps aren't expensive, and we could *invite* SBAFers to take part.

I'm not too interested in discussing motives or potential dishonesty. There are enough unconscious biases to justify verification of any single experiment by any human or group of humans.

Totally agree. I suggested the same several days and many pages ago and I still think that would be worth more than any amount of theory and infinitely more than people claiming they know what other people did or did not hear.

 

[JayGilb]

Totally agree. I suggested the same several days and many pages ago and I still think that would be worth more than any amount of theory and infinitely more than people claiming they know what other people did or did not hear.

Should people invite their kids over to switch wires or perform the test correctly with an actual abx box ?

 

[julian_hughes]

Should people invite their kids over to switch wires or perform the test correctly with an actual abx box ?

It doesn't matter. It's just switching. If it's blind with no clues then it doesn't matter if it's done by a performing seal, a trained goat, or miserable man from a web forum who hates kids but has a lab coat.

 

[SIY]

It doesn't matter. It's just switching. If it's blind with no clues then it doesn't matter if it's done by a performing seal, a trained goat, or miserable man from a web forum who hates kids but has a lab coat.

I would point out again that doing it with human switchers will greatly reduce test sensitivity. Which was certainly a very curious aspect to this claim.

 

[Blumlein 88]

I would point out again that doing it with human switchers will greatly reduce test sensitivity. Which was certainly a very curious aspect to this claim.

Not if they are highly trained switchers able to swap wiring silently in one second.