Blind test: we have a volunteer!!!

[amirm]

I am sorry, but you are testing for experiential judgement -from a single individual, no less.

You can't externalize this test, by definition.

We are testing the amp, not the headphone. The amp's performance is not dependent on the person using it.

 

[DimitryZ]

We are testing the amp, not the headphone. The amp's performance is not dependent on the person using it.

It's not.

But the perception is based on a human being hearing this amp with particular speakers/headphones. And you specifically framed this challenge as perceptual to a specific person (GS) who is a human who has to use speakers/phones to "perceive" this amp. This is very much defintional.

I gently suggest that both of you have a polite and private conversation that would define the test parameters. I recommend that money is taken out of the equation if apologies are offered. You both have my messages outlining the likely parameters of the deal. Can you both pleasantly surprise us all by actually collaborating?

It's the right thing to do. And serious charity money could be raised, in a fun and productive way!

 

[solderdude]

No speakers are used in this test. Just high-end planar headphones.

When the only difference in a test is an amplifier which has the exact same (in this case purely resistive) load and the headphone isn't even moved, the amplifiers in question provide the exact same volume, the used auditory system is the same one and time is spread over the test (AB switching several times) then the only variable in that specific test is the used amplifier.

It really does not matter how 'wonky' the FR is because it will be equally wonky on both amps during evaluation.
So when he 'evaluates' bass, mids, treble, cleanness, bass impact, air, imaging even PRaT and he consistently rates it differently for both amps it is basically the same as comparing A to B at the moment of switching. The headphone is out of the equation.

I get your point when you would have him describe the sonic signatures of both amps and then compare it to that of others using different headphones. That's not the test so irrelevant.

@BrEpBrEpBrEpBrEp suggestion to compare notes of perceived SQ determined statistically often enough when not knowing is accurate enough.

 

[amirm]

But the perception is based on a human being hearing this amp with particular speakers/headphones. And you specifically framed this challenge as perceptual to a specific person (GS) who is a human who has to use speakers/phones to "perceive" this amp. This is very much defintional.

Whatever his experience is, we are asking him to repeat exactly that without knowing which is which. We also sampling the stimulus that is driving his headphone. That is the data we need to find out if his perception are telling him what the truth or creative stories.

We do this all the time in audio research. There is nothing unusual and different here.

 

[amirm]

I gently suggest that both of you have a polite and private conversation that would define the test parameters.

Once more, there will be NO private conversations. Last time we (my moderator) tried that, the conversation was recorded without his permission and spun into a story by the blogger. So now, everything will be in writing and in public.

Even if the above was not an issue, I want the benefit of the membership to figure out what to do. There are a lot of variables here and it will take a collective action to figure this out. If he and I just cooked up something, it could very well have flaws in it that could have been avoided.

 

[DimitryZ]

It's not.

But the perception is based on a human being hearing this amp with particular speakers/headphones. And you specifically framed this challenge as perceptual to a specific person (GS) who is a human who has to use speakers/phones to "perceive" this amp. This is very much defintional.

I gently suggest that both of you have a polite and private conversation that would define the test parameters. I recommend that money is taken out of the equation if apologies are offered. You both have my messages outlining the likely parameters of the deal.

It's the right thing to do. And serious charity money could be raised, in a fun and productive way!

Whatever his experience is, we are asking him to repeat exactly that without knowing which is which. We also sampling the stimulus that is driving his headphone. That is the data we need to find out if his perception are telling him what the truth or creative stories.

We do this all the time in audio research. There is nothing unusual and different here.

Surely you accept that GS' stimulus involves a pair of headphones.

Standard DOE practice strongly suggests that the headphone effect is held constant over the experiment, so as to not confound the effect under investigation. This is what my graduate prof taught me at UMASS/LOWELL

 

[noobie1]

Once more, there will be NO private conversations. Last time we (my moderator) tried that, the conversation was recorded without his permission and spun into a story by the blogger. So now, everything will be in writing and in public.

Even if the above was not an issue, I want the benefit of the membership to figure out what to do. There are a lot of variables here and it will take a collective action to figure this out. If he and I just cooked up something, it could very well have flaws in it that could have been avoided.

Transparency is the best policy.

 

[DimitryZ]

Once more, there will be NO private conversations. Last time we (my moderator) tried that, the conversation was recorded without his permission and spun into a story by the blogger. So now, everything will be in writing and in public.

Even if the above was not an issue, I want the benefit of the membership to figure out what to do. There are a lot of variables here and it will take a collective action to figure this out. If he and I just cooked up something, it could very well have flaws in it that could have been avoided.

If there to be no private understanding between you and GS, please officially call of your challenge.

The membership has no involvement here. This is entirely between you and GS (and I am not a fan of the latter) to work it out.

Again, you two can pleasantly surprise everyone by warking it out or reinforce the status quo.

Regards,

DZ

 

[BrEpBrEpBrEpBrEp]

Surely you accept that GS' stimulus involves a pair of headphones.

Standard DOE practice strongly suggests that the headphone effect is held constant over the experiment, so as to not confound the effect under investigation. This is what my graduate prof taught me at UMASS/LOWELL

Even if done digitally, we wouldn't switch headphones halfway through - so whichever is used remains constant throughout.

To my understanding, unless we're positing that a specific interaction between the headphones and amp will be different enough from one amp (Magnius) to the other to cause every difference mentioned in the video, it shouldn't matter.

 

[markus]

Surely you accept that GS' stimulus involves a pair of headphones.

Standard DOE practice strongly suggests that the headphone effect is held constant over the experiment, so as to not confound the effect under investigation. This is what my graduate prof taught me at UMASS/LOWELL

This is virtually impossible when the headphone is reseated. Anybody that has done headphone measurements knows that measured results differ by reseating the headphone.

Anyhow, this reseating effect might be all a subjective reviewer ever really experiences. So I strongly suggest to first find a simpler test that validates if there's an audible difference between amps at all.

Proposal:
Put the headphone on a measuring jig, e.g. this one. Make sure it never ever gets touched throughout the test. Plug headphone in amp A. Then record music samples. Plug headphone in amp B (don't forget to level match) and repeat recordings. Make recorded samples available online so people can ABX them on their computer.

 

[DimitryZ]

This is virtually impossible when the headphone is reseated. Anybody that has done headphone measurements knows that measured results differ by reseating the headphone.

Anyhow, this reseating effect might be all a subjective reviewer ever really experiences. So I strongly suggest to first find a simpler test that validates if there's an audible difference between amps at all.

Proposal:
Put the headphone on a measuring jig. Make sure it never ever gets touched throughout the test. Plug headphone in amp A. Then record music samples. Plug headphone in amp B (don't forget to level match) and repeat recordings. Make these samples available online so people can ABX them on their computer.

If "headphone reseating" is an actual DOE variable, can we all just stop this inquiry?

If there is no way to change the main effect without changing the confounding effect, then a much wider experimental matrix is warranted.

 

[BrEpBrEpBrEpBrEp]

This is virtually impossible when the headphone is reseated. Anybody that has done headphone measurements knows that measured results differ by reseating the headphone.

Anyhow, this reseating effect might be all a subjective reviewer ever really experiences. So I strongly suggest to first find a simpler test that validates if there's an audible difference between amps at all.

Proposal:
Put the headphone on a measuring jig. Make sure it never ever gets touched throughout the test. Plug headphone in amp A. Then record music samples. Plug headphone in amp B (don't forget to level match) and repeat recordings. Make these samples available online so people can ABX them on their computer.

In that case, I would do it twice - record on A, then on B, then on A again (A'), then on B again (B'). Then null A and A', as well as B and B', to make sure that no external factors influenced the recordings (outside noise, jostling the fixture, etc.). Then, if the nulls are satisfactory, A and B could be AB tested.

However, this brings up a big issue - you'd need a measurement fixture with microphones of a sufficient quality to do this properly, as well as a quiet enough room.

If I were in GS' place, I wouldn't be bothered with that, personally. In this hypothetical, I could first compare the recordings of the amps (directly into an ADC) sighted, then blind AB them. I'd then decide if I wanted to proceed with more complex tests.

 

[Roland]

From the preceding discussions we can conclude that:

there is no evidence that amps which measure the same sound the same,

it would be inordinately difficult to establish such evidence, and

this basic tenet of ASR is just a presumption.

Therefore, a Denon AVC-X3700H may sound the same as a Denon PMA-A110, or it might not.

 

[BrEpBrEpBrEpBrEp]

From the preceding discussions we can conclude that:

there is no evidence that amps which measure the same sound the same,

it would be inordinately difficult to establish such evidence, and

this basic tenet of ASR is just a presumption.

Therefore, a Denon AVC-X3700H may sound the same as a Denon PMA-A110, or it might not.

You cannot prove a statement like "amps which measure the same sound the same", it's practically completely unconstrained.

You can't even prove two amps "sound the same". It's basically proving a negative (prove no difference could ever be perceived).

What you can do is establish thresholds for human hearing through extensive studies, and say with a very large degree of confidence that devices with differences below those thresholds will sound the same (edit: this is more or less general practice on this forum).

Or you can do a sighted test and say "these definitely sound different". It's basically worth nothing scientifically (or practically), but you can do it.

These blind tests have been done before, anyways. It's not like anything in this thread is groundbreaking new science (lmao).

 

[markus]

In that case, I would do it twice - record on A, then on B, then on A again (A'), then on B again (B'). Then null A and A', as well as B and B', to make sure that no external factors influenced the recordings (outside noise, jostling the fixture, etc.). Then, if the nulls are satisfactory, A and B could be AB tested.

Good idea. Do several recordings throughout a day to eliminate environmental noise contamination. Those samples could also be used to do a basic null test between A and B. Maybe there's no difference at all and no listening test is warranted.

However, this brings up a big issue - you'd need a measurement fixture with microphones of a sufficient quality to do this properly, as well as a quiet enough room.

This is definitely a variable that needs to be controlled. But if there's an audible difference it should still be detectable even through a bad recording. Just like we can listen through a bad room. In any case use the best available measuring jig.

If I were in GS' place, I wouldn't be bothered with that, personally. In this hypothetical, I could first compare the recordings of the amps (directly into an ADC) sighted, then blind AB them. I'd then decide if I wanted to proceed with more complex tests.

That's the first thing he should have done before broadcasting to the planet in videos describing his subjective experience in flowery language.

 

[restorer-john]

The entire premise that amplifiers level matched and run under clipping are indistinguishable from one another is a narrative prosecuted by people who don't own a lot of amplifers or the ability to switch between them and most likely have never instantly compared several amplifiers, run at varying levels, into various loudspeakers.

You can't even prove two amps "sound the same".

Yes you can. You run the two amplifiers in opposite polarity and listen to the difference signal into speakers (hots only connected*). If the output is silence, the amplifers sound the same. If there is a residual, you have an audible difference.**

*not full bridge (BTL amps)
**amplifiers using onboard D/As etc, this will rarely work, due to different latency issues.

 

[BrEpBrEpBrEpBrEp]

The entire premise that amplifiers level matched and run under clipping are indistinguishable from one another is a narrative prosecuted by people who don't own a lot of amplifers or the ability to switch between them and most likely have never instantly compared several amplifiers, run at vary levels, into loudspeakers.



Yes you can. You run the two amplifiers in opposite polarity and listen to the difference signal into speakers. If the output is silence, the amplifers sound the same. If there is a residual, you have an audible difference.

That's an excellent point actually, forgot about that - though some people will find a way to complain about a null test. However, that only proves they sound the same to you. I'm sure that you could, to a degree, prove that they sound the same to everyone, but you'd need more than a listening test, unless everyone took the test.

 

[DimitryZ]

From the preceding discussions we can conclude that:

there is no evidence that amps which measure the same sound the same,

it would be inordinately difficult to establish such evidence, and

this basic tenet of ASR is just a presumption.

Therefore, a Denon AVC-X3700H may sound the same as a Denon PMA-A110, or it might not.

As a new member that uses both objective and subjective criteria,

Similar measuring items also sound similar.

I was unpleasantly surprised that ASR doesn't use DBTs as a matter of course.

 

[BrEpBrEpBrEpBrEp]

This is definitely a variable that needs to be controlled. But if there's an audible difference it should still be detectable even through a bad recording. Just like we can listen through a bad room. In any case use the best available measuring jig.

Actually, environmental sounds would ruin a blind test by introducing unrelated differences to the recordings, which makes this method difficult. That's my primary issue with non-soundproofed rooms for this, and why I persist to think it's best to start off by doing a straight amp to ADC recording, and move from there.

 

[markus]

From the preceding discussions we can conclude that:

there is no evidence that amps which measure the same sound the same,

it would be inordinately difficult to establish such evidence, and

this basic tenet of ASR is just a presumption.

Therefore, a Denon AVC-X3700H may sound the same as a Denon PMA-A110, or it might not.

All properly conducted listening tests point in the other direction I'm afraid.
And, it's rather easy to do such ABX tests. Audiophiles spend a lot more time and money on things that don't improve anything.