• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Just how legit is this person's blind test results?

Galliardist

Major Contributor
Joined
Jun 26, 2021
Messages
2,558
Likes
3,273
Location
Sydney. NSW, Australia
You know what I mean! Properly blind test those two amps and try to replicate (or not) his results.
Doing a different test - which it would have to be, with different samples of the amps, listening to different sound through them, with different headphones - would prove nothing.

If you are going to take up the time of a group of people and set up a legit double blind test environment, then at lest run a test that tells the world something useful!
 
OP
nyxnyxnyx

nyxnyxnyx

Addicted to Fun and Learning
Joined
May 22, 2019
Messages
506
Likes
474
I would point out again that doing it with human switchers will greatly reduce test sensitivity. Which was certainly a very curious aspect to this claim.
then how about, doing that test but with improvements? I think since you can point out the flaws, you can also provide answers to make it better.
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,383
Likes
24,749
Location
Alfred, NY
then how about, doing that test but with improvements? I think since you can point out the flaws, you can also provide answers to make it better.
I'd suggest starting with a clear statement of the question to be answered before even thinking about experimental design. That has not yet been done.
 

Head_Unit

Major Contributor
Forum Donor
Joined
Aug 27, 2018
Messages
1,341
Likes
688
every criteria that needs to be met in order to create an ideal blind test
Well to have a valid scientific test, you can only change ONE variable, because if you change more than one variable you cannot know which of the variables (or multiple) caused the changes.

This is why most all audio testing is invalid as truly scientific testing, because knowledge about the equipment being changed could of course affect your perceptions on a psychological basis. You probably REALLY WANT that new $6400 Anthem AVM 70 + ATI 525NC combo to sound at least "$5300 better" than the old Denon AVR-X3600H...even if, well, it really just sounds different but not better.

My suggestion for testing in general is yes having someone invisibly change the cables or streamer or amp or whatever while you are out of the house, each day you don't know if they did or not, you just make notes. Each day you decide if the system sounds different or not, then you compare to their record of when they actually made changes. This wipes any "A/B/X testing is too pressured/hurried" "switching is not a natural way to listen" complaints.
 

Blumlein 88

Grand Contributor
Forum Donor
Joined
Feb 23, 2016
Messages
20,522
Likes
37,053
Well to have a valid scientific test, you can only change ONE variable, because if you change more than one variable you cannot know which of the variables (or multiple) caused the changes.

This is why most all audio testing is invalid as truly scientific testing, because knowledge about the equipment being changed could of course affect your perceptions on a psychological basis. You probably REALLY WANT that new $6400 Anthem AVM 70 + ATI 525NC combo to sound at least "$5300 better" than the old Denon AVR-X3600H...even if, well, it really just sounds different but not better.

My suggestion for testing in general is yes having someone invisibly change the cables or streamer or amp or whatever while you are out of the house, each day you don't know if they did or not, you just make notes. Each day you decide if the system sounds different or not, then you compare to their record of when they actually made changes. This wipes any "A/B/X testing is too pressured/hurried" "switching is not a natural way to listen" complaints.
Man such a thing in a sense has already been done. Differences were missed in such long term listening. Differences that were readily discernible in short term listening with quick switching. I know how intuitive it seems. I know how you feel it is so right in your guts. It just is not so. We should long ago have understood this as audiophiles and stop with the incessant whining for long term listening tests. You don't hear more with such tests, you hear far, far less. Such a protocol is simply so poor as to be not worth doing. So quit asking it be done.
 

jsrtheta

Addicted to Fun and Learning
Joined
May 20, 2018
Messages
936
Likes
991
Location
Colorado
Well to have a valid scientific test, you can only change ONE variable, because if you change more than one variable you cannot know which of the variables (or multiple) caused the changes.

This is why most all audio testing is invalid as truly scientific testing, because knowledge about the equipment being changed could of course affect your perceptions on a psychological basis. You probably REALLY WANT that new $6400 Anthem AVM 70 + ATI 525NC combo to sound at least "$5300 better" than the old Denon AVR-X3600H...even if, well, it really just sounds never but not better.

My suggestion for testing in general is yes having someone invisibly change the cables or streamer or amp or whatever while you ar, etc.e out of the house, each day you don't know if they did or not, you just make notes. Each day you decide if the system sounds different or not, then you compare to their record of when they actually made changes. This wipes any "A/B/X testing is too pressured/hurried" "switching is not a natural way to listen" complaints.
The originators and promoters of ABX (e.g., Dave Clark, Tom Nousaine, Peter Aczel, etc.) had no problem with the comparisons going over long periods of time, like months. Anyone who feels "pressured" or "hurried" is not taking a valid test.
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
The originators and promoters of ABX (e.g., Dave Clark, Tom Nousaine, Peter Aczel, etc.) had no problem with the comparisons going over long periods of time, like months. Anyone who feels "pressured" or "hurried" is not taking a valid test.
When the test remains blind and controlled it can be stretched over a long time.
This is very different from sighted comparisons over a long period.

Anecdotal: I one built a pre-amp with 2 circuits I could switch between. One DC coupled with OP227 and compound decoupling and another one using the TL072 with no power decoupling and a bunch of non-biased electrolytics in the signal path. Using a toggle switch at the rear of the pre-amp (so not visible)
I always could tell what was used and demonstrated the obvious differences to those that were interested. They all heard it too.
Then, one day I forgot it was in the 'poor' position and listened to it for months believing it was in the 'best' position.
I heard all of the 'goodness' in that period and none of the 'poor sound' I could so easily tell when switching. The 'poor' circuit was not that poor.

Also had a girlfriend that I instructed to switch cables every now and then when I was at work and keep log. Alas she only did this one time over the month yet I thought this all went as planned. My 'notes' on sound were worthless in the end as I thought I heard differences where there were none.
So... long term test says nothing you can hear differences where there are none but expect them to be.

The discussed test, however, has nothing to do with long term memory nor is that more 'accurate'. The test also was not immediate. No idea how smart the ABX box is (monitoring levels) that might be used for part trois.

If he can still do it then an investigation should be started as to the why.
 

jsrtheta

Addicted to Fun and Learning
Joined
May 20, 2018
Messages
936
Likes
991
Location
Colorado
When the test remains blind and controlled it can be stretched over a long time.
This is very different from sighted comparisons over a long period.

Anecdotal: I one built a pre-amp with 2 circuits I could switch between. One DC coupled with OP227 and compound decoupling and another one using the TL072 with no power decoupling and a bunch of non-biased electrolytics in the signal path. Using a toggle switch at the rear of the pre-amp (so not visible)
I always could tell what was used and demonstrated the obvious differences to those that were interested. They all heard it too.
Then, one day I forgot it was in the 'poor' position and listened to it for months believing it was in the 'best' position.
I heard all of the 'goodness' in that period and none of the 'poor sound' I could so easily tell when switching. The 'poor' circuit was not that poor.

Also had a girlfriend that I instructed to switch cables every now and then when I was at work and keep log. Alas she only did this one time over the month yet I thought this all went as planned. My 'notes' on sound were worthless in the end as I thought I heard differences where there were none.
So... long term test says nothing you can hear differences where there are none but expect them to be.

The discussed test, however, has nothing to do with long term memory nor is that more 'accurate'. The test also was not immediate. No idea how smart the ABX box is (monitoring levels) that might be used for part trois.

If he can still do it then an investigation should be started as to the why.
The actual "official" ABX machine did the switching so you wouldn't even know if it was amp "B" you were listening or if you were still hearing amp "A". Of course, the idea was to tell which, "A" or "B", was in fact "X".

Of course, there is always the Clark Johnsen triple blind technique...
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
The actual "official" ABX machine did the switching so you wouldn't even know if it was amp "B" you were listening or if you were still hearing amp "A". Of course, the idea was to tell which, "A" or "B", was in fact "X".

Of course, there is always the Clark Johnsen triple blind technique...
No it didn't.
He took delivery after his test was done (involving kids to swap wires and using a regular switch which is not needed with the ABX box at all and switching can be instant)

And blind AB (actually XX) as one has to determine which amp was under A and which one under B at different times under the test. It doesn't have to be ABX nor double blind or triple blind to be valid, The proper controls just have to be in place.

If Marvin passes part trois and posts actual measurements (using his AP555) under actual load and shows channel imbalance was not an issue (showing L and R of the signal to the headphone under load using AP555 dashboard) and doing 20 attempts and acing or even raising doubt it would make me highly interested.
Unfortunately there is no way we can 'work' with Marvin to do some truthfinding as the why of it. Nor is he likely to repeat the test in front of a knowledgeable person checking and witnessing the test.

This all is not going to happen under the premise... I have shown it now YOU prove me why I can.
 

jsrtheta

Addicted to Fun and Learning
Joined
May 20, 2018
Messages
936
Likes
991
Location
Colorado
No it didn't.
He took delivery after his test was done (involving kids to swap wires and using a regular switch which is not needed with the ABX box at all and switching can be instant)

And blind AB (actually XX) as one has to determine which amp was under A and which one under B at different times under the test. It doesn't have to be ABX nor double blind or triple blind to be valid, The proper controls just have to be in place.

If Marvin passes part trois and posts actual measurements (using his AP555) under actual load and shows channel imbalance was not an issue (showing L and R of the signal to the headphone under load using AP555 dashboard) and doing 20 attempts and acing or even raising doubt it would make me highly interested.
Unfortunately there is no way we can 'work' with Marvin to do some truthfinding as the why of it. Nor is he likely to repeat the test in front of a knowledgeable person checking and witnessing the test.

This all is not going to happen under the premise... I have shown it now YOU prove me why I can.
I was simply describing how the original machine worked. That's all.
 

Newman

Major Contributor
Joined
Jan 6, 2017
Messages
3,449
Likes
4,210
we made tremendous progress from "all amps sound the same" to "look, there's Benchmark and this $13000 Mola Mola".
Well the Lenient criteria are not there as some joke, or only for background music. They are really relevant. Think of them as for real-life music listening with full attention and a critical ear.

So the Lenient criteria still allows probably 90% of all amps designed for transparency and 98% of all DACs designed for transparency, to be audibly transparent and hence indistinguishable for practical purposes.

So whoops, we are back from your 2 devices to probably thousands.
 

voodooless

Grand Contributor
Forum Donor
Joined
Jun 16, 2020
Messages
10,223
Likes
17,800
Location
Netherlands
  • Like
Reactions: VQR

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,383
Likes
24,749
Location
Alfred, NY
They used DAC’s with NOS filters :facepalm:

What’s the point of testing stuff that is broken by default?
The displayed spectrum was terrible. I can readily believe that such a poor performer will sound different than a competent unit.
 

voodooless

Grand Contributor
Forum Donor
Joined
Jun 16, 2020
Messages
10,223
Likes
17,800
Location
Netherlands
The displayed spectrum was terrible. I can readily believe that such a poor performer will sound different than a competent unit.
My guess isn’t more a frequency response thing. Since NOS filters are never really NOS, they all exhibit a different response.
 

solderdude

Grand Contributor
Joined
Jul 21, 2018
Messages
15,891
Likes
35,912
Location
The Neitherlands
Here is another blind test in the same forum using an ABX comparator between two DAC's. They were level matched to 0.031dB. Result: 23/24 correct.

Does ASR think this is a convincing demonstration that differences between DAC's is audible?

(Edit) forgot to include the link: https://www.superbestaudiofriends.o...audio-spring2-kte-listening-evaluation.10989/

A can't say what ASR thinks but here's what I think.

Bob I trust to create and do a valid blind test and has a genuine interest in this and knows how to measure.
Upsampling was used but it seems like with using a slow filter. Given Bob's age I reckon the treble roll-off would be above his hearing range.
The choice of reconstruction filters used is dubious.

index.php

Note there is a difference in FR. Wavelight & Holo
12kHz -0.5dB
vs 22kHz - 0.5dB

Have a look at the difference in squarewave response between these 2 DACs. Wavelight - KTE Spring 2
index.php

So here too there is quite a lot of measurable differences between these 2 DACs.

Bob's test was done 2 years ago b.t.w.

This was not the case with Marvin's amp test. The only difference there was distortion and acc. to Marvin this should be inaudible. I would not trust Marvin but do trust Bob.
Strange to decide to compare 2 DACs used that clearly have a very measurable FR.

I would recommend to AB the same FR filtering of 1 DAC where only FR was changed similar to what is shown above to eliminate the chance of that difference being responsible.
 
Last edited:

Galliardist

Major Contributor
Joined
Jun 26, 2021
Messages
2,558
Likes
3,273
Location
Sydney. NSW, Australia
Here is another blind test in the same forum using an ABX comparator between two DAC's. They were level matched to 0.031dB. Result: 23/24 correct.

Does ASR think this is a convincing demonstration that differences between DAC's is audible?

(Edit) forgot to include the link: https://www.superbestaudiofriends.o...audio-spring2-kte-listening-evaluation.10989/
Remember - the proposition is that if we remove all the known audible differences between two DACs and operate them correctly, they should sound the same. It's unlikely with this pairing that we can remove all the known audible differences, therefore there is no remarkable claim here for audibility.

I think we can let pass the claim that the level of performance of these two amps is "very fine", since that is subjective.

As before, we don't have full measurements so the test is not completely reported in a way we can evaluate. It looks better than the other one.
 

krabapple

Major Contributor
Forum Donor
Joined
Apr 15, 2016
Messages
3,168
Likes
3,714
Here is another blind test in the same forum using an ABX comparator between two DAC's. They were level matched to 0.031dB. Result: 23/24 correct.

Does ASR think this is a convincing demonstration that differences between DAC's is audible?

(Edit) forgot to include the link: https://www.superbestaudiofriends.o...audio-spring2-kte-listening-evaluation.10989/

No one says devices *designed* to sound nontransparent ( tube amps, silly DACs like this one) should not sound different from devices aiming for audible transparency

Can we move beyond these crummy, atypical but not surprising examples?
.
 

VQR

Active Member
Joined
Oct 15, 2021
Messages
141
Likes
332
As before, we don't have full measurements so the test is not completely reported in a way we can evaluate. It looks better than the other one.
Purr1n strikes me as the type to project. Notice how he complains ad nauseum about ASR cooking measurements, yet neglects to note his test conditions in his own measurements. I remember he did a graph showing differences between cables, but failed to note amplitude or frequency. (EDIT: he did note amplitude, just not frequency. Graph is in post 6. No bandwidth detailed... wanna bet the measurement was in the RF range? I doubt this was unintentional.)

Since any valid criticism of his methods are met with bans and scorn, I laugh at him calling ASR a cult. Last I checked, ASR isn't the one attributing mystical properties to Schitt nor simping for particular brands. The casual racism too is a massive red flag for anyone reading criticisms of Amir at SBAF.
 

Attachments

  • variance.png
    variance.png
    67.8 KB · Views: 41
Last edited:
Top Bottom