• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Blind test: we have a volunteer!!!

Status
Not open for further replies.

Jdunk54nl

Addicted to Fun and Learning
Joined
Aug 5, 2020
Messages
968
Likes
1,048
Location
Arizona
Last edited:

Kervel

Active Member
Forum Donor
Joined
May 20, 2019
Messages
112
Likes
139
Say you were testing 2 devices in ABX. The test asks you to rank their qualities and your preference on a 1-10 scale instead of asking if what you hear is same/different.

If you are consistent in your rankings, you will be able to determine whether or not you can hear a difference, if there are assignable qualities, and if you're able to produce a preference.

Exactly, ABX protocol identifies differences between amps. To test that Schiitt sounds bad (a much stronger claim--not only a difference but also a preference), we need a different protocol. For example, let X1 be randomly amp A or B; and Y1 the other amp. Golden can listen and switch between X1 and Y1 and needs to guess which amp sounds best. Next round again let nature choose whether X2 is A or B, set Y2 as the other amp, and repeat the test.
 

DimitryZ

Addicted to Fun and Learning
Forum Donor
Joined
May 30, 2021
Messages
667
Likes
342
Location
Waltham, MA, USA
An apology from @GoldenOne and a retraction of a substantial financial exposure from @amirm would be the best way to proceed, IMHO.

Then the parties should name a pair of charities for the members to donate to.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,394
Location
Seattle Area
Amir bailed on the opportunity to have someone he trusts present at throughout the testing process to verify legitimacy of the test beyond what can be captured on video, allowing more room for interpretation and more opportunities to consider the provided evidence to be inconclusive.
Please don't make up nonsense. No one bailed on anything. If there is a volunteer and wants to attend his video production, they are more than welcome and encouraged to do so.

What I have done is to say that I can't put $1,000 on the line for a simple "different or not" test which has protocols that may be in error, thus making such a test trivial to pass. If our blogger wants to proceed without the $1000, I am still happy to critique his test protocol and help in any way I can to make his test more valid. I am sure others would too.

Indeed, an ABX or even AB test like this is a good filter to know if there is any "there there." To the extent he can't pass this test, then we are done. If he passes it, then we have phase to 2 of this test to show that he can identify that the impairments he heard in Magnius are there. To the extent this is a review he has done and is representing it as durable facts -- like measurements are -- then he should be able to show that others can hear these problems too.

To that end, in his AB testing, I highly suggest he not only test himself but whoever else is in the room. This help filter out some of the "tells" as others may not be aware of them as he might be.
 

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
Please don't make up nonsense. No one bailed on anything. If there is a volunteer and wants to attend his video production, they are more than welcome and encouraged to do so.

What I have done is to say that I can't put $1,000 on the line for a simple "different or not" test which has protocols that may be in error, thus making such a test trivial to pass. If our blogger wants to proceed without the $1000, I am still happy to critique his test protocol and help in any way I can to make his test more valid. I am sure others would too.

Indeed, an ABX or even AB test like this is a good filter to know if there is any "there there." To the extent he can't pass this test, then we are done. If he passes it, then we have phase to 2 of this test to show that he can identify that the impairments he heard in Magnius are there. To the extent this is a review he has done and is representing it as durable facts -- like measurements are -- then he should be able to show that others can hear these problems too.

To that end, in his AB testing, I highly suggest he not only test himself but whoever else is in the room. This help filter out some of the "tells" as others may not be aware of them as he might be.
As one of the claims in his video was 'sounds different despite great measurements' then testing that falls within scope of your original challenge. That leaves procedural matters to be agreed.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,394
Location
Seattle Area
The issue of "tells" is a big problem and needs very skilled people to tease them out. The best example is this blind test conducted by Boston Audio Society years ago: https://www.bostonaudiosociety.org/bas_speaker/abx_testing2.htm

"At this point I [test conductor] thought that I could reliably distinguish between the "A" and "B" paths on the basis of the slight noise level increase which occurred when the PCM-F1 was inserted into the chain, and which was marginally audible due to the high gain of the Naim MAP 250 power amplifier combined with the low peak signal levels through the F1, which the peak-hold meters showed to have risen no higher than -20 dB. (0 dB is the digital clip point, and these peak levels were somewhat unfair to the digital processor since 20 dB of its signal-to-noise ratio was being thrown away.) [In other words, for this segment of the test the F1 was in effect a 13-bit processor. - Ed.]

I expressed my desire to try the test, and Remington went to cue up the record again, but I requested to be allowed to undertake the test with no signal passing though the system. Before realizing the import of what he was saying, Vanderkooy interjected: "Ah! You're going to listen to the sound of the relays." Yes, there is indeed a slight audible difference between the acoustic "click" made when the "A" and "B" relays pull in. This is due to the unavoidable differences in the mounting positions of the relay on the A/B/X box chassis and, although slight, it can be heard if one listens for it. I replied that I was going to listen to the difference in background hiss, and the subsequent series of blind trials showed conclusively that the two signal paths could be reliably distinguished on this basis alone."


Professor Vanderkooy is one of our audio luminaries by the way when it comes to signal processing. He was brought into the testing as another person to make sure the test is correct and indeed he found that critical flaw. This is why I earlier mentioned that our tester's "friend" is not going to do. We need sharp eyes from someone not his camp to look for such things.

As it happens, the person that was being tested was oblivious to such tells and went ahead to embarrass himself by not being able to tell the difference. I am not going to give that benefit of doubt to our blogger.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,394
Location
Seattle Area
As one of the claims in his video was 'sounds different despite great measurements' then testing that falls within scope of your original challenge.
No it doesn't. I was very specific on what my beef was in his video in the other thread:

"the dynamic range feels quite compressed...it doesn't come across in measurements"

"... this is a fatiguing amp to me..."

"there is slight graininess and lack of separation...."

"there is not a lot of texture [in bass]..."

"timbre is not great in this amp..."

"mid texture and detail is not good..."

And this is in the first 7 minutes of this video!

I was challenged on this by another poster and I presented the scenario of showing these in controlled testing in exchange for $1000 donation.

By your standard, if he had said the weather was cloudy, all he had to do was prove that it would be just as well.

A video where someone said two devices measure the same but in very intensive study he could barely make up a difference between them would not have garnered much response from me let alone such a challenge. He said all kinds of outrageous things for nearly 20 minutes. You have to be really, really on his side to want to shrink this down to "oh can he tell the difference."
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,394
Location
Seattle Area
I think that GO is fully within his rights to remain anonymous at this point in time considering the fact that his original MQA video may have had an adverse impact on Tidal, MQA and any manufacturers with MQA gear. Realistically, it may have not been a significant impact as any publicity is good publicity, but some companies may not see it this way.
It is within his rights to stay anonymous and it is also within our rights to put doubt as to his level of ethics. He examined my background and pulled it into his audio arguments, but says that must not happen to him.

In a test that is remote without me having an ability to attend, ethics plays a super important role. Take that away and the test gets much harder because you have to account for everything. And I mean everything. This is why Casinos not only have people looking over the gamblers to make sure they are not cheating, they have people looking over these people to make sure they are not in cahoots with the gambler! Once trust is thrown out the window, you better check, double and triple check.
 

Andysu

Major Contributor
Joined
Dec 7, 2019
Messages
2,887
Likes
1,478
Someone here is really strapped for money for a audiophile double blind listening $£ test.
 

bboris77

Senior Member
Joined
Oct 23, 2018
Messages
444
Likes
931
It is within his rights to stay anonymous and it is also within our rights to put doubt as to his level of ethics. He examined my background and pulled it into his audio arguments, but says that must not happen to him.

In a test that is remote without me having an ability to attend, ethics plays a super important role. Take that away and the test gets much harder because you have to account for everything. And I mean everything. This is why Casinos not only have people looking over the gamblers to make sure they are not cheating, they have people looking over these people to make sure they are not in cahoots with the gambler! Once trust is thrown out the window, you better check, double and triple check.
I get what you are saying, that is why I suggested having someone you personally know in the UK supervise the test.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,394
Location
Seattle Area
As a way of background, I have offered thousands of dollars for such tests in the past. The protoco
A DBX shows that a person can tell a difference. Not if something sounds better or even if something has the qualities they claim. That is up to the person's subjective impressions.
This is true but let's again remember that we are dealing with a reviewer here, not an individual who expresses an opinion. This reviewer has claimed that if measurements are bad, it means the device is bad. But if measurements are good they may a) be gamed by me for an ulterior reason and b) his ears to declare the truth. By definition then, he says his ears are proxy for all others and best we watch his videos and get to that.

This then calls for us verifying whether his impressions are durable and translatable to others. This is why the rest of us listening to what he heard is a super valid aspect of this testing. If it turns out that nobody can hear this "bass compression" or "grittiness" then that alone puts huge doubt in our blogger's position. And disqualifies him from earning the $1,000 for his charity.

This is the key thing that is at stake here. What a random anonymous person thinks or doesn't think about some amp is of zero importance. What this blogger thinks, is not that. He has made a challenge and put forth theories of why he is right. Let's work through it and figure this out.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,394
Location
Seattle Area
I get what you are saying, that is why I suggested having someone you personally know in the UK supervise the test.
I know people there but not with the right qualifications to catch the type of intricate problems we can be dealing with. I have made this offer in the past but it always included me conducting the test/switching. In this case that is not possible so I created this thread to see what we can get done.
 

Inner Space

Major Contributor
Forum Donor
Joined
May 18, 2020
Messages
1,285
Likes
2,938
The issue of "tells" is a big problem and needs very skilled people to tease them out. The best example is this blind test conducted by Boston Audio Society years ago: https://www.bostonaudiosociety.org/bas_speaker/abx_testing2.htm

I remember reading about that test at the time. Afterward Tiefenbrun came out with a series of lame excuses - mostly that his ears were blocked up after the transatlantic flight. He really was the worst character and the prime mover behind the modern era of audio BS. People remember Jean Hiraga and others, but no one did more damage than Tiefenbrun.

For several years before that, there was an orgy of rebuilding and re-equipping in the studio world, because digital was here to stay. I recall frequent blind tests between one thing and another, most of which were fruitless, because of subtle pops and clicks and hums and hisses. My takeaway was it's much easier to pass a real-world blind test, than it is to design a completely valid one in the first place.
 

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
By your standard, if he had said the weather was cloudy, all he had to do was prove that it would be just as well.
Better if you don't mischaracterise my argument.

The observation was "this measures great, that measures great, those two measure great, that DAC and that DAC measure great, but they all sound different: measurements don't tell the full story ..."

That's not a conversation about the weather. You can test the observation by detecting difference between items that measurements indicate should be transparent.

Given that the discussed test protocols have no method for differentiating the observed subjective characteristics that you specifically object to, but merely support declaring difference, how can anyone actually take up the challenge, if the above is insufficient for you?

Was the challenge merely rhetorical?
 

bboris77

Senior Member
Joined
Oct 23, 2018
Messages
444
Likes
931
On a related note, we really need to find a simple A/B switch with either 1/8 or 1/4 jacks that does not have any crosstalk between the inputs and does not affect the signal in any way at all. Plus, it cannot have any weird switching anomalies which would enable the listener to figure out which input is being used. For example, one of my RCA switchers with a rotary selector switch would crackle when I switched from Input 2 to Input 1, but then not crackle when I went from Input 1 to Input 2. Go figure.

It would help immensely when it comes to performing blind tests of amplifiers.
 

BrEpBrEpBrEpBrEp

Active Member
Joined
May 3, 2021
Messages
201
Likes
245
On a related note, we really need to find a simple A/B switch with either 1/8 or 1/4 jacks that does not have any crosstalk between the inputs and does not affect the signal in any way at all. Plus, it cannot have any weird switching anomalies which would enable the listener to figure out which input is being used. For example, one of my RCA switchers with a rotary selector switch would crackle when I switched from Input 2 to Input 1, but then not crackle when I went from Input 1 to Input 2. Go figure.

It would help immensely when it comes to performing blind tests of amplifiers.
I created the below thread to gather such devices and hopefully find members who own them and can test/measure them.

https://www.audiosciencereview.com/forum/index.php?threads/options-for-cheap-abx-boxes.23925/
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,394
Location
Seattle Area
The observation was "this measures great, that measures great, those two measure great, that DAC and that DAC measure great, but they all sound different: measurements don't tell the full story ..."
That's your observation, not mine. I post what I observed which is what got challenged and bet was put on the table for proper test. Go ahead and create your challenge and don't pollute this one with you want.
 

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
That's your observation, not mine. I post what I observed which is what got challenged and bet was put on the table for proper test. Go ahead and create your challenge and don't pollute this one with you want.
No, that observation is from the transcript of GO's video.
 

krabapple

Major Contributor
Forum Donor
Joined
Apr 15, 2016
Messages
3,169
Likes
3,715
I love Americans, I really do. But their propensity to throw down cash bets on just about anything in order vainly attempt to prove a point is terribly tiresome. It's essentially a pathetic "my c#ck is bigger than yours unless you show me" type challenge.

:rolleyes:

Of all the peoples I've visited, on six continents, the ones who reminded me the most of Americans, *by far*, are Aussies.
 

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
:rolleyes:

Of all the peoples I've visited, on six continents, the ones who reminded me the most of Americans, *by far*, are Aussies.
Can't decide whether to reply "oi, you calling me a seppo, mate?" or "wanna bet?".
 
Last edited:
Status
Not open for further replies.
Top Bottom