• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required as is 20 years of participation in forums (not all true). Come here to have fun, be ready to be teased and not take online life too seriously. We now measure and review equipment for free! Click here for details.

Blind test: we have a volunteer!!!

Status
Not open for further replies.

solderdude

Major Contributor
Joined
Jul 21, 2018
Messages
9,116
Likes
19,445
Location
The Neverlands
Discussing methodology and how to ensure a fair test is clearly very important and of value.
That it is. I haven't seen the 'rules' and methods laid out yet.
Nor has @GoldenOne participated in this test.

I don't think the test is going to go ahead and if it does what the agreed upon testing looks like and is monitored.
Glad to see my hunch was wrong.

Edited.
 
Last edited:

bboris77

Senior Member
Joined
Oct 23, 2018
Messages
360
Likes
757
I mean a cursory scroll through this thread shows that this is already impossible. It's a shame how childish and petty people are on this website sometimes, reminds me of a school playground. All the criticism of what 'he/they will say when he gets it wrong' and the guy hasn't even said anything of the sort yet. What about if he gets it right? You think people here won't do the exact same thing and come out with a multitude of different excuses/explanations? Personally it's something I struggle to believe without seeing someone do it myself, we have the opportunity to do that. Can we at least prevent the inevitable shitshow that will ensue, regardless of the results, until the actual results are out? Discussing methodology and how to ensure a fair test is clearly very important and of value. The underlying scorn and sneering, in my opinion, not so much
Agree with taking out the negative attitude out of it. It’s not like @GoldenOne said he’d be bending spoons or anything. It is entirely plausible that, under the right circumstances, a trained listener can successfully distinguish between specific amps/DACs.

The purpose of this test is to establish what the threshold is - what the difference is in measured performance at which one can reliably tell two devices apart. It could be that it has to be very large, in which case it would be confirmed that the objective argument has been right all along.

If it is proven that a perceptive listener can reliably tell a difference between two devices that measure very similarly and above the transparency threshold, that would mean that there is something else that is not being measured. While it is unlikely this will happen, it is possible.

Personally, I think that @GoldenOne should be allowed to pick the devices in question because he is the one that claims to be able to distinguish between them. If he passes the “test”, then these particular devices should be measured to ensure that they both measure above the traditional threshold of transparency. If they do, it means that different measurements need to be devised that would account for this.

I guess my general point is that this should be a collaborative rather than confrontational effort. All in the interest of exploration of this area rather than proving someone is wrong. Ideally, both sides would come out of the process agreeing with whatever the outcome is.
 
Joined
Apr 19, 2021
Messages
48
Likes
77
Location
UK

JSmith

Addicted to Fun and Learning
Joined
Feb 8, 2021
Messages
851
Likes
1,316
Location
Algol Perseus
I'm really not sure what you're trying to say
Apologies mate I took your post out of context, however I'm just trying to show this has been done before.

I grasp what you mean about pre-judgement, however no opinions will be present in the resultant data, assuming the test is well controlled and valid.



JSmith
 
Joined
Apr 19, 2021
Messages
48
Likes
77
Location
UK
Apologies mate I took your post out of context, however I'm just trying to show this has been done before.

I grasp what you mean about pre-judgement, however no opinions will be present in the resultant data, assuming the test is well controlled and valid.



JSmith
No worries, easy to do online. I agree, I'm not sure how rigorous/controlled the previous two you linked were, but either way adding to the data pool (with hopefully a well controlled test like you say) should be a good thing.
 

KSTR

Major Contributor
Joined
Sep 6, 2018
Messages
1,354
Likes
2,834
Location
Berlin, Germany
Yours is not:)

The value in this is the actual test and whatever level of rigorousness we can achieve. The less doubt in the actual procedure and results, the easier it is to shift the conversation to something more useful. It would be a pity if it didn't happen because we chased him away.
Yes. The only way is a constructive joint effort, not to make minced meat out of anybody involved ;-).
Best IME is to work it all out behind the scenes, with best effort for procedural correctness, and to monitor and document everything, then publish (maybe with the gentlemen bail out option that if no positive result --> test did never happen). No matter the outcome, there will always be very controversial discussion but at least it is delayed to "after-the-fact".

OTOH, setting up the proxy-type of ABX-test with loopback recordings of different DACs (using 2x or 4x sample rate for the recording, to catch the fully analog output with mirrorred frequencies etc) might be more feasible, at least as starting point?
In case this would actually succeed with DACs that are "different enough" then no need to do the in-situ laboratory rat sort of tests.
 

KSTR

Major Contributor
Joined
Sep 6, 2018
Messages
1,354
Likes
2,834
Location
Berlin, Germany
Thinking about this, I could make three set of files, one with the linear difference (frequency/phase response) factored out, the second with the nonlinear distortion etc factored out (measured FR's applied to original directly), and the last with nothing corrected (other than level). I would think that could provide a lot of insight, assuming we get at least some positives.
I have two ADI-2 Pro's, one as the test the other as DUT, now we only need a second DUT that's worse enough ;-)
 
Joined
Nov 18, 2019
Messages
22
Likes
67
As a scientific venture this project is a farce at best (a sociopathic hatchet job at worst).

No scientists worth their salt carry on like this. There is adversarial cut-and-thrust in academic debate. There are strongly held positions in professional arenas outside academia. But no professional scientist anywhere would expect to be taken seriously as social gladiator or popcorn cruncher.
More than this, allowing the extraneous influences painfully evident here into an experiment invalidates it before it has begun.

If you really want to do this, and the idea isn't already pre-contaminated, allow Golden to make his specific assertion (choosing the DACs he professes to be able to discriminate), and have the lab work done by a neutral and competent third party. You seem to have a little money to throw at the problem - why not find, by mutual agreement, a friendly postgraduate research student somewhere.

Meantime put a hold on public debate lest outcomes are considered to have been steered unduly.

Otherwise it's all howling at the moon. Does audiophilia a disservice too - no matter which side of the objective/subjective fence you like to sit(/straddle).
 

artburda

Active Member
Joined
Feb 12, 2019
Messages
152
Likes
190
Location
Switzerland
Blind tests should be done quietly, and the results published and analyzed privately with any and all parties admitting to the publishing of results beforehand. If test subjects claimed they knew the results beforehand and were proved wrong, they would at least have some chance to deal with their dissonance before being publicly stoned.
The whole "MQA is not lossless Tidal Test" thing started out with a multi-plattform campaign by the potential blind testing subject. The opposite of quiet. Why can't the blind test be hyped up? Why shouldn't science be fun and engaging and attracting lots of people that maybe don't have a scientific background? I think it's a good thing to get as much as possible attention and show joe the plumber how he can make evidence-based decisions when it comes to audio. Why not make audio science more tangible and educate people? I'm sure this could be the start of a series with all kinds of scientific tests involving youtubers and there are more than enough audio youtubers with good video production capabilities to really have an impact in the audio community.
 

Hai-Fri. Audio

Member
Forum Donor
Joined
May 28, 2021
Messages
29
Likes
44
Personally, I think that @GoldenOne should be allowed to pick the devices in question because he is the one that claims to be able to distinguish between them. If he passes the “test”, then these particular devices should be measured to ensure that they both measure above the traditional threshold of transparency. If they do, it means that different measurements need to be devised that would account for this.

I guess my general point is that this should be a collaborative rather than confrontational effort. All in the interest of exploration of this area rather than proving someone is wrong. Ideally, both sides would come out of the process agreeing with whatever the outcome is.
He makes explicit claims on many videos, we just need to test one of those.

I don't like the condescension of some commenters here before anything has taken place but I don't see how any of that could cause the tester any harm. If the reviewer is so ready to dish out slander, surely he is able to handle some spectator heckling.
 

bboris77

Senior Member
Joined
Oct 23, 2018
Messages
360
Likes
757
He makes explicit claims on many videos, we just need to test one of those.
That is a good idea. I would personally love to have him take a blind test between the Bifrost 2 and any other DS Schiit/Topping/JDS Labs DAC. I know that this is getting into the arguably non-transparent area, but that is exactly my point. We should not start with something super difficult to distinguish between.

Another selfish reason is the fact that when I did a level-matched but sighted test between the Bifrost 2 and the cheapo delta-sigma DAC that is built into my Steinberg UR12, I could not reliably pick them apart.
 

jensgk

Active Member
Forum Donor
Joined
Mar 21, 2020
Messages
125
Likes
355
Location
Denmark
Agree with taking out the negative attitude out of it. It’s not like @GoldenOne said he’d be bending spoons or anything. It is entirely plausible that, under the right circumstances, a trained listener can successfully distinguish between specific amps/DACs.
And it is entirely possible he was able to hear a difference between the devices he claimed. Since they are not available anymore we will never know.

The witch hunting in this thread is no different than bullying. You don't stress a test subject like this. If he fails any test here, you will never know if it is because he is nervous and/or under stress.

But since Amir is so keen on tests, I think it would be much more fair to test Amir:

You should all:
  1. Go through all Amir's writing and videos on this site.
  2. Find out what claims Amir has made regarding his knowledge and hearing.
  3. Design some tests to see if we can know if he is able to do what he claims.
  4. Test him on video.
  5. Publish the result.
I have (off course) not done 1.-5. above, so I don't know what claims he has made, so this is just some suggestions on what kind of tests, I would consider:
  • See if he can hear the difference between 320kbs ogg-vobis and CD
  • See if he can pass (without preparation) some written MIT examinations on electronics, test theory, maths, acoustics, etc.
  • See if he can hear the difference on some of his test tracks, if we EQ them a bit, within the claims he makes within his speaker and headphone tests.
So @amirm are you up to it?
 

bboris77

Senior Member
Joined
Oct 23, 2018
Messages
360
Likes
757
And it is entirely possible he was able to hear a difference between the devices he claimed. Since they are not available anymore we will never know.

The witch hunting in this thread is no different than bullying. You don't stress a test subject like this. If he fails any test here, you will never know if it is because he is nervous and/or under stress.

But since Amir is so keen on tests, I think it would be much more fair to test Amir:

You should all:
  1. Go through all Amir's writing and videos on this site.
  2. Find out what claims Amir has made regarding his knowledge and hearing.
  3. Design some tests to see if we can know if he is able to do what he claims.
  4. Test him on video.
  5. Publish the result.
I have (off course) not done 1.-5. above, so I don't know what claims he has made, so this is just some suggestions on what kind of tests, I would consider:
  • See if he can hear the difference between 320kbs ogg-vobis and CD
  • See if he can pass (without preparation) some written MIT examinations on electronics, test theory, maths, acoustics, etc.
  • See if he can hear the difference on some of his test tracks, if we EQ them a bit, within the claims he makes within his speaker and headphone tests.
So @amirm are you up to it?
See, this is confrontational. I get your points about putting @GoldenOne on the spot and all the pressure, but let's not make this about @amirm again. That other thread was all about that.

I think a lot of us are genuinely tying to get the two of them to collaborate rather than fight and argue. Even if it is only for entertainment purposes, it would be awesome to see whether @GoldenOne can distinguish these devices from one another in a blind test. Heck, he can set this up this test by himself using the tracks that knows well and see if he can pass with no pressure. If he does, then it should be no issue even when someone is observing. It's not he is running a 100-meter race in front of an audience.

I'll give you an analogy - my wife is not colorblind but I am. I cannot pass an Ishihara test under any circumstances. I just do not see the damn numbers. She can pass it whether she is alone or broadcast online in front of a million people. Either you see the numbers or you do not.
 

acbarn

Addicted to Fun and Learning
Forum Donor
Joined
Mar 11, 2020
Messages
983
Likes
2,352
Location
California
In a YouTube review, GO made some very specific claims about the Schiit Magnius, which Amir challenged. This will be an opportunity for GO to show that what he described was not expectation bias and that his subjective reviews are, in fact, legitimate. He has as much to gain from this as his detractors (there’s also the $1K, of course.)
 
Last edited:

Raindog123

Addicted to Fun and Learning
Joined
Oct 23, 2020
Messages
667
Likes
1,152
Location
Melbourne, FL, USA
So @amirm are you up to it?
Are you donating a $1,000 to @amirm 's favorite charity too? :)


I think a lot of us are genuinely tying to get the two of them to collaborate rather than fight and argue.
In all fairness, it's not entirely about 'genuine collaboration', just scroll a-page-or-two back here. And its ok... (The way I see it, there is also some lesson-teaching - to stand by your words/claims. On all sides.) :)
 
Last edited:
Status
Not open for further replies.

Similar threads

Top Bottom