Blind test: we have a volunteer!!!

[jensgk]

Are you donating a $1,000 to @amirm 's favorite charity too?

No, I don't have money for that, but if it is important, I am sure a nice sum could be raised on this site or on other sites

 

[jensgk]

In a YouTube review, GO made some very specific claims about the Schiit Magnius, which Amir challenged. This will be an opportunity for GO to show that what he described was not expectation bias and that his subjective reviews are, in fact, legitimate. He has as much to gain from this as his detractors (there’s also the $1K, of course.)

On this site and on Youtube, Amir made some very specific claims about <<something to be determined>>, which others have challenged. This will be an opportunity for Amir to show that what he described was not expectation bias and that his subjective reviews are, in fact, legitimate. He has as much to gain from this as his detractors

 

[Zensō]

On this site and on Youtube, Amir made some very specific claims about <<something to be determined>>, which others have challenged. This will be an opportunity for Amir to show that what he described was not expectation bias and that his subjective reviews are, in fact, legitimate. He has as much to gain from this as his detractors

Fair enough. GO took up the challenge though, which he could have simply ignored…

 

[jensgk]

Fair enough. GO took up the challenge though, which he could have simply ignored…

It is not quite clear what he agreed to. The test is not even designed yet, so how could he know what he agreed to?

 

[Raindog123]

It is not quite clear what he agreed to. The test is not even designed yet, so how could he know what he agreed to?

MQA Deep Dive - I published music on tidal to test MQA | Page 165 | Audio Science Review (ASR) Forum

(But yes, details do need to be hammered out)

 

[AdamG]

Let’s keep this conversation civil and free from cheap personal shots across the Bow. Please and thank you.

 

[bboris77]

I think the ball is in @GoldenOne's court now.

 

[magicscreen]

Successful how?

He was able to make a difference between those 2 DACs.

I can do a successful blind test between 2 DACs anytime as well.
If only I had the time and the patience.
I have almost zero ability to concentrate.

 

[solderdude]

He was able to make a difference between those 2 DACs

You don't mean THIS test ?

where this DAC was compared with >3rd harmonics at -70dB (audible range)

 

[GoldenOne]

??? You can use a simple AB headphone switcher. Feed both amps the same signal using an RCA splitter, match levels and you will be there.

I do not have one at this moment but am going to have one made to do this/further tests.
Friend of mine made one recently with >150dB crosstalk separation so that should be suitable

playback signal should be mono, not stereo

I disagree with this. We don't listen to music in mono, I'm not used to it, and it removes much of the elements of stuff like spatial presentation which in my opinion are often some of the more apparent changes. Playing mono wouldn't be representative of real-world use.

When there are no differences, will we again hear that cables (audio and power), switches, relays and splitters effected "resolution" and caused it? And it doesn't make any difference if you measure that they have no effect, you will still hear the same argument.

I never implied that cables make any difference. Beyond having decent shielding I don't think they do. And you're right, there is no measurable difference other than in some super edge cases where one is outright problematic.
But there ARE measurable differences in DACs, where the opinion differs is how audible they are. I'm not claiming there's any fairy dust going on. Everything is measurable and I've never suggested otherwise.

While I use Roon multi-zone for that purpose, I have not studied it with rigor to know that it has no "tells." One DAC may for example be a bit ahead and behind for example.

That's why I like to focus on headphone amp because he has really gone on the limb with Magnius and said all kinds of things that are wrong with it. So it should be walk in the park for him to pass that test blind.

There are definitely aspects that make the tests a bit easier or harder. DACs a basic test is easier to set up given as you can just connect them to a preamp or switch and away you go. But as you mentioned the sync issue could be an issue. In my experience the majority of the time unless the DAC itself has some extensive internal DSP (denafrips dacs seem to have a notable delay) the sync is great, but it's just a case of how perfect is deemed acceptable. I could try connecting L channel of one dac and R channel of other dac to the ADC, play an impulse response test through both to get an idea of how closely synced they are if that might help.
The other issue being volume control.
But I'd agree on doing the amp test first.

Would be nice. I wonder why he wouldn't if he is so confident of his abilities.

This isn't me being confident in MY abilities. I don't believe I have some golden-ear hearing. Honestly I just think that some of this stuff is more audible than many assume. I don't think this is anything to do with me specifically.
Humans can swim. But if you put someone in a pool and they are absolutely 100% convinced that they cannot swim (even if they can) then you can't force them to float

Shouldn't the test also be made sure to be done with XLR outputs? Since IIRC the Magnius had much worse performance on the 1/4" out

Balanced out on both amps whatever they end up being will be used to keep things equal.

Another variable that cannot be forgotten in this type of blind test are the headphones being used as well as the exact output volume/power level. Anyone doing the post-mortem analysis must use the exact same setup.

I'd like to use susvara if @amirm is ok with it.

When comparing amps with different gain (or when switching between gains) one has to carefully 'match' the levels. After that I don't think one can reliable tell amps apart. Subjective reviewers are highly unlikely to do this within 0.1dB and that's where the 'reports' come from. Incorrectly performed tests.

I think the best way to control volume would likely be to use the same preamplifier. Volume match the amps themselves and then control volume there on out from the preamp.
At the moment I'm using the Holo Serene (measurements here: https://www.l7audiolab.com/f/measurements-of-holoaudio-serene-preamplifierpre-retail/ but can borrow my old goldpoint or a khozmo attenuator from a friend.

the amount of traffic this drives to his YT channel will be worth the embarrassment.

As I've mentioned before, I'm not doing this for 'attention' and the amount of traffic ASR has sent to the channel is quite frankly negligible. More viewers came from Gmail.
(See attached image)

Take that $1000 donation and buy GoldenOne a plane ticket to ASR HQ to conduct the test.

Aside from the fact that I'd rather not be travelling internationally right now, I'd much rather the money go to charity.
(Btw the charity I'm nominating will be https://www.miracle-ear.com/miracle-ear-foundation , I figured it's fairly appropriate

Do two level matched, equal duration recordings into an ADC and use foobar's ABX plugin.

This wouldn't be comparative because with an amp you're completely removing the reactive load, which is an inherent part of using speakers/headphones on an amp.
It would be an interesting followup to see how many of these characteristics may/may not carry over when just recorded into an ADC.
But for now, let's keep things simple and equivalent to real-world use.
Headphones, on the amp. Nothing extra.

I am happy to do that. I can even loan him the RME ADC if he doesn't have one

I do have the ADI-2 ADC. And DeltaWave is pretty excellent at handling clock-drift etc. But see the above post. I'd prefer to keep things equivalent to real world use for now. Happy to do this as a secondary followup though

There's nothing scientific about the way this 'blind test' is being hyped up. Sure, you may be using scientific methods to conduct the test, but the way its being prepared is going to discourage unbiased test participants, if any test participants at all.

If you were about to do a taste test for Coca-Cola and you had a crowd of people watching and yelling about your potential ridiculed results, would you really want to do the test?

Blind tests should be done quietly, and the results published and analyzed privately with any and all parties admitting to the publishing of results beforehand. If test subjects claimed they knew the results beforehand and were proved wrong, they would at least have some chance to deal with their dissonance before being publicly stoned.

What I'm saying is noone's going to step up to a blind test where there's a chance they will be publicly murdered by words post test whether right or wrong. Amir has enough clout now that he could conduct anonymous blind tests and damn near everyone would believe the results.

This. I've not responded to any of the "information-free" posts on this thread basically just making fun of the whole prospect. I'm still going to do it because $1000 to charity is a good thing. But It's hardly encouraging and hardly scientific when people seem not to be interested in what might happen, and learning from it one way or another, but instead more interested in laughing at someone.
Honestly the approach of many here is childish.

That it is. I haven't seen the 'rules' and methods laid out yet.
Nor has @GoldenOne participated in this test.

I don't think the test is going to go ahead and if it does what the agreed upon testing looks like and is monitored.

I'm here
Yesterday I was completing my final university exam so wasn't about.

As a scientific venture this project is a farce at best (a sociopathic hatchet job at worst).

No scientists worth their salt carry on like this. There is adversarial cut-and-thrust in academic debate. There are strongly held positions in professional arenas outside academia. But no professional scientist anywhere would expect to be taken seriously as social gladiator or popcorn cruncher.
More than this, allowing the extraneous influences painfully evident here into an experiment invalidates it before it has begun.

If you really want to do this, and the idea isn't already pre-contaminated, allow Golden to make his specific assertion (choosing the DACs he professes to be able to discriminate), and have the lab work done by a neutral and competent third party. You seem to have a little money to throw at the problem - why not find, by mutual agreement, a friendly postgraduate research student somewhere.

Meantime put a hold on public debate lest outcomes are considered to have been steered unduly.

Otherwise it's all howling at the moon. Does audiophilia a disservice too - no matter which side of the objective/subjective fence you like to sit(/straddle).

+1

Seriously this is supposed to be scientific. It's supposed to be interesting and the intent is to learn something new. The world doesn't advance by pointing and laughing at people. It advances by people trying stuff and those things sometimes having unexpected results, after which we have a deeper look at what might be going on and re-evaluate assumptions that new evidence contradicts.


In any case, for now I'll get an amp switcher sorted and then we can go from there and discuss further the conditions of the test (though as said I'm still happy to do May vs E30. Measurements for both here:
https://www.superbestaudiofriends.org/index.php?threads/kte-may-technical-measurements.8933/
https://www.audiosciencereview.com/...may-probably-the-best-discrete-r2r-dac.10161/
https://www.audiosciencereview.com/forum/index.php?threads/topping-e30-dac-review.12119/ )

 

[richard12511]

I do not have one at this moment but am going to have one made to do this/further tests.
Friend of mine made one recently with >150dB crosstalk separation so that should be suitable


I disagree with this. We don't listen to music in mono, I'm not used to it, and it removes much of the elements of stuff like spatial presentation which in my opinion are often some of the more apparent changes. Playing mono wouldn't be representative of real-world use.


I never implied that cables make any difference. Beyond having decent shielding I don't think they do. And you're right, there is no measurable difference other than in some super edge cases where one is outright problematic.
But there ARE measurable differences in DACs, where the opinion differs is how audible they are. I'm not claiming there's any fairy dust going on. Everything is measurable and I've never suggested otherwise.


There are definitely aspects that make the tests a bit easier or harder. DACs a basic test is easier to set up given as you can just connect them to a preamp or switch and away you go. But as you mentioned the sync issue could be an issue. In my experience the majority of the time unless the DAC itself has some extensive internal DSP (denafrips dacs seem to have a notable delay) the sync is great, but it's just a case of how perfect is deemed acceptable. I could try connecting L channel of one dac and R channel of other dac to the ADC, play an impulse response test through both to get an idea of how closely synced they are if that might help.
The other issue being volume control.
But I'd agree on doing the amp test first.



This isn't me being confident in MY abilities. I don't believe I have some golden-ear hearing. Honestly I just think that some of this stuff is more audible than many assume. I don't think this is anything to do with me specifically.
Humans can swim. But if you put someone in a pool and they are absolutely 100% convinced that they cannot swim (even if they can) then you can't force them to float


Balanced out on both amps whatever they end up being will be used to keep things equal.


I'd like to use susvara if @amirm is ok with it.


I think the best way to control volume would likely be to use the same preamplifier. Volume match the amps themselves and then control volume there on out from the preamp.
At the moment I'm using the Holo Serene (measurements here: https://www.l7audiolab.com/f/measurements-of-holoaudio-serene-preamplifierpre-retail/ but can borrow my old goldpoint or a khozmo attenuator from a friend.


As I've mentioned before, I'm not doing this for 'attention' and the amount of traffic ASR has sent to the channel is quite frankly negligible. More viewers came from Gmail.
(See attached image)


Aside from the fact that I'd rather not be travelling internationally right now, I'd much rather the money go to charity.
(Btw the charity I'm nominating will be https://www.miracle-ear.com/miracle-ear-foundation , I figured it's fairly appropriate



This wouldn't be comparative because with an amp you're completely removing the reactive load, which is an inherent part of using speakers/headphones on an amp.
It would be an interesting followup to see how many of these characteristics may/may not carry over when just recorded into an ADC.
But for now, let's keep things simple and equivalent to real-world use.
Headphones, on the amp. Nothing extra.


I do have the ADI-2 ADC. And DeltaWave is pretty excellent at handling clock-drift etc. But see the above post. I'd prefer to keep things equivalent to real world use for now. Happy to do this as a secondary followup though


This. I've not responded to any of the "information-free" posts on this thread basically just making fun of the whole prospect. I'm still going to do it because $1000 to charity is a good thing. But It's hardly encouraging and hardly scientific when people seem not to be interested in what might happen, and learning from it one way or another, but instead more interested in laughing at someone.
Honestly the approach of many here is childish.


I'm here
Yesterday I was completing my final university exam so wasn't about.



+1

Seriously this is supposed to be scientific. It's supposed to be interesting and the intent is to learn something new. The world doesn't advance by pointing and laughing at people. It advances by people trying stuff and those things sometimes having unexpected results, after which we have a deeper look at what might be going on and re-evaluate assumptions that new evidence contradicts.


In any case, for now I'll get an amp switcher sorted and then we can go from there and discuss further the conditions of the test (though as said I'm still happy to do May vs E30. Measurements for both here:
https://www.superbestaudiofriends.org/index.php?threads/kte-may-technical-measurements.8933/
https://www.audiosciencereview.com/...may-probably-the-best-discrete-r2r-dac.10161/
https://www.audiosciencereview.com/forum/index.php?threads/topping-e30-dac-review.12119/ )

I think as long you do the test, pass or fail, the outcome will be positive for audio science. I do think fail is the most likely outcome if the test is setup properly, but I still think this is useful for advancing audio science. If you pass, that's even more interesting and useful imo. Then we can examine what is causing those differences that you hear.

Also, to others in this thread. Just because someone claims to hear differences, that's no reason to make fun of them when they fail to hear those differences under controlled conditions. There's not a human on earth who isn't effected by expectation bias to some degree. If he fails, it just means he's susceptible to placebo, as all humans are. Lets make this a great learning experience, regardless of the outcome.

 

[hardisj]

As I've mentioned before, I'm not doing this for 'attention' and the amount of traffic ASR has sent to the channel is quite frankly negligible. More viewers came from Gmail.
(See attached image)

Well, that graphic puts to bed any notion of you posting here to garner views. 0.30% of your traffic views comes from here. Funny enough, twice as much comes from audiogon. And they even watch the video about 3 minutes longer than viewers here do.

 

[GaryH]

• See if he can hear the difference between 320kbs ogg-vobis and CD

And stipulate the exact test conditions e.g.:
- using already tested DAC/amp + headphone/speaker that are measurably audibly transparent (not just a random laptop)
- running in WASAPI/ASIO mode
- using files that do not have any ultrasonic content which could possibly lead to audible-band distortion via IMD
- at a volume that he would set for comfortable listening throughout the whole track (not turning up the volume to very high levels in quiet sections to artificially accentuate normally inaudible artifacts)
- producing a Foobar200 ABX log with verifiable signature to check here, without that results are invalid

• See if he can hear the difference on some of his test tracks, if we EQ them a bit, within the claims he makes within his speaker and headphone tests

This could possibly be determined from the fact he's said he could reach up to level 5 on the How to Listen software from Harman (trained listeners are required to pass at a very minimum level 8, Dr Sean Olive has reached level 15 on Band ID Peaks). @Sean Olive to what degree of change in frequency response (dB and Q-factor) does level 5 of How to Listen test?

 

[Tks]

Know what kinda sucks? Even if people can or can't blind test DACs, I'm still here unable to blind test 96kbps OPUS vs FLACs (hi-res or not).

Yo bossman, I know you answered me before on how Harman has some sort of listener training program. Or was it some other company? Or was it multiple companies that offer such education. But do you have any details on what one has to do to become: "a trained listener".

For that matter, can anyone else chime in?

 

[LTig]

Know what kinda sucks? Even if people can or can't blind test DACs, I'm still here unable to blind test 96kbps OPUS vs FLACs (hi-res or not).

Yo bossman, I know you answered me before on how Harman has some sort of listener training program. Or was it some other company? Or was it multiple companies that offer such education. But do you have any details on what one has to do to become: "a trained listener".

For that matter, can anyone else chime in?

Start with big differences (e.g. Flac vs 32 kBps mp3) to become familiar with how coding artifacts sound. Then increase the bit rate slowly.

 

[GaryH]

Know what kinda sucks? Even if people can or can't blind test DACs, I'm still here unable to blind test 96kbps OPUS vs FLACs (hi-res or not).

Yo bossman, I know you answered me before on how Harman has some sort of listener training program. Or was it some other company? Or was it multiple companies that offer such education. But do you have any details on what one has to do to become: "a trained listener".

For that matter, can anyone else chime in?

See the last paragraph of my previous post directly above yours. Here's a direct link to download Harman's free How to Listen training software I mentioned:

http://harmanhowtolisten.blogspot.com/2011/01/welcome-to-how-to-listen.html

 

[Tks]

Start with big differences (e.g. Flac vs 32 kBps mp3) to become familiar with how coding artifacts sound. Then increase the bit rate slowly.

Changes per track, and I generally don't know what to listen out for in the higher rates at all.

 

[Blaspheme]

I mean a cursory scroll through this thread shows that this is already impossible. It's a shame how childish and petty people are on this website sometimes, reminds me of a school playground. All the criticism of what 'he/they will say when he gets it wrong' and the guy hasn't even said anything of the sort yet. What about if he gets it right? You think people here won't do the exact same thing and come out with a multitude of different excuses/explanations? Personally it's something I struggle to believe without seeing someone do it myself, we have the opportunity to do that. Can we at least prevent the inevitable shitshow that will ensue, regardless of the results, until the actual results are out? Discussing methodology and how to ensure a fair test is clearly very important and of value. The underlying scorn and sneering, in my opinion, not so much.

Yeah, as a relatively new reader, the cartoonish shite is pretty unappealing. If I liked US-style wresting matches maybe, but no, not really.

 

[Blaspheme]

See the last paragraph of my previous post directly above yours. Here's a direct link to download Harman's free How to Listen training software I mentioned:

http://harmanhowtolisten.blogspot.com/2011/01/welcome-to-how-to-listen.html

I spent some time on this a while back. Training takes practice (no surprise when you think about it). It was pretty cool and informative, although in my case the musical examples were akin to torture.

 

[amirm]

As a scientific venture this project is a farce at best (a sociopathic hatchet job at worst).

No scientists worth their salt carry on like this.

First, we are not trying to make "science." Poster has said that he hears all kinds of impairments in Schiit Magnius despite its superlative measurements. He has further dissuade people from buying said product saying Schiit made a mistake to produce it and that somehow they were forced to do it (for the sake of measurements). So we have a situation where I have give the same product raving review, and he has given the opposite. We need to figure out what is going on here. The easiest explanation -- one that is backed by science -- is that sighted test was performed incorrectly (bias, levels not matched, etc.). If so, a controlled test deals with this effectively. We would be following audio science here, not making one.

As to no one accepting such tests, I have done so on numerous occasions. When challenged that no one could pass a 320 kbps MP3 against the original, I immediately went and ran a double blind ABX test:

---

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/19 19:45:33

File A: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44.wav
File B: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44_01.mp3

19:45:33 : Test started.
19:46:21 : 01/01 50.0%
19:46:35 : 02/02 25.0%
19:46:49 : 02/03 50.0%
19:47:03 : 03/04 31.3%
19:47:13 : 04/05 18.8%
19:47:27 : 05/06 10.9%
19:47:38 : 06/07 6.3%
19:47:46 : 07/08 3.5%
19:48:01 : 08/09 2.0%
19:48:19 : 09/10 1.1%
19:48:31 : 10/11 0.6%
19:48:45 : 11/12 0.3%
19:48:58 : 12/13 0.2%
19:49:11 : 13/14 0.1%
19:49:28 : 14/15 0.0%
19:49:52 : 15/16 0.0%
19:49:56 : Test finished.

----------
Total: 15/16 (0.0%)

---

And the clip I used was the clip that was provided by the person challenging me! Discussion ended there. I have done this on numerous occasions and it is super helpful in cutting through the arguments.

By the same token, maybe our volunteer can pass the test as well as I did above then it adds a lot to our discussions.

If you really want to do this, and the idea isn't already pre-contaminated, allow Golden to make his specific assertion (choosing the DACs he professes to be able to discriminate), and have the lab work done by a neutral and competent third party. You seem to have a little money to throw at the problem - why not find, by mutual agreement, a friendly postgraduate research student somewhere.

Huh? We are dealing with a specific video and review that I post. The DAC test is proposed as an alternative but I have not accepted it. We need to verify what the video says in clear words for full 20 minutes. Not interested in testing other hypothesis with DACs and such.

This is why the suggest to record the outputs and have the tester listen to those is appealing. He is welcome to say what is wrong with that protocol but a protest like yours is not going to work. I have taken my double blind tests under the most hostile arguments that would make us look like church goers in contrast!