Blind test: we have a volunteer!!!

[Jinjuku]

If you were about to do a taste test for Coca-Cola and you had a crowd of people watching and yelling about your potential ridiculed results, would you really want to do the test?

Funny how pro athletes can make claims about their abilities all the time and often time prove those claims in front of global audiences.

 

[McFly]

Funny how pro athletes can make claims about their abilities all the time and often time prove those claims in front of global audiences.

Oh you're a professional blind tester?

 

[hardisj]

Maybe it's just me but this thread as well as the MQA test thread (or how it has been trending the past couple weeks) both just seem like they're the epitome of "petty". Or at least the way it is going down in public in the manner it is.

I don't have a horse in this race. I honestly don't give two shits about the results. I've been on Apple Music for years and that isn't changing. Big whoop about TIDAL or their pseudo-lossless service. And what someone else tells me they hear has never once impacted my financial decisions. Nor has it been reason for me to lose sleep at night.

So... Dude loses the bet... okay ... but he's not going to go to his YT page and tell people to ignore everything he says from now on. Dude wins the bet, great, charity benefits, but the commotion and "told you so" won't get far beyond the walls of ASR. Maybe if the stakes were more legitimate; GS stops doing subjective reviews or ASR members stop saying you can't hear differences between DACs that measure similarly *with the current set of measurements* and then more *legitimate* work must be done to understand the differences heard. I mean, otherwise, what is the expected outcome? Bragging rights? Okay ...

I guess I just felt the need to add my two cents in here because, at least to me, this is the kind of thing that just pushes the whole hobby in to the point of silliness and chest puffing for no real benefit. This is essentially the Ted Denny v Audioholics thing, maybe just in reverse.

Personally, I'd rather just see the charity get $1k and everyone go on their regularly scheduled days. I'll kick in $20 to that cause.

Well, there's my two cents.

 

[Jinjuku]

Oh you're a professional blind tester?

Bottom line: If you say you can from a stand still jump up and clear a 6ft high bar all I gotta do is bring out a tape measure.

Claims are being scrutinized to see if sighted bias effecting outcomes.

 

[MrPeabody]

All the gear he offered for testing is well measuring, well designed stuff. It's not like he's offering to test a magnius vs a 200ohm OI OTL tube amp with a focal clear.

Here, the crucial point has been COMPLETELY MISUNDERSTOOD, and already at least six people have "liked" the comment.

The experiment here is to determine whether he (GO) can truly hear the difference between the Schiit amp that he panned, and another amp that is sonically transparent, i.e., another amp that has no audible sound signature whatsoever. It is utterly pointless for him to demonstrate that he can hear the difference between the Schitt amp and some other amp that, so far as anyone has established with any true certainly, has a sound signature of its own. Goodness gracious, people, how is this point not obvious?

If he has amp that Amir deems to have no audible sound signature, fine. But the comment he made insinuated that he did not quite understand that the experiment in which he is the central player requires, absolutely, that the other amp be an amp that Amir deems to have no audible sound signature.

 

[Chocomel]

Here, the crucial point has been COMPLETELY MISUNDERSTOOD, and already at least six people have "liked" the comment.

The experiment here is to determine whether he (GO) can truly hear the difference between the Schiit amp that he panned, and another amp that is sonically transparent, i.e., another amp that has no audible sound signature whatsoever. It is utterly pointless for him to demonstrate that he can hear the difference between the Schitt amp and some other amp that, so far as anyone has established with any true certainly, has a sound signature of its own. Goodness gracious, people, how is this point not obvious?

If he has amp that Amir deems to have no audible sound signature, fine. But the comment he made insinuated that he did not quite understand that the experiment in which he is the central player requires, absolutely, that the other amp be an amp that Amir deems to have no audible sound signature.

He seems to understand quite well considering the amps/DACs he mentioned that might be suitable for the test.

 

[MrPeabody]

see also Peabody p.8!
https://www.audiosciencereview.com/...nd-test-we-have-a-volunteer.23857/post-804230

There is a very real and very significant difference between what many other people where doing and what I did. Many other people were trying to add all manner of things that weren't in any way prompted by anything that either of the two principles had said. What I wrote was not like those many other posts, because what I wrote was a direct response to something that GO had written, that clearly indicated that he fundamentally misunderstood the nature of the experiment. My reason, for pointing out that he seemed to misunderstand the experiment, wasn't really for the benefit of Amir, and was not for the purpose of insuring that the experiment would be done properly. I had no reason for any concern of that sort, because the point I made is a point that Amir very obviously understands perfectly well.

Rather, my purpose was to alert GO to the fact that he evidently did not correctly understand the point of the experiment. If anyone doubts whether GO had said anything suggesting that he doesn't quite understand the point of the experiment, there is this comment (and two or three that followed it):

If not, I'd probably be OK to use the atom but honestly haven't spent much time with it so would need to check if I can actually hear a difference between it and magnius first. There are definitely some amps I can't hear the difference between so I'm not making any claims of audibility between those two given as I've not tried yet.

Say what? This makes no sense. He is saying that before he agrees to the Atom, he wants to first confirm that he can hear a difference between it and the Schiit. Say what? And he is saying that if he can't hear a difference between them, then all bets are off. Say what? This doesn't make any sense at all and it very clearly indicates that he is confused as to the point of the experiment. If he compares the Schiit to some amp that he knows will not sound the same to him as the Schitt, there is no point whatsoever to this experiment. Amir knows this, and I know this, and I am sure that there are many others who likewise understand this, perfectly well. Based on the above comment by GO, and by two or three comments that followed it, I don't see any way to avoid the conclusion that he fundamentally misunderstands the point of the experiment.

 

[GoldenOne]

You seem to think that all you need to prove, to be shown correct, is that you can hear a difference between the Schiit amp that you criticized and some of the other funky-sounding amps in your collection.

That's not what I'm implying, hence why the discussion of what amps are to be used is going ahead.
I'm quite happy to choose amps that Amir is happy with. At the moment I'm thinking magnius vs SA-1 should be a good comparison. Both of them are on the way to me at the moment.

 

[MrPeabody]

He seems to understand quite well considering the amps/DACs he mentioned that might be suitable for the test.

Might be suitable for the test? They are not the least bit suitable for the test unless Amir says that he deems them to be sonically transparent. I cannot tell whether you understand this fundamental point. Nothing that you've said thus far gives me any reason to think that you understand it.

 

[GoldenOne]

Say what? This makes no sense. He is saying that before he agrees to the Atom, he wants to first confirm that he can hear a difference between it and the Schiit. Say what? And he is saying that if he can't hear a difference between them, then all bets are off. Say what? This doesn't make any sense at all and it very clearly indicates that he is confused as to the point of the experiment. If he compares the Schiit to some amp that he knows will not sound the same to him as the Schitt, there is no point whatsoever to this experiment. Amir knows this, and I know this, and I am sure that there are many others who likewise understand this, perfectly well. Based on the above comment by GO, and by two or three comments that followed it, I don't see any way to avoid the conclusion that he fundamentally misunderstands the point of the experiment.

The point of the experiment as I understand it is to attempt to demonstrate that there is an audible difference between two amps that according to test condition measurements, should sound identical.

What my assumptions/views about audibility are prior to the experiment shouldn't matter if there is indeed no difference. I cannot 'will' a reliably discernible difference into existence where one does not exist, no matter how firmly I believe in it.
But I also never made the claim that I can tell the difference between EVERY amp. That would be silly. And so to agree to a test between the magnius and something that I've not actually tried would be a complete unknown.

In any case we've moved away from the atom for the reasons @solderdude mentioned. Magnius vs SA-1 is looking like a more appropriate test given as they're both balanced, both similar power ratings etc, and also adds another element of interest to the test given as one is discrete and one is opamp based which has been a long-standing subjective debate.

 

[solderdude]

Might be suitable for the test? They are not the least bit suitable for the test unless Amir says that he deems them to be sonically transparent. I cannot tell whether you understand this fundamental point. Nothing that you've said thus far gives me any reason to think that you understand it.

Conclusions
Clearly the Singxer SA-1 is designed using instrumentation to make sure it delivers extremely low levels of noise and distortion. While our testing is stressful enough to show a slight weakness in driving low impedance headphones, the unit is nevertheless worthy of top of the class designation in my book.

I am happy to recommend the Singxer SA-1 headphone amplifier.

You are correct that it's Amirs money on the line. I believe the test done will get the seal of approval from Amir otherwise he won't put up his money.
I get the feeling GO is willing to do this for science and will try his best if not for himself then for the cause Amir might be donating to.

GO will most likely fail the test in that case no money is lost by Amir, Amir got vindicated, and GO can think about proper blind testing in the future when doing reviews or simply states something is his subjective finding in the future.

I'd like to see this test happening for prosperity and so there is something that can be linked to as 'kind of scientific evidence'.
I have done similar tests in the past and failed but maybe, secretly because it would shatter my view, he can ace it and then 'we' have some digging to do. Unlikely... because there is such a thing as nulling.. but a fun test after all. I think Amir's money is safe but when not it is going to charity !!
That cannot possibly be a bad thing.

 

[Raindog123]

Goodness gracious, people, how is this point not obvious?

All I can say is: “Chill, dude!”

 

[DimitryZ]

Maybe it's just me but this thread as well as the MQA test thread (or how it has been trending the past couple weeks) both just seem like they're the epitome of "petty". Or at least the way it is going down in public in the manner it is.

I don't have a horse in this race. I honestly don't give two shits about the results. I've been on Apple Music for years and that isn't changing. Big whoop about TIDAL or their pseudo-lossless service. And what someone else tells me they hear has never once impacted my financial decisions. Nor has it been reason for me to lose sleep at night.

So... Dude loses the bet... okay ... but he's not going to go to his YT page and tell people to ignore everything he says from now on. Dude wins the bet, great, charity benefits, but the commotion and "told you so" won't get far beyond the walls of ASR. Maybe if the stakes were more legitimate; GS stops doing subjective reviews or ASR members stop saying you can't hear differences between DACs that measure similarly *with the current set of measurements* and then more *legitimate* work must be done to understand the differences heard. I mean, otherwise, what is the expected outcome? Bragging rights? Okay ...

I guess I just felt the need to add my two cents in here because, at least to me, this is the kind of thing that just pushes the whole hobby in to the point of silliness and chest puffing for no real benefit. This is essentially the Ted Denny v Audioholics thing, maybe just in reverse.

Personally, I'd rather just see the charity get $1k and everyone go on their regularly scheduled days. I'll kick in $20 to that cause.

Well, there's my two cents.

Make that four.

 

[MrPeabody]

That's not what I'm implying, hence why the discussion of what amps are to be used is going ahead.
I'm quite happy to choose amps that Amir is happy with. At the moment I'm thinking magnius vs SA-1 should be a good comparison. Both of them are on the way to me at the moment.

But you absolutely did write something where you made it all too apparent that you thought it reasonable for you to confirm that you can hear a difference between the Schiit and the Atom before you would agree to it. Please do not deny this, because it is there, very plainly. Here it is in fact:

If not, I'd probably be OK to use the atom but honestly haven't spent much time with it so would need to check if I can actually hear a difference between it and magnius first. There are definitely some amps I can't hear the difference between so I'm not making any claims of audibility between those two given as I've not tried yet.

I hope you realize why this doesn't make sense in that it is entirely in contradiction with the point of the experiment. Just in case you still do not quite understand, I will say it succinctly:

The point of the experiment is to determine whether you can hear the difference between the Schiit amp and another amp that has been deemed, by way of objective measurements that Amir has performed, to have no sonic signature.

If your claim for the Schiit is only that it is flawed relative to your subjective notion of what kind of distortion is desirable for an amp to have, then there is nothing here that is worth pursuing. But if your claim is that the Schiit amp is flawed relative to the sound of an ideal amplifier that hasn't any sonic signature of its own, then and only then is there anything here worth pursuing. If you are not able to demonstrate the ability to hear the difference between the Schiit and an ideal amplifier that hasn't any sonic signature of its own, then your claims regarding the Schiit are without merit, assuming that what you are claiming is that its sound is flawed relative to the sound of an ideal amplifier that hasn't any sonic signature of its own.

At this juncture, I think it is indicated for you to clear up what you are saying exactly, about the sound of the Schiit. Are you simply saying that it does not meet your subjective preferences? Or are you saying that in a controlled test where you listen to the Schiit and to another amp that is deemed free of any sonic signature (as determined by Amir's objective methodology), that you will be able to identify the Schiit by its sonic signature? Will you please clarify?

 

[solderdude]

I think he clarified his standpoint already clearly.

This ball started rolling because in his subjective review he mentioned the Magnius sounded 'poor' (as in different).
He was challenged by Amir to show this in a blind test and he accepted. (the MQA thread triggered all of this)
There was talk that the test would have to be agreed upon by both parties and Amir created this thread to discuss this.
That's what's being done. I don't get the criticism. Let the man try. Be constructive about the test and rely on Amir to set the test parameters/protocol.
I am quite certain when the test is not agreeable to Amir it won't happen.

 

[Chocomel]

Might be suitable for the test? They are not the least bit suitable for the test unless Amir says that he deems them to be sonically transparent. I cannot tell whether you understand this fundamental point. Nothing that you've said thus far gives me any reason to think that you understand it.

The fact i use "might" instead of "are" should tell you.

 

[MrPeabody]

The point of the experiment as I understand it is to attempt to demonstrate that there is an audible difference between two amps that according to test condition measurements, should sound identical.

That's close to my understanding of the point of the experiment. I would however say, "demonstrate whether GoldenOne is or is not able to tell the difference between the Schiit amp and another amp that is likewise deemed (by Amir's objective measurements) to be sonically transparent."

... But I also never made the claim that I can tell the difference between EVERY amp. That would be silly. And so to agree to a test between the magnius and something that I've not actually tried would be a complete unknown.

I have found no reason for you to think it pertinent to point out that you haven't claimed the ability to distinguish between any two amps presented to you. To the point, your second sentence immediately above is logically inconsistent with your fairly-worded statement of the point of the experiment. On the one hand, you understand that the point of the experiment is to learn whether you can hear the difference between two amps that are deemed to sound identical. On the other hand, you aren't amenable to the experiment unless you have had the opportunity to verify that you can hear a difference in the sound of the two amps. If you have not had this opportunity, I don't follow why you think this would be a reason to think that the experiment wouldn't be valid. I'm not following this at all. If you want to compare the Schiit to the other amp that is proposed to you, prior to the experiment, I don't see any problem at all with this. But, if you conclude that you cannot hear any difference, this wouldn't be a reason for you to think that the experiment isn't valid. Rather, the proper thing for you to do in this eventuality would be to withdraw in a sportsmanlike manner, explaining that you conducted the experiment in private and were not able to hear any difference between the two amplifiers.

... Magnius vs SA-1 ... adds another element of interest to the test given as one is discrete and one is opamp based which has been a long-standing subjective debate.

It is very good that the design and construction of the other amp is dissimilar to the Schiit, since this will preclude any protests to the effect that the other amp was only a clone of the Schiit. And indeed this does add another element of interest, especially if it turns out that you cannot reliably tell them apart. In this eventuality, I will not be surprised if more than one person leaps to the conclusion that the reason they sounded different is that one of them is discrete and the other not. I will not be surprised if one or more people assert that because of this difference it was already known that they would sound different and that the experiment was a waste of time.

 

[amirm]

OK, I am going to do a full reset on this plan!!! The effort was never meant to see if tiny audible differences exist in amps. It is about verifying what he is saying in his video. And there, he goes a million times past this boundary. I am going to show a transcript I took just now of his video and let that sink in and then I will talk about what we need to do to deal with his assertions:
-

Getting a $200 amp for a $600 amp you should probably reevaluate your chain (chuckle).

The dynamic range feels quite compressed…. A bit of caricature of Schiit house sound.

Mentions Only I track from Skinny Living: The loud stuff sounds like what should be but the quiet stuff sounds like it is pushed up to meet the loud stuff – everything sounds like it has roughly the same volume. It takes a lot of delicacy of music.

[…] That doesn’t come across in measurements and I don’t know why. This is not just something I have found; other people have as well.

I don’t get ear fatigue easily….I can listen to Aria on Benchmark AHB2 all day but within a couple of hours of that – not even that – I was feeling tired [from Schiit Magnius], my ears needed a break. I had to take a break. I could not critically listen. This is a fatiguing amp to me.

There is just a slight graininess and lack of separation that really gets to me.

Plays track from Omnitica (EDM) and says bass is good but there is not a huge amount of texture; it doesn’t present low-end timbre very well. In fact timbre overall is not great on this amp; that will be a bit of recurring theme.

In the mids is where stuff starts to get a little wonky… mid texture and detail is not good…

Plays Day by Day by Manganas Garden. The vocals here just lack mid-range energy. They don’t feel like there is any texture or realism to them. [talks about vocal separate heard on AHB2]… but on Magnius is really mediocre. Separation is not a strong point of this.


Treble resolution is odd…. It is really detailed….[plays Enough to Believe from Bob Moses] … there is slight graininess over it all. The timbre is pretty good… there is a graininess that comes across very aggressively on some tracks. Plays Take What You Want by Will Malone…they just feel a bit dry and grainy. They are detailed but they don’t sound correct. Switched to Benchmark AHB2: suddenly there is texture. There is timbre.

The problem I have with this (Schiit Magnius) is that a lot of times it sounds artificial. The treble is so hit and miss.

Low end could be a little faster but is good for an amp this price. Overall I have not really enjoyed the sound of this amp. And I don’t mean it is not as good as Benchmark AHB2… I mean it [Schiit Magnius] is not a good amp at this price. Even when I switch to an amp like JDS Atom… absolutely nothing special (it is a $100 amp), and that sounds a lot better on this track (Take What You Want). It is not as detailed of an amp but it doesn’t have that aggression, it doesn’t have that congestion in the mids. The overall presentation I like better on JDS Atom than [Schiit Magnius]. Both measure excellently so objectively is not too much of a debate but this [Schiit Magnius] feels compressed, forced, the single ended output should not exist, it is just bad. The low-end is soft, unresolving, it is kind of muddy, the low-end output is just not good.

He talks about GoldPoint volume control he is using. It is a passive input selector and volume control (model SA4).

A lot of this I talk about is not Schiit’s fault. Let’s talk about measurements. Measurements can absolutely tell you if something is bad….. All of these measure great but sound different (points to a few amps). Measurements don’t tell the full story and there are a couple of problems to them. First is places like Audio Science Review don’t do consistent testing. If you look at the measurements for this [Magnius] they don’t include intermodulation distortion… that is odd because IMD/intermodulation distortion is much more audible than harmonic distortion. It should be included because by Schiit’s own measurements IMD is higher than THD….

I dislike that Audio Science Review don’t include the same test for each of their products. They [Audio Science Review] frequency miss out things that most people consider important and they only include tests that fit whatever story they want to tell, be it good or bad.

The second part is that Audio Science Review don’t include music. Actual music is not a sine wave (funny there that MQA he tested with such test tones!). There are a lot of topologies that manufactures can implement that measure well but don’t sound good.

A real-world example of this is nested feedback which is not what this [Magnius] is using… but Magni Heresey is using. That’s where you have a lots of op-amps that have feed forward correction (!) and so for a repeating steady state signal, a sine wave, that’s no problem. As soon as you throw music at it with transients, it struggles. A similar thing happens in real life with active noise cancelling headphones…. The same thing can happen with some amp topologies.

Jason from Schiit Audio spoken at a couple of interviews how when Audio Science Review started to trash their low-end… people stopped buying their low-end products, their sales plummeted.

Most people don’t know how to read measurements… Schiit was forced to change how they make products… doesn’t matter if they think it is good…People buy what measures well don’t care how it sounds and that is how we wound up with Schiit Magnius. I don’t like this Amp very much [Magnius]. It is pretty fatiguing, glarry and aggressive at times…but it almost doesn’t matter because people are just going by the measurements even though measurements don’t tell the whole story. And I think that is bad.

People ask how do you know it is not placebo. How do you know you are not just imagining the difference…. The answer is that you can prove it… I have a video coming that is doing just that. I have taken a bunch of DACs and I have done steady state measurements like Audio Science Review as well as null tests with actual music and showing what the differences are there and the results from that has been very interesting…. Says nulls [between DACs] agree with his listening impressions.

If you want something better, get Asgard 3 from Schiit. It is a discrete amp and no feedback trickery as well so sounds a lot better.

--

I don't think you could murder everything about audio science any better than he did in that review! There is a lot to parse here but to the point of testing, I have a good idea on how to proceed.

 

[amirm]

So, let's take his observations at face value. I have the two amps he compared: JDS Atom and Schiit Magnius. By the way, strange that he now says he is not very familiar with the sound of Atom. He was pretty sure in that video but we digress. I can capture the output of both and we do an online blind test. We do this not only to see if they are the same but whether people can read any of the things our volunteer blogger says he heard. I have collected his music tracks and can even attempt to capture those.

If we cannot hear anything that he is talking about in blind test, then his observations are moot anyway. Of course, he is also welcome to take part in this test.

Good news here is that there is nothing we need him to do. We have his contributions and our job is now to verify what he is saying as being true. What he talks about as I wrote is quite substantial issues so should be very obvious.

For his part, he could do a blind test between Schiit Magnius and JDS Atom. I highly suggest that he remove that passive volume control which presents a variable impedance to each headphone which could change their frequency response. Talk about testing things in a completely different manner than what people use. He is also doing null tests between DACs he says in the video. How about doing that between these two amps and posting the results here?

 

[richard12511]

No, it just means the blind test is a wrong method.
You cannot use your fallible ears+brain to prove that your ears+brain is fallible.

What other method would you propose? Obviously sighted is out of the picture, since it introducing many more issues.