Blind test: we have a volunteer!!!

[amirm]

So I want to go back to this line in the video:

"And you might ask: “Well, how do you know it's not just placebo? How do you know you're not just imagining the difference between the Holo May and the E30?” And the answer is, well, you can prove it, and I've got a video coming which is doing just that. I've taken a bunch of different DACs and I've done steady state measurements like in Audio Science Review as well as null tests with actual music and showing what differences are. "

Looks like our blogger is proposing his own objective proof which involves capturing the output and performing null tests. So I suggest he goes through with this and provides us such nulling experiment with the two amps as the starting test. I wouldn't mind seeing his "steady state measurements" either.

 

[Slayer]

So I want to go back to this line in the video:

"And you might ask: “Well, how do you know it's not just placebo? How do you know you're not just imagining the difference between the Holo May and the E30?” And the answer is, well, you can prove it, and I've got a video coming which is doing just that. I've taken a bunch of different DACs and I've done steady state measurements like in Audio Science Review as well as null tests with actual music and showing what differences are. "

Looks like our blogger is proposing his own objective proof which involves capturing the output and performing null tests. So I suggest he goes through with this and provides us such nulling experiment with the two amps as the starting test. I wouldn't mind seeing his "steady state measurements" either.

Why, why must this back and fourth continue ? If you and Goldenone have agreed on a test and the parameters for that test. I see no need for the back and fourth of he he said this, he said that etc. It's time everyone let all accusations simply lay or put on pause for now.
Amir, I respect your credentials and all the hard work you do in the name of this hobby. Sometimes maybe you should simply take a step back and realize just because someone may not have the same take (sort of speak) regarding certain audio claims, doesn't mean it's a personal attack against you.

You have made the challenge and Goldenone has excepted. Can we now let the mudslinging so to speak end for now. Then when the challenge is over, we can debate the outcome whatever it may be. Cheers

 

[Shazb0t]

So I want to go back to this line in the video:

"And you might ask: “Well, how do you know it's not just placebo? How do you know you're not just imagining the difference between the Holo May and the E30?” And the answer is, well, you can prove it, and I've got a video coming which is doing just that. I've taken a bunch of different DACs and I've done steady state measurements like in Audio Science Review as well as null tests with actual music and showing what differences are. "

Looks like our blogger is proposing his own objective proof which involves capturing the output and performing null tests. So I suggest he goes through with this and provides us such nulling experiment with the two amps as the starting test. I wouldn't mind seeing his "steady state measurements" either.

Properly done level matched null testing would end his YouTube channel.

 

[amirm]

Why, why must this back and fourth continue ? If you and Goldenone have agreed on a test and the parameters for that test. I see no need for the back and fourth of he he said this, he said that etc.

Huh? There has been no agreement on the test. He proposed some bits of a test, I provided feedback on why it is not good enough (testing DACs instead, using a friend as the proctor, issues with switching, etc.). Our membership proposed a different one (digital capture and software ABX) but our volunteer doesn't want to do that. In the post you quoted, I am bringing up another test that he himself said is the right proof point.

You have made the challenge and Goldenone has excepted.

He hasn't accept the challenge. He is running with a new one that can be used as a warm-up exercise but not the real deal.

 

[Hai-Fri. Audio]

I am "guilty" of being one of these cross-members whose point of view seems to shift from time to time, so I feel I need to provide my view on this subject. For me, it is not about pleasing people because I actually tend to piss people off when my position shifts on any given subject. The reason for these changes in my views is simply evolution in my understanding of the situation and people involved. Sure, sometimes I am guilty of assuming too much about people (mainly Amir) based on my instinctive perception of a situation or their own actions, but I am fully willing to admit my mistakes when I acquire more information about the issue or a person.

There is nothing that prevents a person from changing their mind on a subject when presented with more evidence. You have to put people's comments in a historical context of what was known about a subject matter at a given point in time. I refuse to be in any inflexible camp (i.e. objectivist/subjectivist) which tends to put a set of blinders on and bulldozes over anyone who dares to oppose their worldview. Personally, as the time goes on, I tend to be more on the objectivist side, without ruling out a possibility of further advances in how we measure audio reproduction.

Don't feel guilty at all, mate. I've followed the thread on and off and from what I can tell you've been very level headed and up front with your queries. Code switching, or behavioral/cultural shifts in order to better communicate/navigate within different communities is a good skill to have. Putting that effort into your communication helps everyone involved.

It's fascinating on a human level. This site and that one (and others) seem to have several members in common. They tend to modulate how they present themselves depending on where they are. Maybe a modern iteration of "people pleasing" ... could call it "forum pleasing".

When one utilizes that skill to achieve malicious ends - e.g. presenting an image of gentlemanly conduct in one forum while spreading unsubstantiated slander in another - it becomes something else entirely.

There is a chasm of nuances between code-switching and being two-faced.

 

[Blaspheme]

You setup a blind test and switch speakers and ask the tester to give an overall score and one each for different aspects, bass, mid-range treble. Here is an example from Sean Olive test of different EQ systems:

View attachment 133807

As you see, listeners rated different frequency bands and their ratings were then shown as a mean and error bar (distribution). Our blogger split the performance of the amplifier into bass, mid-range and treble as well so fits this methodology. So he would simply repeat his testing except this time it would be blind. We would then have others in the room take the same test and provide similar scores. A statistical analysis then shows whether the results are significant, what the distribution is, etc.

Ordinarily we don't do this for electronics because the differences about timbre, etc. are ruled out. But our blogger has ruled them in so if we follow his lead, then this is the type of data that needs to be collected.

Yes, I recollect now, thank you for the extract.

Can we do something like this here? For example, delineate subjective observations used by GO in the video into reasonably clear categories he can score while listening in a blinded experiment. Use the Stereophile glossary for reference for terms of the subjective reviewers' art—or just negotiate them—whatever suits. Perhaps select musical passages that are relevant to each criteria, again similar to the video. And so on.

The point being to evaluate more than just 'different' with some kind of rigour. Could be more than two amps compared, but that isn't critical. If GO can consistently assign scores to the different devices used in the experiment while listening, through multiple iterations, that would presumably be an objective indication that the characteristic is being heard. The technical procedures need to be solid, of course, but that's being covered. Does that sort of thing seem adequate and workable?

 

[BrEpBrEpBrEpBrEp]

OK, I am going to do a full reset on this plan!!! The effort was never meant to see if tiny audible differences exist in amps. It is about verifying what he is saying in his video. And there, he goes a million times past this boundary. I am going to show a transcript I took just now of his video and let that sink in and then I will talk about what we need to do to deal with his assertions:
-

Getting a $200 amp for a $600 amp you should probably reevaluate your chain (chuckle).

The dynamic range feels quite compressed…. A bit of caricature of Schiit house sound.

Mentions Only I track from Skinny Living: The loud stuff sounds like what should be but the quiet stuff sounds like it is pushed up to meet the loud stuff – everything sounds like it has roughly the same volume. It takes a lot of delicacy of music.

[…] That doesn’t come across in measurements and I don’t know why. This is not just something I have found; other people have as well.

I don’t get ear fatigue easily….I can listen to Aria on Benchmark AHB2 all day but within a couple of hours of that – not even that – I was feeling tired [from Schiit Magnius], my ears needed a break. I had to take a break. I could not critically listen. This is a fatiguing amp to me.

There is just a slight graininess and lack of separation that really gets to me.

Plays track from Omnitica (EDM) and says bass is good but there is not a huge amount of texture; it doesn’t present low-end timbre very well. In fact timbre overall is not great on this amp; that will be a bit of recurring theme.

In the mids is where stuff starts to get a little wonky… mid texture and detail is not good…

Plays Day by Day by Manganas Garden. The vocals here just lack mid-range energy. They don’t feel like there is any texture or realism to them. [talks about vocal separate heard on AHB2]… but on Magnius is really mediocre. Separation is not a strong point of this.


Treble resolution is odd…. It is really detailed….[plays Enough to Believe from Bob Moses] … there is slight graininess over it all. The timbre is pretty good… there is a graininess that comes across very aggressively on some tracks. Plays Take What You Want by Will Malone…they just feel a bit dry and grainy. They are detailed but they don’t sound correct. Switched to Benchmark AHB2: suddenly there is texture. There is timbre.

The problem I have with this (Schiit Magnius) is that a lot of times it sounds artificial. The treble is so hit and miss.

Low end could be a little faster but is good for an amp this price. Overall I have not really enjoyed the sound of this amp. And I don’t mean it is not as good as Benchmark AHB2… I mean it [Schiit Magnius] is not a good amp at this price. Even when I switch to an amp like JDS Atom… absolutely nothing special (it is a $100 amp), and that sounds a lot better on this track (Take What You Want). It is not as detailed of an amp but it doesn’t have that aggression, it doesn’t have that congestion in the mids. The overall presentation I like better on JDS Atom than [Schiit Magnius]. Both measure excellently so objectively is not too much of a debate but this [Schiit Magnius] feels compressed, forced, the single ended output should not exist, it is just bad. The low-end is soft, unresolving, it is kind of muddy, the low-end output is just not good.

He talks about GoldPoint volume control he is using. It is a passive input selector and volume control (model SA4).

A lot of this I talk about is not Schiit’s fault. Let’s talk about measurements. Measurements can absolutely tell you if something is bad….. All of these measure great but sound different (points to a few amps). Measurements don’t tell the full story and there are a couple of problems to them. First is places like Audio Science Review don’t do consistent testing. If you look at the measurements for this [Magnius] they don’t include intermodulation distortion… that is odd because IMD/intermodulation distortion is much more audible than harmonic distortion. It should be included because by Schiit’s own measurements IMD is higher than THD….

I dislike that Audio Science Review don’t include the same test for each of their products. They [Audio Science Review] frequency miss out things that most people consider important and they only include tests that fit whatever story they want to tell, be it good or bad.

The second part is that Audio Science Review don’t include music. Actual music is not a sine wave (funny there that MQA he tested with such test tones!). There are a lot of topologies that manufactures can implement that measure well but don’t sound good.

A real-world example of this is nested feedback which is not what this [Magnius] is using… but Magni Heresey is using. That’s where you have a lots of op-amps that have feed forward correction (!) and so for a repeating steady state signal, a sine wave, that’s no problem. As soon as you throw music at it with transients, it struggles. A similar thing happens in real life with active noise cancelling headphones…. The same thing can happen with some amp topologies.

Jason from Schiit Audio spoken at a couple of interviews how when Audio Science Review started to trash their low-end… people stopped buying their low-end products, their sales plummeted.

Most people don’t know how to read measurements… Schiit was forced to change how they make products… doesn’t matter if they think it is good…People buy what measures well don’t care how it sounds and that is how we wound up with Schiit Magnius. I don’t like this Amp very much [Magnius]. It is pretty fatiguing, glarry and aggressive at times…but it almost doesn’t matter because people are just going by the measurements even though measurements don’t tell the whole story. And I think that is bad.

People ask how do you know it is not placebo. How do you know you are not just imagining the difference…. The answer is that you can prove it… I have a video coming that is doing just that. I have taken a bunch of DACs and I have done steady state measurements like Audio Science Review as well as null tests with actual music and showing what the differences are there and the results from that has been very interesting…. Says nulls [between DACs] agree with his listening impressions.

If you want something better, get Asgard 3 from Schiit. It is a discrete amp and no feedback trickery as well so sounds a lot better.

--

I don't think you could murder everything about audio science any better than he did in that review! There is a lot to parse here but to the point of testing, I have a good idea on how to proceed.

It seems that there are a series of criticisms levelled mostly at the bass, mids, and treble in that transcript. So could we not have @GoldenOne do a blind AB between the Magnius and a reference amp, where he ranks A and B in terms of bass, mids, and treble, then writes down which amp is which? Randomize A and B again, rinse, repeat 10+ times.

This checks whether he can tell them apart at all (does he select consistently which is which) and whether his blind impressions align to the sighted ones (does he consistently rank the Magnius the lower of the two in all 3 categories).

If he is able to get everything correct, others can try the test as well to prove that these impressions are repeatable across people and that there isn't a subtle tell in the switching process, or the test can be set up again with a more sophisticated process/switching device, etc.

Am I missing anything in this procedure?

 

[BrEpBrEpBrEpBrEp]

Of course, since @GoldenOne proposes a null-test for DACs, we could just digitally null recordings of the two amps and save a lot of time that way, no? And, as proposed before, do the blind test digitally as well - which should easily be more than good enough to reveal the flaws in the transcript when using a good ADC.

 

[amirm]

Can we do something like this here? For example, delineate subjective observations used by GO in the video into reasonably clear categories he can score while listening in a blinded experiment. Use the Stereophile glossary for reference for terms of the subjective reviewers' art—or just negotiate them—whatever suits. Perhaps select musical passages that are relevant to each criteria, again similar to the video. And so on.

I am Ok if he uses the same music he used in the review. We can do this type of testing but statistical analysis is much more complex. You have to tease out that the variable that was most significant was the amp, not the content, the order, or the listener, etc. This type of analysis takes a page in a research paper usually.

On categories, I think it is best to go with three categories he had for tonality plus dynamics he had in the video.

 

[amirm]

Of course, since @GoldenOne proposes a null-test for DACs, we could just digitally null recordings of the two amps and save a lot of time that way, no?

Of course. This is what I suggested earlier. He says it is his method of choice. It is ours as well. So I suggest that gets done before any video production.

 

[DimitryZ]

Of course, since @GoldenOne proposes a null-test for DACs, we could just digitally null recordings of the two amps and save a lot of time that way, no? And, as proposed before, do the blind test digitally as well - which should easily be more than good enough to reveal the flaws in the transcript when using a good ADC.

Actually, it seems no.

DACs work into resistive dominated input impedances of preamps.

Amps drive real-world speaker/headphone loads.

And it's far from easy to digitize what the listener is exposed to.

If I understand correctly, @amirm is proposing a "preference" blind test, with three frequency bands, where the @GoldenOne is expected to prefer a reference amp against Schiit amp in each category.

The parties should work this out privately.

 

[BrEpBrEpBrEpBrEp]

DACs work into resistive dominated input impedances of preamps.

Amps drive real-world speaker/headphone loads.

And it's far from easy to digitize what the listener is exposed to.

So why not load the amplifier with headphones in parallel with the ADC?

Edit: and I'm pretty sure this was said earlier in this thread as well.

 

[DimitryZ]

So why not load the amplifier with headphones in parallel with the ADC?

You would be missing the headphone's actual response into an ear and brain/ear interaction - both very important.

Amplifier listening tests definitionally require speakers.

Bummer.

 

[BrEpBrEpBrEpBrEp]

You would be missing the headphone's actual response into an ear and brain/ear interaction.

Which will neatly occur when someone does the AB test digitally using a pair of headphones. We're trying to test the amplifier here, not the headphones.

 

[amirm]

You would be missing the headphone's actual response into an ear and brain/ear interaction - both very important.

You can wear the headphone if you like while you capture. From my testing, this makes a tiny difference in impedance. And at any rate, it is a variable in real life anyway.

 

[DimitryZ]

Which will neatly occur when someone does the AB test digitally using a pair of headphones. We're trying to test the amplifier here, not the headphones.

Nope.

You need the same headphones and the same amp (assuming digitizing process is fully transparent). If you record with one set and listen with another you are introducing two uncontrolled variables - replay amp and replay phones.

If you are looking for small differences in a "band preference test," as @amirm is proposing, adding significant uncontrolled variables is verboten.

Design of experiments is hard.

 

[amirm]

You need the same headphones and the same amp (assuming digitizing process is fully transparent). If you record with one set and listen with another you are introducing two uncontrolled variables - replay amp and replay phones.

Our blogger can capture, listen and say whether they sound different. Again, remember he has said he has done this type of capture with DACs and declared it representative of what he is hearing.

 

[BrEpBrEpBrEpBrEp]

Nope.

You need the same headphones and the same amp (assuming digitizing process is fully transparent). If you record with one set and listen with another you are introducing two uncontrolled variables - replay amp and replay phones.

If you are looking for small differences in a "band preference test," as @amirm is proposing adding significant uncontrolled variables is verboten.

Design of experiments is hard.

... the idea is that these differences, as perceived by @GoldenOne , are significant and general enough to go into a review published on YouTube.

 

[DimitryZ]

You can wear the headphone if you like while you capture. From my testing, this makes a tiny difference in impedance. And at any rate, it is a variable in real life anyway.

I am sorry, but you are testing for experiential judgement -from a single individual, no less.

You can't externalize this test, by definition.

 

[DimitryZ]

... the idea is that these differences, as perceived by @GoldenOne , are significant and general enough to go into a review published on YouTube.

Entirely different question, as described in my previous post, and requiring a statistically significant number of subjects. Well outside the challenge of this thread.

As I pointed out before, this is "do subjective reviews have a right to exist question" which goes to extendability of human perception. We should all start with Thomas Aquinas and proceed from there.