• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Blind test: we have a volunteer!!!

Status
Not open for further replies.

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
OK, I am going to do a full reset on this plan!!! :) The effort was never meant to see if tiny audible differences exist in amps. It is about verifying what he is saying in his video. And there, he goes a million times past this boundary. I am going to show a transcript I took just now of his video and let that sink in and then I will talk about what we need to do to deal with his assertions:
-

Getting a $200 amp for a $600 amp you should probably reevaluate your chain (chuckle).

The dynamic range feels quite compressed…. A bit of caricature of Schiit house sound.

Mentions Only I track from Skinny Living: The loud stuff sounds like what should be but the quiet stuff sounds like it is pushed up to meet the loud stuff – everything sounds like it has roughly the same volume. It takes a lot of delicacy of music.

[…] That doesn’t come across in measurements and I don’t know why. This is not just something I have found; other people have as well.

I don’t get ear fatigue easily….I can listen to Aria on Benchmark AHB2 all day but within a couple of hours of that – not even that – I was feeling tired [from Schiit Magnius], my ears needed a break. I had to take a break. I could not critically listen. This is a fatiguing amp to me.

There is just a slight graininess and lack of separation that really gets to me.

Plays track from Omnitica (EDM) and says bass is good but there is not a huge amount of texture; it doesn’t present low-end timbre very well. In fact timbre overall is not great on this amp; that will be a bit of recurring theme.

In the mids is where stuff starts to get a little wonky… mid texture and detail is not good…

Plays Day by Day by Manganas Garden. The vocals here just lack mid-range energy. They don’t feel like there is any texture or realism to them. [talks about vocal separate heard on AHB2]… but on Magnius is really mediocre. Separation is not a strong point of this.


Treble resolution is odd…. It is really detailed….[plays Enough to Believe from Bob Moses] … there is slight graininess over it all. The timbre is pretty good… there is a graininess that comes across very aggressively on some tracks. Plays Take What You Want by Will Malone…they just feel a bit dry and grainy. They are detailed but they don’t sound correct. Switched to Benchmark AHB2: suddenly there is texture. There is timbre.

The problem I have with this (Schiit Magnius) is that a lot of times it sounds artificial. The treble is so hit and miss.

Low end could be a little faster but is good for an amp this price. Overall I have not really enjoyed the sound of this amp. And I don’t mean it is not as good as Benchmark AHB2… I mean it [Schiit Magnius] is not a good amp at this price. Even when I switch to an amp like JDS Atom… absolutely nothing special (it is a $100 amp), and that sounds a lot better on this track (Take What You Want). It is not as detailed of an amp but it doesn’t have that aggression, it doesn’t have that congestion in the mids. The overall presentation I like better on JDS Atom than [Schiit Magnius]. Both measure excellently so objectively is not too much of a debate but this [Schiit Magnius] feels compressed, forced, the single ended output should not exist, it is just bad. The low-end is soft, unresolving, it is kind of muddy, the low-end output is just not good.

He talks about GoldPoint volume control he is using. It is a passive input selector and volume control (model SA4).

A lot of this I talk about is not Schiit’s fault. Let’s talk about measurements. Measurements can absolutely tell you if something is bad….. All of these measure great but sound different (points to a few amps). Measurements don’t tell the full story and there are a couple of problems to them. First is places like Audio Science Review don’t do consistent testing. If you look at the measurements for this [Magnius] they don’t include intermodulation distortion… that is odd because IMD/intermodulation distortion is much more audible than harmonic distortion. It should be included because by Schiit’s own measurements IMD is higher than THD….

I dislike that Audio Science Review don’t include the same test for each of their products. They [Audio Science Review] frequency miss out things that most people consider important and they only include tests that fit whatever story they want to tell, be it good or bad.

The second part is that Audio Science Review don’t include music. Actual music is not a sine wave (funny there that MQA he tested with such test tones!). There are a lot of topologies that manufactures can implement that measure well but don’t sound good.

A real-world example of this is nested feedback which is not what this [Magnius] is using… but Magni Heresey is using. That’s where you have a lots of op-amps that have feed forward correction (!) and so for a repeating steady state signal, a sine wave, that’s no problem. As soon as you throw music at it with transients, it struggles. A similar thing happens in real life with active noise cancelling headphones…. The same thing can happen with some amp topologies.

Jason from Schiit Audio spoken at a couple of interviews how when Audio Science Review started to trash their low-end… people stopped buying their low-end products, their sales plummeted.

Most people don’t know how to read measurements… Schiit was forced to change how they make products… doesn’t matter if they think it is good…People buy what measures well don’t care how it sounds and that is how we wound up with Schiit Magnius. I don’t like this Amp very much [Magnius]. It is pretty fatiguing, glarry and aggressive at times…but it almost doesn’t matter because people are just going by the measurements even though measurements don’t tell the whole story. And I think that is bad.

People ask how do you know it is not placebo. How do you know you are not just imagining the difference…. The answer is that you can prove it… I have a video coming that is doing just that. I have taken a bunch of DACs and I have done steady state measurements like Audio Science Review as well as null tests with actual music and showing what the differences are there and the results from that has been very interesting…. Says nulls [between DACs] agree with his listening impressions.

If you want something better, get Asgard 3 from Schiit. It is a discrete amp and no feedback trickery as well so sounds a lot better.

--

I don't think you could murder everything about audio science any better than he did in that review! There is a lot to parse here but to the point of testing, I have a good idea on how to proceed.
Well, that was an anti-climax. Too much animus for this to work? Maybe the answer was yes. Choose another word if you like.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,393
Location
Seattle Area
Let's deal with some low hanging fruit in our bloggers video:

[from blogger video transcript]
The second part is that Audio Science Review don’t include music.

Every one of my headphone amps comes with listening tests with music. Here it is for Magnius:

-------
Subjective Listening Tests
Only my very low sensitivity, low impedance Ether CX headphones were ready to go with balanced cables so I tested with that. In low gain, Magnius was able to drive the Ether CX to near deafening levels. The sound was exceptionally clean with authoritative bass. Indeed it is so good I have been enjoying listening to my audiophile playlist as I have been typing this review!

Switching to high gain spanked the Ether CX like nobody's business. I could easily get the drivers to bottom out and start to crackle!!! I only did this for a second or two so please don't try to do it yourself as you risk hearing damage. I am here to do stupid things so you don't need to!
-------

While the write-up is brief, I go through a suite of standardized tracks I have created for headphone amplifiers. I have listened to some 200 headphone amps this way. And I have formal training in this region. This is not some ad-hoc, "let me pay this track and oh, the bass is not so fast." I use these tracks to identify weaknesses in design that are not apparent in measurements such as peak handling. I listen for distortion for example and can easily spot flaws with a number of amps.

So your assertion that ASR reviews are faulty because they don't involve music testing and listening is completely false. Anyone buying a product because of my review will land on subjective testing as well as objective.

Either you did not read my reviews before making those strong assertions or chose to ignore them. Neither is appreciated and points to some kind of animosity toward us/me.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,393
Location
Seattle Area
Well, that was an anti-climax. Too much animus for this to work?
Not remotely so. To be sure, I have zero respect for our blogger. What you saw in there is a drop in the bucket compared to horrible things he has said about me, other companies, etc. I am able and have shown to be able to look past all that and deal with the substance. And the substance here is that a blogger thinks he is reliably hearing things that cannot be explained. If he is right, we need to be schooled. If not, he does.

This type of tension and battle is a huge constant in our lives right now. Online bloggers routine do what he is doing and position them as the holder of truth. Most won't engage with us to test their theories. This blogger has (he has little choice given his objective stance on MQA). So let's move forward and get something done. I have put $1000 forward for heaven's sake and volunteering even more resources.
 

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
Not remotely so. To be sure, I have zero respect for our blogger. What you saw in there is a drop in the bucket compared to horrible things he has said about me, other companies, etc. I am able and have shown to be able to look past all that and deal with the substance. And the substance here is that a blogger thinks he is reliably hearing things that cannot be explained. If he is right, we need to be schooled. If not, he does.

This type of tension and battle is a huge constant in our lives right now. Online bloggers routine do what he is doing and position them as the holder of truth. Most won't engage with us to test their theories. This blogger has (he has little choice given his objective stance on MQA). So let's move forward and get something done. I have put $1000 forward for heaven's sake and volunteering even more resources.
Ok, but I didn't say whose animus. :)

I watch YouTube sparely and reluctantly, it's unfortunate that it has such a gravitational pull. On the bright side, I found your written style a bit zealous initially, but watching you on video gave a more charismatic and thoughtful voice to your words that mitigated that impression. I also appreciate the producers role is more taxing/challenging than that of the spectator.
 
Last edited:

bboris77

Senior Member
Joined
Oct 23, 2018
Messages
444
Likes
931
What happened to the $1000 offer in this revised scenario?

Why are the goalposts being moved now?

It seems that this thread is now turning into a forensic analysis and criticism of @GoldenOne's reviews and digs at ASR. @amirm, I get that you are annoyed by all that has transpired before, but if you have doubts about the integrity of the volunteer, you should have not agreed to this test in the first place. Of course, that does not mean that you are not allowed to withdraw the offer upon further analysis of his allegations - that is ok. I am just wondering if that is what is actually going on here. It would be nice to have some clarity about why this change in tone occurred.

As for the revised test that you are proposing, how would you capture the sound, and would that not add another set of complexities and variables to the situation? Not only are you adding another A to D conversion when you are capturing the signal but then there is the D to A conversion at the participant's end (their DAC) plus the amplifier that they are using to listen to the signal. It is definitely not as pure a test as a simple switcher between two amps would be, as originally proposed.
 

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
What other method would you propose? Obviously sighted is out of the picture, since it introducing many more issues.
Not what magicscreen is getting at, maybe, but ABX may not be ideal for non-categorical discrimination (ie what we want for audio) according to some of the literature. Per Pisoni and Lazarus, sequential or 4IAX gave better results in this study
When listeners heard adjacent pairs of these stimuli in an ABX discrimination test, they were able to discriminate between stimuli drawn from different phonetic categories, but could not discriminate between stimuli drawn from within the same phonetic category, even though the acoustic differences were comparable in the two cases.

Categorical perception is an unusual result to obtain in psychophysical experiments with nonspeech sounds (Liberman, 1970). Stimuli that vary along physical continua, such as frequency or intensity, ordinarily are perceived in a continuous mode; that is, the discrimination functions are usually monotonic with the physical scale, and the listeners can discriminate many more stimuli than they can reliably identify in absolute terms (Pollack, 1952, 1953).
and in the conclusion
... We suggest that the 4IAX discrimination procedure employed in the present study provides listeners with access to auditory information which is obscured by the traditional ABX procedure. Differences in discrimination may be due to the relatively greater demands placed on short-term memory in the ABX procedure which requires that Ss respond to the absolute rather than comparative differences between stimuli. Thus, listeners may be forced to rely on a phonetic rather than auditory coding in order to respond in the ABX discrimination test.
Speech has more research material in this area than audio, so filtering for relevant new research is required. We are interested in continuous, non-categorical and qualitative differences that may be subtle (if they exist at all). Given that we have so far seen few positive results via ABX—and assuming genuine curiosity—I would pursue other blind test methods to rule out over-representation of type 2 errors (false negatives) via the test procedure.
 
Last edited:

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
Screw that. Doritos party mix and Heineken all the way for me. If you’re gonna pig out, do it in style lol.
Haha, not an option for me sad to say. But you used "Heineken" and "style" in the same sentence, which I can't reasonably parse. That said, the beer thread on this forum is epic.
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,827
Fuller transcript from the video in the first post for those wondering.

From the description:
1622755928007.png

Starting from 16:36.
@GoldenOne said:
So people are buying stuff when it measures well. Jason from Schiit Audio has spoken in a couple interviews about how when Audio Science Review started effectively trashing their lower end products—it didn't affect their high end so much, but their lower end products particularly—people stopped buying them. Their sales plummeted because people would Google for a review of a Magni or whatever and they would get an Audio Science Review review at the top. It would say, “Look it doesn't measure as well. Something like a Topping E30 is better you should go buy that.” It's the same price and so everyone did that because most people don't know how to read measurements. Most people don't actually understand what they're telling you and they don't know the context behind them and so when someone who is just looking to buy the best product for their money they find someone who's providing them seemingly pretty solid evidence that this is a bad product and so they don't buy it.

As a result Schiit was forced to change how they make products. They were forced to make heretical editions of their products. Doesn't matter if they think it's good. At first you know they made the Magni and the Magni Heresy. They kept both. One which measured really well and one which they thought was designed to sound the best but I'm guessing given their internal sales numbers (which obviously only they know this) the Magni 3+ probably didn't sell very well. And so when it came to the Magnius they didn't even bother with the discrete version anymore because they knew it wouldn't sell, it wouldn't be worthwhile, why do that when, just screw it, if people want something that measures well, doesn't matter how it sounds, then that's what they'll get. And that's how we've ended up with the Magnius.

I don't like this amp very much. I don't think it sounds great. I think it's okay. It's pretty fatiguing, it's a bit glary and aggressive at times. But it almost doesn't matter because people are going to just go by the measurements, even though the measurements don't tell the whole story. And I think that's bad because I don't think that you can judge a product based on measurements alone.

You can absolutely tell if something's bad: if something measures awfully it's probably going to be awful, but just because something measures really well it does not mean it's going to be a good product. And you might ask: “Well, how do you know it's not just placebo? How do you know you're not just imagining the difference between the Holo May and the E30?”

And the answer is, well, you can prove it, and I've got a video coming which is doing just that. I've taken a bunch of different DACs and I've done steady state measurements like in Audio Science Review as well as null tests with actual music and showing what differences are. The results from that have been quite interesting so subscribe if you'd like to see that because that explains a little bit about how measurements can be misleading. … So the fact that the nulls lined up a lot more with subjective impressions with actual music was very interesting.

Edit: Removed extra line break.
 
Last edited:

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
Fuller transcript from the video in the first post for those wondering.
Starting from 16:36.
Just the last bit:
And the answer is, well, you can prove it, and I've got a video coming which is doing just that. I've taken a bunch of different DACs and I've done steady state measurements like in Audio Science Review as well as null tests with actual music and showing what differences are. The results from that have been quite interesting so subscribe if you'd like to see that because that explains a little bit about how measurements can be misleading. … So the fact that the nulls lined up a lot more with subjective impressions with actual music was very interesting.
Has the test result video GO refers to here gone up yet?
 

PierreV

Major Contributor
Forum Donor
Joined
Nov 6, 2018
Messages
1,437
Likes
4,686
OK, I am going to do a full reset on this plan!!! :) The effort was never meant to see if tiny audible differences exist in amps. It is about verifying what he is saying in his video. And there, he goes a million times past this boundary. I am going to show a transcript I took just now of his video and let that sink in and then I will talk about what we need to do to deal with his assertions:

So, what has changed between the time a challenge was offered and now? A transcript of the video was made?
It is a bit confusing to read you discussing minor protocol details calmly for a few days and then going into full reset mode.
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,393
Location
Seattle Area
So, what has changed between the time a challenge was offered and now? A transcript of the video was made?
No, there was no transcript that I knew about. I typed all of that today I realized were going down the wrong direction in just detecting difference -- any difference -- instead of what my original beef was as I expressed in the precursor to this thread:

Let me show you what *is* ironic. OP is a subjectivist and puts his sighted unreliable listening way ahead of any measurements. Just check out this review of Schiit Magnius which he did in February of this year. I gave it a glowing review and this is his take on it. The title is, ready for it? "
Schiit Magnius Review - Measurements don't tell the whole story.....


"the dynamic range feels quite compressed...it doesn't come across in measurements"

"... this is a fatiguing amp to me..."

"there is slight graininess and lack of separation...."

"there is not a lot of texture [in bass]..."

"timbre is not great in this amp..."

"mid texture and detail is not good..."

And this is in the first 7 minutes of this video!

It ends with some nonsense about how Schiit was "forced" into designing Magnius to please people who look for good measurements. And that this can't be their fault, i.e. it is our fault!

Now he says measurements matter with MQA? If measurements are not important, why didn't he do a listening test of MQA instead? Let's see if it has or has not any bass texture and speed. What it is timbre doing. Does it have texture? Or does it not? We need to know!

My suggestion: be very careful who you are holding up as your expert witness. And making snide remarks. When the truth is not on your side, it can backfire big time!

As you see, I had the rough outline of the subjective things said in this test at that time. They were what got me to put my $1000 challenge.

What got me to reset was the practical challenge in conducting this remote test without someone like me to watch over it. That we have a more accurate path forward which is to do the capture and let everyone examine what these devices are producing.
 

restorer-john

Grand Contributor
Joined
Mar 1, 2018
Messages
12,580
Likes
38,281
Location
Gold Coast, Queensland, Australia
Not only are you adding another A to D conversion when you are capturing the signal but then there is the D to A conversion at the participant's end (their DAC) plus the amplifier that they are using to listen to the signal. It is definitely not as pure a test as a simple switcher between two amps would be, as originally proposed.

In short, valid amplifier comparisons cannot be done unless the participants are in the room with the amplifier, the source and the speakers, with the ability to instantly switch between the level matched amplifiers. Nothing else is useful. Electrical captures or acoustic captures are interesting, but not evidence for audible differences (or lack thereof) one way or another. It's not rocket science.
 

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
In short, valid amplifier comparisons cannot be done unless the participants are in the room with the amplifier, the source and the speakers, with the ability to instantly switch between the level matched amplifiers. Nothing else is useful. It's not rocket science.
Good point (both posts). It makes sense if you believe ADC/DAC is a solved problem, and that all competent gear sounds the same (oversimplifying for brevity) but doesn't really address the claims in GO's video (which I'm watching now, ahhh that winning accent).
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,827
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,393
Location
Seattle Area
In short, valid amplifier comparisons cannot be done unless the participants are in the room with the amplifier, the source and the speakers, with the ability to instantly switch between the level matched amplifiers. Nothing else is useful. Electrical captures or acoustic captures are interesting, but not evidence for audible differences (or lack thereof) one way or another. It's not rocket science.
We are not talking about subtle differences here. Our blogger thinks there are large degradations with Schiit Magnius. If they are so small as to fade away in ADC cycle, then he would need to revisit his position.
 

Blaspheme

Senior Member
Joined
Apr 14, 2021
Messages
461
Likes
515
We are not talking about subtle differences here. Our blogger thinks there are large degradations with Schiit Magnius. If they are so small as to fade away in ADC cycle, then he would need to revisit his position.
Engineer's problem with language. What's the context? General listening: the differences are subtle, casual listeners probably won't notice, careful/critical listeners may do so (or imagine same*). Audiophile commentary: omg huge. If we are listening on different amp and headphones? Uncontrolled variables.

*hence the test idea
 
OP
amirm

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,393
Location
Seattle Area
From GoldenOnes's video transcript:

First is places like Audio Science Review don’t do consistent testing. If you look at the measurements for this [Magnius] they don’t include intermodulation distortion… that is odd because IMD/intermodulation distortion is much more audible than harmonic distortion. It should be included because by Schiit’s own measurements IMD is higher than THD….
I see this common myth often so while we are waiting let me address this.

And amplifier doesn't have one type of non-linearity reserved for "THD" versus "IMD." The same non-linearity gets applied whether you use one tone (THD) versus two (IMD). Given this, you can decompose the IMD test into two components: the first tone's distortions + second tone's distortion.

TI has a nice little tutorial on this. Take an amplifier with this type of non-linearity:

1622773460558.png


The formula at the bottom let's us see the same curve using a power series (which we would then approximate by not using all the terms). If you have to test tones, then we have a second equation:

1622773559012.png


The two terms are just sine waves (with phase information). If you had one tone, then it would just be one sine wave. Having two doesn't change the nature of the equation (1). It simply acts on the signal you feed it (Vin).

The notion from our blogger that "IMD intermodulation is worse" is therefore completely non-sensical. Music has thousands of tones so no two tones tells you what distortion there is.

So to be clear yet again, IMD does NOT tell you more about distortion in the amp. From perceptual point of view, IMD is much harder to analyze since the psychoacoustics of its multiple tones and intermodulation products is much harder to analyze.

What IMD does that THD doesn't do is that it has dual tones with one being at low frequency and the other high (60 Hz and 7 kHz). So to the extent the amp has some kind of frequency dependency, it could generate different results with dual IMD tones than a single 1 kHz.

So a lot of introduction but why don't I run IMD test for amplifiers? Because the story gets told without it. If the engineering is poor, it will absolutely show in the measurement set I show. And if it is great, that is seen as well especially when we see the full spectrum of 1 kHz FFT and not just a "dumb" THD or IMD number.

Running more objective tests doesn't give you more data at some point. It just clutters the review, implying you know more by producing more graphs. In reality it loses the audience.

The set of measurements I run have proven themselves to be complete for all but the weirdest designs. In those cases, additional measurements can be requested and I run them. Otherwise, more is not better. It is just busy work.
 
Status
Not open for further replies.
Top Bottom