Why Audio Science Review's measurements are so different compared to other sources?

[lashto]

When researching for measurements I have seen multiple websites and sources that provide these. This is one of those websites directed by its creator Amir. He is really knowledgeable guy which his posts have taught me a lot of audio processing and measurement, but something I have noticed is how divergent some of his measurements are compared to other websites and/or the manufacturer itself. I know there could be bias or set-up measurements that do not really represent the actual performance of the gear, but there has been some discussion about some equipment that has been measured by Amir and other sources such as Jude and the manufacturers that show significant differences in the measurements that cannot be explained as simple production variations.

Others have explicitly said that Amir hacks the measurements to be worse because he does not like a manufacturer, thing I cannot affirm since I have never seen something that indicates this to be true, but his work can be radical compared to the results others get. I want to express that this is an honest question and I do not have another interest other than knowing what is this about and what could be the reasons for this. I do not want to attack him or anyone involved in this, I just would like to have a better framework of the people doing the measurements. Thanks for your time.

PS: this will be also published in Head-Fi Sound Science subforum to know their opinion about this issue.
PS(2): I want to reiterate that this is not an attack on the forum and its creator, this is an honest question that has been bugging me since I discovered this website.
PS(3): Do not direct me to the thread he explains how he measures and with what equipment, I already read it multiple times. I want to know the reasons the differences arise from different subjects and their measurements.

I am sure this was already asked but I don't want to read the whole thread and kinda feel the need to ask again: shouldn't you first and foremost provide some examples!? Or at least One, single, clear example!? Otherwise it's just funky trolling.

And no, the Schiit Ygg mesurements do not count as an example. That stuff was discussed till the end of the world and back and it's all just lalalalala and no issue was ever found with the ASR measurements.

 

[SimpleTheater]

There is no speaker on earth that will even remotely meet these numbers. At 90dB average SPL in an average listening situation you have the lot, THD, IMD, and FM.

EDIT - See Klippel for a lot of hard science on this.

Who’s talking about speakers? We’re talking about audible distortion. Of course no speaker has ever been that good, but who said there was? Not sure what you’re talking about, maybe just a misunderstanding, as Amir is demanding even less distortion than that link.

 

[scott wurcer]

Who’s talking about speakers? We’re talking about audible distortion. Of course no speaker has ever been that good, but who said there was? Not sure what you’re talking about, maybe just a misunderstanding, as Amir is demanding even less distortion than that link.

Do you have a means to listen directly to the signals out of your DAC? The speaker is in the chain, I find claims of -40dB speaker distortion having no masking effect on -110dB electronics distortion as extraordinary. I want to see the peer reviewed evidence. I'm talking about THD not EMI incursion or other non-input related design issues.

 

[KeithPhantom]

I would like to express my most sincere apologies to those who think I am here to waste your precious time. The reason I am here is to learn about a hobby that I like a lot and do not want to be ignorant of the experiences and the inner workings of the pieces of technology that allow my music to be reproduced.

I provided examples that in my ignorance seemed to be different, but it was quickly explained the reasons why this could be and that answered my question. As I stand right now, I want to pursue more knowledge by asking questions and making mistakes, this knowledge giving me more power to make informed decisions, and if it needed, to clarify and instruct others on topics they may not have the knowledge required to know the reason they experience something or they behave something. Now, if you think I'm here to "troll", you may point it out in one of my entries and if found true, I will apologize even though this is not my intention from the first time.

 

[SimpleTheater]

Do you have a means to listen directly to the signals out of your DAC? The speaker is in the chain, I find claims of -40dB speaker distortion having no masking effect on -110dB electronics distortion as extraordinary. I want to see the peer reviewed evidence.

Still not sure what you’re talking about. We’re talking DACs. That being said , I want the cleanest signal possible hitting my speakers, that way I only have one item in my chain that produces audible distortion.

 

[pozz]

I would like to express my most sincere apologies to those who think I am here to waste your precious time. The reason I am here is to learn about a hobby that I like a lot and do not want to be ignorant of the experiences and the inner workings of the pieces of technology that allow my music to be reproduced.

I provided examples that in my ignorance seemed to be different, but it was quickly explained the reasons why this could be and that answered my question. As I stand right now, I want to pursue more knowledge by asking questions and making mistakes, this knowledge giving me more power to make informed decisions, and if it needed, to clarify and instruct others on topics they may not have the knowledge required to know the reason they experience something or they behave something. Now, if you think I'm here to "troll", you may point it out in one of my entries and if found true, I will apologize even though this is not my intention from the first time.

Don't worry about it. Good to have you around.

 

[scott wurcer]

that produces audible distortion.

And lots of it.

 

[Trouble Maker]

Just saw one Gearslutz

The seems like a place a bunch of 27 year old incels making music in their parents basement would flock to for audio advice.

 

[JohnBooty]

Please read the following in the proper context, which is that I'm a huge admirer of Amir's work and the ASR community. I think ASR provides tremendous value to the audio world.

(Also, Amir has explained himself regarding this issue many times - I am not trying to change minds at this point)

This is criticism that seems to be mostly unfounded to me. Can you provide an example(s) where Amir did not recommend the Amp/DAC, but all of the issues were below the audible threshold?

An example would be the Crown XLS 1502 review. The measurements reveal this amp not to be particularly great, but I also believe folks would be hard pressed to tell the difference between this amp and one with higher SINAD in an average residential room where the noise floor is ~30-40dB.

It's the kind of real-world consideration that doesn't need to be repeated for many of the hobbyists and professionals that make up the bulk of ASR's forum membership, but for each of us there are dozens if not hundreds or thousands of others reading the reviews.

Besides, don't we already do some of that?

We laugh at products like the Sony bookshelves with "supertweeters" that play frequencies no human can hear.

Why don't we have the same skepticism and pragmatism when it comes to other measurements (SINAD, etc) that, past a certain threshold, guarantee transparency? If there's a real audible difference, why not do listening tests?

Like everybody else here, I certainly lust after gear like the Benchmark AHB2... I am an engineer and I want those numbers, dammit... but on the other hand, we should recognize that it's unlikely to provide audible benefit compared to amps with significantly lower SINAD.

Again, these aren't complaints. Just one thing I don't totally agree with on ASR... in a sea of many many things that I do appreciate.

 

[blueone]

Why don't we have the same skepticism and pragmatism when it comes to other measurements (SINAD, etc) that, past a certain threshold, guarantee transparency? Like everybody else here, I certainly lust after gear like the Benchmark AHB2... I am an engineer and I want those numbers, dammit... but on the other hand, we should recognize that it's unlikely to provide audible benefit compared to amps with significantly lower SINAD.

Measurements for electronics are about the degree of engineering excellence and detecting anomalous behavior. Arguing about what's audible and what isn't doesn't strike me as productive, because there are too many variables. Test tones or music? Speakers or headphones (for DACs and pre-amps)? Trained listeners with normal hearing or just anyone? Very quiet room or average room?

I don't know what's audible and what isn't when and under what circumstances. One thing is for sure, if equipment measures better it won't sound worse. Some people do like colorations, but I can't imagine making a buying decision based on someone else's preferences. Measurements, parts quality, build quality, fit and finish, ergonomics, functionality, and no obnoxious behavior (turn on or off thumps, switching noises, static, etc) are what carry buying decision weight. Well, and cost. I can't imagine how to judge audibility as a decision-making factor for electronics.

Do you disagree?

 

[Racheski]

Please read the following in the proper context, which is that I'm a huge admirer of Amir's work and the ASR community. I think ASR provides tremendous value to the audio world.

(Also, Amir has explained himself regarding this issue many times - I am not trying to change minds at this point)

An example would be the Crown XLS 1502 review. The measurements reveal this amp not to be particularly great, but I also believe folks would be hard pressed to tell the difference between this amp and one with higher SINAD in an average residential room where the noise floor is ~30-40dB.

It's the kind of real-world consideration that doesn't need to be repeated for many of the hobbyists and professionals that make up the bulk of ASR's forum membership, but for each of us there are dozens if not hundreds or thousands of others reading the reviews.

Again, these aren't complaints. Just one thing I don't totally agree with on ASR... in a sea of many many things that I do appreciate.

Alright, I'll give you that one, although it is like the most recommended non-recommendation one could make haha, "It is hard to fathom that a properly engineered and produced mainstream brand amplifier with so much power can be sold for so little money. This is a robust but bargain amplifier designed for heavy punishment in live sound. Can it be used for hi-fi use? Sure but it will severely limit both the resolution and potentially bandwidth of upstream sources. Its best use would be as a subwoofer amplifier."

 

[pozz]

The seems like a place a bunch of 27 year old incels making music in their parents basement would flock to for audio advice.

No. A lot of serious musicians and engineers are on that forum, and a lot of amateurs.

The science is less emphasized than here, but there is a good collection of experience that's worth reading as well. You end up having to do a lot of sifting. Plus they have no Thomas to moderate. Massive disadvantage.

 

[Trouble Maker]

No. A lot of serious musicians and engineers are on that forum, and a lot of amateurs.

The science is less emphasized than here, but there is a good collection of experience that's worth reading as well. You end up having to do a lot of sifting. Plus they have no Thomas to moderate. Massive disadvantage.

Maybe I'm not giving it a fare shake by this, but the name is enough to completely turn me off. The audiophile community (ASR much less than other places, just speaking more broadly) as a whole has a serious exclusivity problem about a broad swath of issues; age, gender, race, class, etc. Anything that walls off something like the joy of listening to and making music that should be open to all is not a good thing. And people will complain about this sector dying due to 'lack of young people' picking up the hobby. It's kind of mind boggling. Maybe the musician and engineering side is more open, but with a forum name like that, I doubt it.

 

[Racheski]

Why don't we have the same skepticism and pragmatism when it comes to other measurements (SINAD, etc) that, past a certain threshold, guarantee transparency? If there's a real audible difference, why not do listening tests?

Like everybody else here, I certainly lust after gear like the Benchmark AHB2... I am an engineer and I want those numbers, dammit... but on the other hand, we should recognize that it's unlikely to provide audible benefit compared to amps with significantly lower SINAD

I agree with you on this point for DACs & Head Amps (I'm much less familiar with the amp reviews). There are examples of recommended new devices whose only notable improvement over cheaper options are better SINAD, dynamic range, and power, etc...but the improvements are probably not audible.

 

[KeithPhantom]

People are having arguments about ASR on other forums all the time. Just saw one Gearslutz (worth reading the whole page):

It is surprising to find professionals that do not believe in measurements.

 

[idiomatically]

It is surprising to find professionals that do not believe in measurements.

This is the first clear cut sign of cognitive bias, imagine if people built houses without measurements? Or Medical Physicists designed medical equipment or nuclear sciences without measurements... there is a reason we have engineers build bridges etc. It‘s like they will use measurements in every other aspect of their lives, but hold the phone when it comes to audio somehow the measurements don’t matter... yep, how is that logical in any way shape or form.

That‘s like people saying they won’t do chemo to treat cancer etc and use natural home remedies instead, how does that work out?

Here’s the kicker, if you walk into any Quality Assurance department of any audio company, like AudioQuest etc guess what you will find? Measurements, that is exactly how they determine a pass or fail for quality, they measure them. How to they design a cable? Off a spec sheet full of measurements.

I have also been lucky enough to work with some very high end production studio’s in the past and not one of them used $1000+ power cables and all that crap typical “audiophiles” believe in.

 

[SIY]

Here’s the kicker, if you walk into any Quality Assurance department of any audio company, like AudioQuest etc guess what you will find?

They have a QA department?

 

[pozz]

Maybe I'm not giving it a fare shake by this, but the name is enough to completely turn me off. The audiophile community (ASR much less than other places, just speaking more broadly) as a whole has a serious exclusivity problem about a broad swath of issues; age, sex, race, class, etc. Anything that walls off something like the joy of listening to and making music that should be open to all is not a good thing. Maybe the musician and engineering side is more open, but with a forum name like that, I doubt it.

It's close to 20 years old at this point and has gained a lot of traction, being partially a platform and a marketplace. Mostly guys discussing gear with other guys, of course.

It is surprising to find professionals that do not believe in measurements.

If you read some audio history you'll find that the accurate measurement techniques and interpretation required more physics knowledge than is typically held, were expensive and gave only rough guidance (1/3 octave was typical) until just recently, when computing became cheap and analysis software readily available. Bad impressions tend to stick. This of course doesn't excuse the wilful ignorance or stubbornness shown by some.

 

[Blumlein 88]

Maybe I'm not giving it a fare shake by this, but the name is enough to completely turn me off. The audiophile community (ASR much less than other places, just speaking more broadly) as a whole has a serious exclusivity problem about a broad swath of issues; age, gender, race, class, etc. Anything that walls off something like the joy of listening to and making music that should be open to all is not a good thing. And people will complain about this sector dying due to 'lack of young people' picking up the hobby. It's kind of mind boggling. Maybe the musician and engineering side is more open, but with a forum name like that, I doubt it.

It is the largest forum for pros in the music recording business. Back in the beginning it was mostly full time pros only. Very respected and rightly so at the time as a forum. True not all pros are big on science, but they make their living with what they do. And yes OTOH some are very big on science.

In the business with so many new never before possible products due to digital, and gear lust for older gear too, GearSLUTZ was very appropriate. You've done a big time award winning platinum recording people want to know, "what mic pre was that, what compressor did you use, what mike on the vocals". Of course it was a bit of a joke name too. They regularly told people, it is more about know how and who does a recording than what gear being used. Real pros make good recordings, and wankers can't get a good jingle out of Abbey Road.

Now with the influx of affordable quality recording interfaces, affordable good microphones (do you realize how expensive condenser mikes of any kind once were?), and the music business being in decline leading to lots of amateurs and self recorded bands topics have lower Signal to noise than once was the case. Still lots of good info if you pay attention.

 

[pozz]

I agree with you on this point for DACs & Head Amps (I'm much less familiar with the amp reviews). There are examples of recommended new devices whose only notable improvement over cheaper options are better SINAD, dynamic range, and power, etc...but the improvements are probably not audible.

Audibility shouldn't be at the front of mind for electronics. That should be reserved for system compatibility and optimization. Audibility is a speaker thing. Small proof is that speakers discussion here always reference tonality, while electronics discussions are mostly technical, making sure the signal is passed from one component to the next without problems (like the common "How many watts do I need?" question).