• WANTED: Happy members who like to discuss audio and other topics related to our interest. Desire to learn and share knowledge of science required. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

Why Audio Science Review's measurements are so different compared to other sources?

richard12511

Major Contributor
Forum Donor
Joined
Jan 23, 2020
Messages
4,335
Likes
6,700
Definitely. There's often a complex stew of factors that goes into any buying decision. Measurements are one factor. Features, build quality, appearance...those are factors as well. oh, and price of course.

Agreed. I would even say that for DACs, measurements are at the very bottom of the list of things that matter to me. Features and appearance are far more important than measurements, given that they all sound the same.
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,383
Likes
24,749
Location
Alfred, NY
Basic problem: SINAD conflates several things, which may or may not be audible under different circumstances. For example, something with low harmonic distortion but a higher noise floor will be different than something with a low noise floor and higher harmonic distortion. Not to mention the confounders of harmonic distribution and noise floor spectral balance.

I don't want to be Debbie Downer, but this seems like a fool's errand.
 

Racheski

Major Contributor
Forum Donor
Joined
Apr 20, 2020
Messages
1,116
Likes
1,699
Location
Chicago
Absolutely, BUT....

In the name of pragmatism, let's also consider that the future of our hobby depends on attracting newcomers, who by definition start out completely uninformed.

Most will not be lazy "tell me what to buy" types, but all will be uninformed. :)

(Also, a lot of the thoughtful and polite ones, who don't ask such obnoxious and lazy questions, will nonetheless read the replies to those lazy "tell me what to buy posts" as they search for newbie-friendly information...)
Maybe we just leave the graphs alone and write the “ASR Guide to Buying your first DAC/Amp” ? We can explain the SINAD graph & the nuances of how it relates to audible transparency in the guide, and of course other relevant information folks want newbies to consider for their first purchase. That should ensure a more consistent baseline level of knowledge for newcomers, and lead to more informed “what should I buy?” discussions if still necessary.

Or maybe do both the guide and update the graphs.
 
OP
KeithPhantom

KeithPhantom

Addicted to Fun and Learning
Forum Donor
Joined
May 8, 2020
Messages
641
Likes
658
Are you new to all this?
Not to the audiophiles not believing in measurements, but even those who are trained with emphasis on measurements not believing in them is surprising to say the least. By the way, I'm not an audiophile.
 

Jimbob54

Grand Contributor
Forum Donor
Joined
Oct 25, 2019
Messages
11,064
Likes
14,694
Basic problem: SINAD conflates several things, which may or may not be audible under different circumstances. For example, something with low harmonic distortion but a higher noise floor will be different than something with a low noise floor and higher harmonic distortion. Not to mention the confounders of harmonic distribution and noise floor spectral balance.

I don't want to be Debbie Downer, but this seems like a fool's errand.

Dare I ask which bit if this spiralling thread that's referring to? Slicing the charts differently?
 

AnalogSteph

Major Contributor
Joined
Nov 6, 2018
Messages
3,334
Likes
3,278
Location
.de
Dare I ask which bit if this spiralling thread that's referring to? Slicing the charts differently?
Indeed.

He's right, btw.

Let's say we've got two DACs of nominal 2 Vrms output:
1) SNR = 120 dB, THD = -100 dB
2) SNR = 100 dB, THD = -120 dB.

SINAD for these would be identical, and they may well be equivalent as a high-level source followed by a volume control, but I assure you that I would very much recommend against using (2) directly on a typical speaker power amplifier of ~29 dB gain if you are allergic to noise and/or have sensitive speakers and/or short listening distance.
 
Joined
Jul 3, 2020
Messages
5
Likes
3
I can’t speak for others, but I’ve had no trouble reproducing Amir’s measurements when I’ve had a shot at the same gear. And as I’ve noted before, Amir is the only reviewer besides me who offers to share his AP project files.

One other observation: the one time I posted honest measurements with full disclosure of methods at Head Fi, my posts were edited and removed because of embarrassment to an advertiser. I never posted there again.


Hi There! Could you direct me to a post that would help me try to reproduce results and perform some tests?

Specifically, I piloted several pre-amp/processors at home -- Datasat LS10, Integra DRC1.1, Emotiva XMC-1. From a listening perspective I found the XMC-1 was my favorite, so I got the RMC-1 upgrade. I'd love to try to do a side-by-side test of the LS-10, DRC1.1, and RMC-1 to see what results I get, and would love the help.

Also, I'd be happy to lend the LS10 or DRC1.1 for testing by ASR/Amir.
 

SIY

Grand Contributor
Technical Expert
Joined
Apr 6, 2018
Messages
10,383
Likes
24,749
Location
Alfred, NY
Hi There! Could you direct me to a post that would help me try to reproduce results and perform some tests?

Specifically, I piloted several pre-amp/processors at home -- Datasat LS10, Integra DRC1.1, Emotiva XMC-1. From a listening perspective I found the XMC-1 was my favorite, so I got the RMC-1 upgrade. I'd love to try to do a side-by-side test of the LS-10, DRC1.1, and RMC-1 to see what results I get, and would love the help.

Also, I'd be happy to lend the LS10 or DRC1.1 for testing by ASR/Amir.

Piloted?

I'd start with doing some controlled listening tests to see if your preferences are actually about the sound. That means ears-only (no peeking) and level-matched (to within 0.1dB or so). You may find that the results have an influence on your desire to do measurements, especially after seeing the price tag on an AP setup...

Here's a good place to start.
 

Racheski

Major Contributor
Forum Donor
Joined
Apr 20, 2020
Messages
1,116
Likes
1,699
Location
Chicago
If someone with admin rights to the tableau dashboard export can export the flat file of the underlying data as a .csv/.txt/.xlsx I'll play around with the SINAD graph and come back with some samples for members to review. Should I reach out to Amir for this?
 

Jimbob54

Grand Contributor
Forum Donor
Joined
Oct 25, 2019
Messages
11,064
Likes
14,694
If someone with admin rights to the tableau dashboard export can export the flat file of the underlying data as a .csv/.txt/.xlsx I'll play around with the SINAD graph and come back with some samples for members to review. Should I reach out to Amir for this?

Suspect @pozz has it
 

pozz

Слава Україні
Forum Donor
Editor
Joined
May 21, 2019
Messages
4,036
Likes
6,825
If someone with admin rights to the tableau dashboard export can export the flat file of the underlying data as a .csv/.txt/.xlsx I'll play around with the SINAD graph and come back with some samples for members to review. Should I reach out to Amir for this?
I capture the data for ASR. Amir's position is that it should be kept private. You can PM me though and we can talk about what you have in mind if the below doesn't address it.

I actually made a series of SINAD charts around a year ago which had five divisions:
  1. >120dB (>20 bits): Provably inaudible noise and distortion.
  2. 109dB—119dB (18 bits—20 bits): Well beyond 16-bit (CD) resolution.
  3. 96dB—108dB (16 bits—18 bits): Meets and exceeds 16-bit (CD) resolution.
  4. 86dB—95dB (14.3 bits—16 bits): Does not meet 16-bit (CD) resolution.
  5. <85dB (<14.3 bits): Performance is below the minimum lenient threshold defined in this thread.
This was based on a proposal by @martijn86. 96dB was the middle because it's the total resolution of CDs, not considering dither. Besides being a 30 year old format, there's a study I can't recall at the moment which said that 96dB was generally enough for transparent reproduction (other studies push for higher figures for absolute transparency). There was a bunch of discussion and that was the approach for a few months before the data upkeep forced a new approach and highlighted some problems.

The first problem is that psychoacoustic concerns for the entire 120dB range of human hearing are pertinent only for DACs and preamps. Every other device will fail in some metric, least of which is SINAD. This means that you already have to take engineering ability/possibility into account.

The next is that we are assigning a ranking based on a single device and not the chain. Noise and distortion are cumulative. The common assumption holds that upgrading one device in the chain will improve some aspect. But there can be no commentary about the effect of a single device on an undefined chain. This means that we can't address specific use cases.

Related is that people want precision about expectations. The complexity of audio means that we can make basic, only partially helpful comments. For combo DACs/headphone amps where you would just plug in a headphone, you would still have to understand what kind of headphone you're using and its electrical requirements at least to some degree before buying (like impedance, sensitivity). But then on top of that buyers want specifics beyond electrical compatibility: what subjective qualities will be experienced? This is where the talk of blacker backgrounds, greater separation between instruments and so in comes in, which is immensely mistaken form of description and creates an incredible industry problem.

This last point some may consider less important. There's a lot of attention on this one metric, SINAD, but there are bunch more that are in the database (power under different loads, SNR, headphone amp output impedance). Consistency would require us to figure concrete thresholds for all of them. But all that requires some specificity about the use case and device type. It is probably underappreciated just how many devices types there are when considering their construction and purposes. There are over 70 categories in the review index, for example.

The conclusion I've come to is that we should emphasize and make other communities understand that these charts are not a replacement for reading reviews and having some background knowledge. We should resist the impulse to make these charts more meaningful or apply audibility thresholds to them. They are simply a list of results.

I think the reason this is unsatisfying is because the drive towards meaning is part of the day-to-day discourse in audio despite it being a phenomenal problem to assume that every design decision will alter what you hear. We should not add to that by allowing measurement results to be interpreted in a manner which will fetishize them the same way materials and other aspects are fetishized.

In that sense ASR's reviews represent something else entirely, and can't be lumped in with those elsewhere that promise and deliver immediately meaningful conclusions. At ASR the conclusion is subordinate to the measurement information, and the ranked measurements are a general shorthand for the work that's been done up to that point.

If there is one thing to take away it's that it is a fallacy to extrapolate subjective sound expectations from single-number ranked chart.

For the ranking itself, the most recent work by @RickSanchez, @Koeitje and I would be to have user-selectable divisions based on categories of quartile colouring (aka a statistical division of the sample and nothing more), yes/no recommendation colouring and a no division single-tone "off" option, with colour-blind versions of each. This is still a work-in-progress but expect to see it.

What remains is the buyer's guide aspect. What's good enough? What's great? This is largely determined by what's available on the market at what prices, and the collection of measured products. The main reason I wouldn't buy a DAC that shows 80dB SINAD at max output is because I can buy one that shows over 110dB for cheap, with no downsides if I do the research right. The figures are similar for headphone amps and lower for power amps. There are still plenty of details to consider outside of the chart itself.

Having this conversation over and over is probably the best way for it to take hold.
 

martijn86

Active Member
Joined
Aug 30, 2019
Messages
271
Likes
975
Location
The Netherlands
Hey Pozz!

Thanks for the mention. So nice to see you remember by ranking suggestions!

there can be no commentary about the effect of a single device on an undefined chain.

it is a fallacy to extrapolate subjective sound expectations from single-number ranked chart.

The way I've always looked at ranking electronics on ASR is that it gives a metric of engineering quality of the specific product in review.
The meaningful part of divisions, numbers and ranking, to me, was never meant to give an impression of what the product would sound like. It was always an aid to 1) improve the ability to scan the graph and 2) to give novice visitors a frame of reference on the quality that has been achieved. A step in between the panther and looking at the FFT graph.

There are still plenty of details to consider outside of the chart itself.

Absolutely. SINAD vs price is not a good buyers guide. It is however a very helpful piece of the puzzle. Other pieces are functions, controls, price, looks and IO but none are metrics for engineering quality. Once you have a few matches, my advice would be to order your top 3 favorites for an in-home trial. More often than not you can find a store that offers free returns within 30-day's. Keep rotating the products in your system until you tend to only want to use one favorite. Send back the rest with care.
 

RickSanchez

Major Contributor
Cartographer
Joined
Sep 4, 2019
Messages
1,168
Likes
2,490
Location
Austin, TX
... The conclusion I've come to is that we should emphasize and make other communities understand that these charts are not a replacement for reading reviews and having some background knowledge. We should resist the impulse to make these charts more meaningful or apply audibility thresholds to them. They are simply a list of results ...

Completely agree.

The dashboards/charts embedded in the ASR site -- along with some other great tools built by ASR members but hosted elsewhere -- provide a map to the universe of products that have been measured / reviewed here. With some good design those dashboards ideally make it easy to view and drill into that universe in different ways. But in the end the complexity of the products themselves and the complexity of the full audio chain require additional research and understanding.
 

Racheski

Major Contributor
Forum Donor
Joined
Apr 20, 2020
Messages
1,116
Likes
1,699
Location
Chicago
Thanks for the detailed reply. IMO your color coding system based on bits of resolution is much better than the arbitrary quartiles, and some of the problems you pointed out are actually better addressed by this categorization, e.g.
The conclusion I've come to is that we should emphasize and make other communities understand that these charts are not a replacement for reading reviews and having some background knowledge. We should resist the impulse to make these charts more meaningful or apply audibility thresholds to them. They are simply a list of results.

I think the reason this is unsatisfying is because the drive towards meaning is part of the day-to-day discourse in audio despite it being a phenomenal problem to assume that every design decision will alter what you hear. We should not add to that by allowing measurement results to be interpreted in a manner which will fetishize them the same way materials and other aspects are fetishized.
These are contradictory statements. ASR wants to encourage others to read the reviews and increase their background knowledge so that data can be analyzed and interpreted in an appropriate context, but present a graph in a way that can easily have spurious conclusions drawn.

Here is Amir's quote from the Understanding Audio Measurements article regarding the SINAD chart:
Since there are product variations and test conditions that vary somewhat, I don’t suggest readers look too closely at the SINAD numbers. Instead, go by what “bucket” they fall in based on color-coding. For best performance you want products in the blue bucket and avoid those in red.
These are not really "buckets" because the quartiles are dynamic - a DAC in the blue today may not be in the blue 6 months from now, and why does the top 25% represent the best performance? Why not the top 10%? Why not the top 15%? It is arbitrary, and since it is arbitrary users are more likely to draw spurious conclusions like the higher the SINAD, the better sounding the device, or DACs in the blue range are significantly better performing than DACs in the green range.

The color coding has been discussed ad nauseam at this point - so my recommendation is to use your proposed categories based on bits of resolution for the SINAD chart and see how the community reacts. If you don't want to try that, I understand why so no problemo. I could play around in Excel & give you some marginally better ways to visualize the graph, but then we would have to try and translate those changes in Tableau, and that juice is not worth the squeeze.

What I think are the more interesting and meaningful discussions coming out of this (to have in separate threads) are (1) How to "cure" the SINAD Wars/hyperSINADitis & (2) why the barrier to entry into Hi-Fi Audio is relatively higher vs other tech hobbies, and what can be done to address the issue (like a buyer's guide).
 

Koeitje

Major Contributor
Joined
Oct 10, 2019
Messages
2,292
Likes
3,880
Buckets don't have to be fixed. If we use fixed buckets for SINAD we will be revising them every year. The only way to divide the data in groups that don't need adjustment is basing them on what is actually outside human hearing, but that can't be done from a single data point. Percentile based scoring is the best way to show the top performers at this point in time.

By the way, barrier of entry for hifi has never been as low as it is now. But as for providing information for new buyers: that is what the recommendation rating is for.
 
  • Like
Reactions: PSO

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,381
Location
Seattle Area
These are not really "buckets" because the quartiles are dynamic - a DAC in the blue today may not be in the blue 6 months
??? The buckets have not changed since inception. They are defined with hard boundaries.
 
  • Like
Reactions: PSO

amirm

Founder/Admin
Staff Member
CFO (Chief Fun Officer)
Joined
Feb 13, 2016
Messages
44,368
Likes
234,381
Location
Seattle Area
These are not really "buckets" because the quartiles are dynamic - a DAC in the blue today may not be in the blue 6 months from now, and why does the top 25% represent the best performance? Why not the top 10%? Why not the top 15%? It is arbitrary, and since it is arbitrary users are more likely to draw spurious conclusions like the higher the SINAD, the better sounding the device, or DACs in the blue range are significantly better performing than DACs in the green range.
You are misunderstanding the colorization. The metric is simply the range of SINAD, not what percentage fits in each bucket. It is by accident that the four buckets seem to have equal number of products in them (visually that is).
 

Racheski

Major Contributor
Forum Donor
Joined
Apr 20, 2020
Messages
1,116
Likes
1,699
Location
Chicago
You are misunderstanding the colorization. The metric is simply the range of SINAD, not what percentage fits in each bucket. It is by accident that the four buckets seem to have equal number of products in them (visually that is).
The legend in the DAC Measurement comparison chart has them labeled as percentiles - “SINAD Percentile Score”
 
Top Bottom