Standards for Audio Electronics Measurements?

[Chrispy]

I generally think the limited testing here should be accepted for what it is, but having an agreed set of goals would be good, too bad the industry largely isn't going to participate a lot in that. As long as everyone understands what the parameters are in terms of goals, that seems like a good thing once they're agreed upon (and who agrees to them of course). Staying tuned in, should be interesting....

 

[restorer-john]

Balanced automatically gives you double the output voltage. If it's below 4 V the circuit either has a problem, or unbalanced also is below 2 V - so requirement not met.

A dongle is only 5V powered in most cases. 1.5-1.6V maybe if you're really lucky. You won't get your 2V RMS swing unless it has (noisy) onboard step up regulators.

 

[restorer-john]

What say you?

Amplifier meaning what exactly, headphone, power or integrated (with preamp)? The performance expectations would be all over the shop for those three devices.

It needs some serious thought and a lot of considered parameters. Whether that takes the line of guardrail limits to trigger the equivalent of tiered pass/fails or a group of parameters that when met, tip the product into the next tier/bucket I don't know.

And when you say announce expectations, that's almost like a bare-minimum bar that must be cleared to get considered for review?

 

[xaviescacs]

Personally, I like things to be more open and less strict. To me, it's a bit arbitrary, unnecessarily arrogant and potentially harmful for the community. Imagine someone builds a dongle with a lot of awesome, unimaginable features that lasts forever but can't reach the 2 V requirement for whatever reason and never gets a pass. Is that reasonable or informative? Or an audio interface with lots and lots of features for 90 bucks that doesn't reach the SINAD limit for 2 points...

I understand that some manufacturers ask for minimums to avoid being discarded out of the box, but I would avoid establishing them explicitly and just share some loose minimums that in any case doesn't guarantee a pass or a fail, it's part of the game, right? This is not a legal certification.

I love those reviews that somehow contradict themselves, because some spec isn't matched but then a very nice feature or awesome spec appears and finally the question is let open to debate. I don't think it's wise to loose that.

Besides, setting thresholds constitutes a major compromise. Are we sure we won't be changing them or make exceptions in some devices? What about devices already reviewed then?

To me, the risk is defeating the purpose. Just my opinion.

 

[TimoJ]

AVRs:
Output level: Minimum of (undistorted) 2 volts for unbalanced, 4 volts for balanced.
DAC section:
SINAD: AVRs, better than 95 dB. AVPs, better than 100 dB
DNR: >= 115 dB
Jitter (all inputs): < -115 dB (in FFT spectrum)
Filter attenuation 44.1 kHz sampling at 24 kHz: better than -100 dB
Frequency Response flatness: Less than 0.2 dB @ 20 kHz

You should add more tests for AVRs/AVPs. Those are not used as DACs in pure direct mode, so measuring just that may not give the full picture of their performance in real life usage.

You should make a test setup for at least two channel room correction and measure performance with the correction active. It only requires a suitable mixer, no speakers/mic are needed. That would at least give some indication how things are when room correction is active.

 

[phoenixdogfan]

Haven't thought through all the details for different device categories. Thinking out loud, it would be kind of like this:

DAC:
Output level: Minimum of 2 volts for unbalanced, 4 volts for balanced.
SINAD: >= 110 dB
DNR: >= 115 dB
Jitter (all inputs): < -115 dB (in FFT spectrum)
Filter attenuation 44.1 kHz sampling at 24 kHz: better than -100 dB
Frequency Response flatness: Less than 0.2 dB @ 20 kHz

Don't know yet how to include responses for the sweeps such as IMD. And for multitone, need to do some research to see what is a reasonable minimum.

AVRs:
Output level: Minimum of (undistorted) 2 volts for unbalanced, 4 volts for balanced.
DAC section:
SINAD: AVRs, better than 95 dB. AVPs, better than 100 dB
DNR: >= 115 dB
Jitter (all inputs): < -115 dB (in FFT spectrum)
Filter attenuation 44.1 kHz sampling at 24 kHz: better than -100 dB
Frequency Response flatness: Less than 0.2 dB @ 20 kHz

Amplifier Section:
SINAD with digital or analog input > 80 dB
SNR: ???

You get the idea.

I think you might need to consider tube amps a special case. Obviously, they are not going to clear the 80db SINAD hurdle by virtue of the express purpose of their design. Given that, the question then is whether there is a level of SINAD below which even a device designed to produce euphonic distortions should no longer be considered performant?

 

[kchap]

A dongle is only 5V powered in most cases. 1.5-1.6V maybe if you're really lucky. You won't get your 2V RMS swing unless it has (noisy) onboard step up regulators.

Good point. I was testing my new toy, an inline USB power monitor. With a 2A load the plug pack voltage was 4.8V.

 

[Beershaun]

Haven't thought through all the details for different device categories. Thinking out loud, it would be kind of like this:


Amplifier Section:
SINAD with digital or analog input > 80 dB
SNR: ???

You get the idea.

Why 80 for Amplifier SINAD? I thought min bar for reproducing CD quality was 96db.

I like @IAtaman's rubric for the different metrics, because it gives you the freedom to grade on a curve as you do with AVRs while still setting a goal to aspire to like "provably transparent" or "transparent at CD quality."

I think explaining the min. and goal for each number as you did for the 2V and 4V number and "CD transparent" SINAD and "provably audibly transparent" SINAD is helpful for manufacturers and for readers to legitimize why you are demanding those numbers. It also gives you some room to move the bar if you wish depending on things like amplifier gain. For example, I'm not sure if there is a desired target gain or if they should be different categories E.G. should an 11db gain amplifier be compared to a 29db gain amplifier?

The reason I suggest that is "You get what you measure." And sometimes undesirable behavior can be encouraged when a business chases a subset of metrics when the whole picture is unclear or under described and end up creating something inferior. Would we want amplifier manufacturers to all start creating lower gain amps just to improve their SINAD numbers? (I don't actually know the answer to this and am interested to learn).

 

[jasonhanjk]

For a dongle to output 2Vrms, it's very suitable for high Z HP.
However most IEM users won't benefit from it.
For portability, dongles are suitable and best paired with IEM.

How about making the measurements into 2 tier.
1st tier: for dongles that are $50 and below:
Output: >=1Vrms
SINAD: >=100dB with 32 ohm load between 1mW to 10mW)
Noise: <=-110dBV (20Hz to 20kHz bandwidth)

2nd tier for dongles above $50.

Once the device pass these 3 criteria, then it make sense to proceed for a full measurement as it posses a high chance to get ASR recommendation.
If it fail, just mention it fail in a list and no further test required.

 

[Beershaun]

I think you might need to consider tube amps a special case. Obviously, they are not going to clear the 80db SINAD hurdle by virtue of the express purpose of their design. Given that, the question then is whether there is a level of SINAD below which even a device designed to produce euphonic distortions should no longer be considered performant?

they could be a separate category but there would need to be a "why" explainer and what is being targeted or desired. You still want low noise floor, high power, but maybe there is some published research on 2nd order harmonics that is correlating it to some sort of listener preference. Then there could be a target. Without a research based target though it's hard to create a separate standard.

 

[wwenze]

To close the circle - and move the industry ahead, I think we need to create "audibly transparent" set of thresholds, confirmed by controlled experiment.
Add some margin, e.g. 3dB SINAD on top and then do simple fail/pass evaluation. You can have another set "audibly transparent for Hi-Res" or smth similar for DAC's.

Once there is scientifically defined "good performance" producers can focus on the right things - better features, form factor, durability etc, instead of chasing 1dB SINAD more, in order to get to the top of the SINAD chart.

I personally fail to get excited by another DAC that is another 1dB better in SINAD, when it is obvious, that even the ones that are 20dB worse have enough fidelity to reproduce music 100% accurately.

"Audibly transparent" is subjective and situational. Reliably it is around 0.1% or -60dB, however with specific testing we can possibly hear down to 0.01% / -80dB. Then there's also gain structure and raw noise values too - IEM users and people who put their OS volume at 20% are going to need even better noise performance. Then there is also the issue of absolute SPL... A speaker with 0dBFS @ 80dB SPL will produce 0dB SPL @ -80dBFS...

Which is why grading by tier is much better in practice. Every 10dB is one tier: 100dB SINAD is green, 110dB SINAD is blue, 90dB SINAD is orange... and the color is appropriate because 90dB SINAD is what we should expect from "free" bundled output these days and anything below is dirt cheap Temu stuff and anything sharing that performance should be appropriately labelled red... Mind you this doesn't mean these product are unusable, they often work well enough for the average Temu customer.

 

[Beershaun]

"Audibly transparent" is subjective and situational. Reliably it is around 0.1% or -60dB, however with specific testing we can possibly hear down to 0.01% / -80dB. Then there's also gain structure and raw noise values too - IEM users and people who put their OS volume at 20% are going to need even better noise performance. Then there is also the issue of absolute SPL... A speaker with 0dBFS @ 80dB SPL will produce 0dB SPL @ -80dBFS...

Which is why grading by tier is much better in practice. Every 10dB is one tier: 100dB SINAD is green, 110dB SINAD is blue, 90dB SINAD is orange... and the color is appropriate because 90dB SINAD is what we should expect from "free" bundled output these days and anything below is dirt cheap Temu stuff and anything sharing that performance should be appropriately labelled red... Mind you this doesn't mean these product are usable, they often work well enough for the average Temu customer.

According to this it's -120db. Where does the -60db number come from? https://www.audiosciencereview.com/...-thresholds-of-amp-and-dac-measurements.5734/

 

[Andreas007]

I think setting a standard by Amir/ASR like "ASR certified" or something else is not necessary. This project started free of any constraints and yet managed to have tremendous impact on the industry (heck, he even got invited to AES). That's quite an accomplishment!
A new "standard" by ASR is simply adding a ton of bureaucracy to this. It would limit Amirs' work and freedom to speak-up and most certainly end up in charging license fees and commercialization. Not a direction I would like to see. But, of course that's for Amir to decide.

 

[aatoma]

As much as I like the idea of being able to say 'this device clears the basics' at a glance (some sort of 'pink panther standard unit') I'm afraid this could quickly become a rabbit hole: at what room temperature? relative humidity? radiation background? and so on, which makes it impossible to go the ISO, ANSI, IEEE etc. way. I don't think the industry at large will follow, but there are already vendors out there proudly showing the link to an ASR review in their product pages. Still, I like the idea. And the thresholds could move forward in time as SOTA progresses, with a periodic review of the agreed 'basics'

 

[amirm]

Good for you that you know what is important and what is not, speaking about commercial audio electronics products.

Looks like you missed the intent. People send me products expecting a positive review only to find out that they miss simple specifications that we like to see before recommending a product. This has little to do with the entire industry. They can do whatever. But if they want or nod, then we owe it to them clarity on what it takes as a minimum.

 

[ralph]

Looks like you missed the intent. People send me products expecting a positive review only to find out that they miss simple specifications that we like to see before recommending a product. This has little to do with the entire industry. They can do whatever. But if they want or nod, then we owe it to them clarity on what it takes as a minimum.

Like the idea but we are talking moving targets here - like this year's standard?

 

[Blumlein 88]

Standards

xkcd.com

 

[Triliza]

I think setting a standard by Amir/ASR like "ASR certified" or something else is not necessary. This project started free of any constraints and yet managed to have tremendous impact on the industry (heck, he even got invited to AES). That's quite an accomplishment!
A new "standard" by ASR is simply adding a ton of bureaucracy to this. It would limit Amirs' work and freedom to speak-up and most certainly end up in charging license fees and commercialization. Not a direction I would like to see. But, of course that's for Amir to decide.

I agree. We have seen even the big guys (Denon, Genelec has commented on some measurements, and many more companies) responding to ASR reviews and many times Amir would comment on how companies have fixed issues from their older models tested, so I would like to think that by now most of the companies are aware of what Amir (and to a degree the ASR fellowship) expect to see from their products. If they do so or not may be related to other factors, costs, intended audience, parts availability and other limitations.

May be wrong on this, but for an initiative of establishing standards to be successful, I think all parts (i.e the industry) should be involved, otherwise it's bound to cause further confusion and not have the desired outcome. And this is how things like that starts, with discussion like this.

 

[Geert]

People send me products expecting a positive review only to find out that they miss simple specifications that we like to see before recommending a product. This has little to do with the entire industry.

You wonder if it's time to publish guidelines, but in practise you're already applying them in your reviews. So it would indeed be more transparent to summarise and publish them. That would also support the public discussion about the validity of the criteria. As such it would also be nice to document the rationale with each criterium.

To avoid discussions about semantics and to avoid misinterpretion of the objective it might be an idea to title the guidelines as for example 'What qualities we appreciate'.

So far for some random thoughts.

 

[amirm]

I am aiming for really low hanging fruit -- what we all like to see but don't have it written down.

And when you say announce expectations, that's almost like a bare-minimum bar that must be cleared to get considered for review?

No, people can send anything for review. This is for manufactures who send me products thinking they already meet our expectations when in reality don't.