Standards for Audio Electronics Measurements?

[Madlop26]

ASR standard for audio companies...., sounds excellent!, and challenging; I am familiar with standard of care in the medical field, quiet accomplishment, but only achieved after serious research and countless debates among experts, certainly we do not have to be at that level, but it seems to me that each parameter will require a base of scientific data, and some consensus among the smartest and more knowledgeable guys in the forum. I am not one of those, but I would be so pleased to be a spectator.

 

[beeface]

The requirement comes from the point of view of driving high impedance headphones. Highly popular headphones have 300 ohm impedance. Divide 1 volt by 300 and you get so little power to drive them. Go to 2 volts and now you have much healthier level of power. 4 volts becomes even nicer and is the differential version of 2 volts.

Thinking about this specific example, I feel like things get a bit murky. A few thoughts, sorta playing devil's advocate:

• A dongle is - according to its form factor - a "portable" product and would be ideally used with headphones designed with portability in mind.
• I imagine open-back Sennheiser HDxxx headphones make up a large portion of the high popular 300ohm headphones sold. These are generally not suitable as portable headphones, especially in public settings.
• There is arguably also an onus on headphone manufacturers to make low impedance products to use with portable devices, whether that's a phone, DAP or dongle.
• The issue of course is that manufacturers can design their products for a specific purpose, but they cannot dictate how consumers use their products.
  • Example: a dongle is designed with a certain customer in mind: they listen to IEMs and mostly listen to their portable system during their daily commute on the train. However, one customer may want to use the dongle to listen to music in hotel rooms because they often travel for business. Another might want to use it with their home desktop because a dongle is an economical option as a combination DAC/headphone amp.
  • Should the manufacturer design their dongle to appeal to all of the consumers mentioned in my previous point? Or is it up to the consumer to choose a product that is suitable?

 

[tmtomh]

No, I am asking if we should announce what our expectations are with respect to measured performance. What those specs are would be the second phase after we answer that question.

I think this would be a good idea - and I would also agree that this should not go so far as to be some kind of ASR Certification, or even an official ASR Standard.

Instead, I would suggest a "sticky" announcement at the top of every review/gear subforum provding two basic categories of info, as you outline in the comment I've quoted:
1. ASR expectations for measured performance; and
2. Benchmark specs that ASR considers to fulfill those expectations for various kinds of gear.

For item 1, I would assume expectations would be as concise and universal as possible, for example:

• Gear should have levels of noise, distortion, and other nonlinearities below the threshold of human hearing when that is feasible for a given category of equipment; and as close to those thresholds as possible when that is not yet feasible for a given category of equipment.
• For certain categories of equipment, levels of noise, distortion, and other nonlinearities should be at least 10dB beyond the threshold of human hearing. These categories include source components whose nonlinearities can be magnified by downstream components and/or internally available DSP; and categories of equipment in which the current state of technology and engineering allow this level of performance to be achieved with run-of-the-mill designs and low price points - in other words, categories in which this level of performance has become a sign of competent design.

Item 2 would be more complicated and take more time - but I think explaining ASR expectations, especially some version of the second item just above, would help manufacturers, vendors, and end-users all better understand the reasoning behind specific ASR spec expectations. For example, it would help folks understand why the DAC SINAD standard is so much higher than the amplifier one: the DAC is upstream and you want as clean a signal as possible to be delivered to the amp; plus it's not yet routine or easily achievable to create a power amp that can deliver 110dB SINAD, whereas a good DAC can easily achieve 115dB or better for just a couple hundred bucks.

Making these basic principles/rationales explicit would hopefully also reduce the "ASR is just stupidly chasing SINAD" critique.

 

[sam_adams]

For the testing of digital devices—DACS and ADCs, I would advocate for the adoption of testing standards based on the published AES standards for the measurement of digital devices. This provides for a reference standard that could be replicated by anyone with the proper understanding of how to implement them. If there is a goal to publish AP testing scripts that could be used by anyone with compatible testing equipment, publish them under an open-source license to a repository on Github or similar so that use of those scripts would have to carry an attribution to ASR/Amir and any modifications to those scripts would have to be shared back to the community.

 

[HarmonicTHD]

So I keep running into companies who send me products with best intentions but miss key aspects that cause me to not recommend. And example is a very nice dongle I recently received that had balanced output but was limited to just 2 volts out. If you don't know, I like to see minimum of 4 volts out from such a port as otherwise, you can find unbalanced dongles at lower cost that do that.

Another example is channel balance issue with had with an AIYIMA amp where there was almost 1 dB differential.

Yet another is expected SINAD for an amplifier. Yes, we don't rate amps on that one number but if SINAD is say, 60 dB, the rest are going to follow.

In many cases decision makers are not knowledgeable in these companies so proudly show me the gear, only then realizing they could have built something better, but didn't.

Note that my focus here is for electronics only. And I am thinking about the fewest key criteria that passes the "acceptable" mark from me, and by implication, from you all. They can do better of course to get higher praise but want to establish what we like to see.

An example for a dongle:

Output voltage: at least 2 volts on unbalanced, 4 volts on balanced.
SINAD: 100 dB or better, 1 kHz, 22.4 kHz bandwidth
SNR at 50mv: 85 dB (?)
SNR at full 2/4 volt output: 110 dB (?)
Output impedance < 1 ohm

Example for Amplifier:
SINAD >= 80 dB
SNR >= 110 dB (?)
Channel balance < 0.5 dB
Crosstalk > 70 dB @20 kHz


This would be presented as general guidelines for companies to adopt (or not). The point of this thread is not to discuss the specifics although you can, but determine if it is time for us to do this. Hate to have companies ready to produce performant products based on objective measurements but not know clearly what those measurements should be.

What say you?

It’s fair to publish minimum expectations.

a) What about Multitone? You know the whole argument, music is not a 1kHz testtone.

b) For Amps. What about powercube „stability“? And overly strong frequency dependent SINAD?

 

[wwenze]

Amp stability into different load types? We can't even decide on how we want to grade peak-powered products (transformer/PSU-limited but have lots of capacitor for temporary power) and thermally-limited products (sufficient electronics capacity but will overheat if subjected to test signal instead of music, see the 3eaudio PSU discussion)

 

[just1n]

So I keep running into companies who send me products with best intentions but miss key aspects that cause me to not recommend. And example is a very nice dongle I recently received that had balanced output but was limited to just 2 volts out. If you don't know, I like to see minimum of 4 volts out from such a port as otherwise, you can find unbalanced dongles at lower cost that do that.

Another example is channel balance issue with had with an AIYIMA amp where there was almost 1 dB differential.

Yet another is expected SINAD for an amplifier. Yes, we don't rate amps on that one number but if SINAD is say, 60 dB, the rest are going to follow.

In many cases decision makers are not knowledgeable in these companies so proudly show me the gear, only then realizing they could have built something better, but didn't.

Note that my focus here is for electronics only. And I am thinking about the fewest key criteria that passes the "acceptable" mark from me, and by implication, from you all. They can do better of course to get higher praise but want to establish what we like to see.

An example for a dongle:

Output voltage: at least 2 volts on unbalanced, 4 volts on balanced.
SINAD: 100 dB or better, 1 kHz, 22.4 kHz bandwidth
SNR at 50mv: 85 dB (?)
SNR at full 2/4 volt output: 110 dB (?)
Output impedance < 1 ohm

Example for Amplifier:
SINAD >= 80 dB
SNR >= 110 dB (?)
Channel balance < 0.5 dB
Crosstalk > 70 dB @20 kHz


This would be presented as general guidelines for companies to adopt (or not). The point of this thread is not to discuss the specifics although you can, but determine if it is time for us to do this. Hate to have companies ready to produce performant products based on objective measurements but not know clearly what those measurements should be.

What say you?

I’m curious what you think for the measurements for DACs, AVRs, etc would be?

Good, better, best? Tiers?

You’ve tested a remarkable number of devices and speakers that it is reasonable to base solely on similar criteria while presenting a value to consumers and manufacturers alike. I believe this community is more than a niche market. ASR is sometimes a first page Google Search result when combining a model and the word ‘review’.

 

[alex-z]

For amps, I think power delivery at 20Hz relative to 1kHz should have a metric, something like 5% deviation or less to be ideal. Bass has the highest power demand with real music and movies. There have been units like the Behringer NX3000D that have problems with that.

 

[GXAlan]

That is a related but different thing. A certification would be much more involved than this simple list of minimum performance criteria.

The problem is that I see some medical products incorrectly using the terms FDA registered, FDA cleared, and FDA approved.

If you create a list of minimum performance criteria, it can be misconstrued as ASR certified or ASR approved.

But it can be helpful to do two things.

1) My bias is to say that annually, we can publish median performance and median cost. So you have a metric that is evolving, cannot be misconstrued as certification, but still provides helpful info for everyone.

2) Establish minimum performance threshold, using a product instead of numbers. For the DAC, the Apple dongle is a great choice.

Any company knows that they need to beat Apple to be recommended and they know they should try to beat the median performance while coming in at a lower price. There is no confusion about a product being “approved” or “certified”

For amplifiers, it’s hard to establish a minimum spec but maybe you need a desktop and HT standard based upon minimum power?

 

[amirm]

’m curious what you think for the measurements for DACs, AVRs, etc would be?

Haven't thought through all the details for different device categories. Thinking out loud, it would be kind of like this:

DAC:
Output level: Minimum of 2 volts for unbalanced, 4 volts for balanced.
SINAD: >= 110 dB
DNR: >= 115 dB
Jitter (all inputs): < -115 dB (in FFT spectrum)
Filter attenuation 44.1 kHz sampling at 24 kHz: better than -100 dB
Frequency Response flatness: Less than 0.2 dB @ 20 kHz

Don't know yet how to include responses for the sweeps such as IMD. And for multitone, need to do some research to see what is a reasonable minimum.

AVRs:
Output level: Minimum of (undistorted) 2 volts for unbalanced, 4 volts for balanced.
DAC section:
SINAD: AVRs, better than 95 dB. AVPs, better than 100 dB
DNR: >= 115 dB
Jitter (all inputs): < -115 dB (in FFT spectrum)
Filter attenuation 44.1 kHz sampling at 24 kHz: better than -100 dB
Frequency Response flatness: Less than 0.2 dB @ 20 kHz

Amplifier Section:
SINAD with digital or analog input > 80 dB
SNR: ???

You get the idea.

 

[amirm]

If you create a list of minimum performance criteria, it can be misconstrued as ASR certified or ASR approved.

That's the risk so we need to weigh the benefit against issues like that.

 

[valerianf]

About the amp section, it would nice not to forgot the real load capability of the amp.
Do we need to recommend class AB Amps that are unable to drive 4 ohms?
Do we need to recommend class D Amps with a hiss or oscillations when driving a non resistive load?
For sure not.

 

[Chrispy]

If it is a list of standards only ASR agrees on, it would be construed as ASR certified/approved as much as the perception of the SINAD race I'd think. Does it matter even then? Someone's gotta do it?

 

[Alice of Old Vincennes]

I’m curious what you think for the measurements for DACs, AVRs, etc would be?

Good, better, best? Tiers?

You’ve tested a remarkable number of devices and speakers that it is reasonable to base solely on similar criteria while presenting a value to consumers and manufacturers alike. I believe this community is more than a niche market. ASR is sometimes a first page Google Search result when combining a model and the word ‘review’.

Correct. Often only place. A musing moment? Who cares. Amir has turned this industry upside down. Have you noticed? Google algorithms have.

 

[AdamG]

That's the risk so we need to weigh the benefit against issues like that.

How about a simple “Meets or exceeds minimum ASR performance thresholds” or “Does not meet minimum ASR performance thresholds”. No Pass or Fail language and no certification or approval endorsement. But the idea of publishing a set of test criteria for Manufacturers to assess if their product will meet the minimum standards might save them and you time and expense only to fail. Helping them understand what is expected can only benefit everyone involved.

It doesn’t have to be a perfect list and as we go along the standards can be updated and modified. Publishing the AP Project Files would demonstrate full transparency and generosity of ASR Spirit. Getting more Manufacturers involved in the process will only benefit both the OEMs and the Consumer. Not having a nebulous moving target can be a insurmountable exposure risk for the OEM and this project can mitigate their risk of bad press and negative test results.

Great idea Boss.

 

[Neale]

Example for Amplifier:
SINAD >= 80 dB
SNR >= 110 dB (?)
Channel balance < 0.5 dB
Crosstalk > 70 dB @20 kHz

... at what power?

 

[IAtaman]

Balanced automatically gives you double the output voltage. If it's below 4 V the circuit either has a problem, or unbalanced also is below 2 V - so requirement not met.

Or it is just a "balanced" connector and not a balanced output, which seems to be quite popular approach these days among dongle manufacturers.

 

[Fidji]

Haven't thought through all the details for different device categories. Thinking out loud, it would be kind of like this:

DAC:
Output level: Minimum of 2 volts for unbalanced, 4 volts for balanced.
SINAD: >= 110 dB
DNR: >= 115 dB
Jitter (all inputs): < -115 dB (in FFT spectrum)
Filter attenuation 44.1 kHz sampling at 24 kHz: better than -100 dB
Frequency Response flatness: Less than 0.2 dB @ 20 kHz

Don't know yet how to include responses for the sweeps such as IMD. And for multitone, need to do some research to see what is a reasonable minimum.

AVRs:
Output level: Minimum of (undistorted) 2 volts for unbalanced, 4 volts for balanced.
DAC section:
SINAD: AVRs, better than 95 dB. AVPs, better than 100 dB
DNR: >= 115 dB
Jitter (all inputs): < -115 dB (in FFT spectrum)
Filter attenuation 44.1 kHz sampling at 24 kHz: better than -100 dB
Frequency Response flatness: Less than 0.2 dB @ 20 kHz

Amplifier Section:
SINAD with digital or analog input > 80 dB
SNR: ???

You get the idea.

To close the circle - and move the industry ahead, I think we need to create "audibly transparent" set of thresholds, confirmed by controlled experiment.
Add some margin, e.g. 3dB SINAD on top and then do simple fail/pass evaluation. You can have another set "audibly transparent for Hi-Res" or smth similar for DAC's.

Once there is scientifically defined "good performance" producers can focus on the right things - better features, form factor, durability etc, instead of chasing 1dB SINAD more, in order to get to the top of the SINAD chart.

I personally fail to get excited by another DAC that is another 1dB better in SINAD, when it is obvious, that even the ones that are 20dB worse have enough fidelity to reproduce music 100% accurately.

 

[pma]

This would be presented as general guidelines for companies to adopt (or not). The point of this thread is not to discuss the specifics although you can, but determine if it is time for us to do this. Hate to have companies ready to produce performant products based on objective measurements but not know clearly what those measurements should be.

What say you?

Good for you that you know what is important and what is not, speaking about commercial audio electronics products.

 

[IAtaman]

Let me put on my consultant hat

There is a model called Kano model, named after Noriaki Kano, a Japanese educator who proposed it as a way to evaluate customer requirements in 1990's which has been simplified and adopted widely by many companies through their Lean or SixSigma programs.

In that model, there are 3 types of customer requirements; "must have" , "the more the better" and "delight factors".

To give a simple example, must have for a hotel would be hygiene, more the better would be price and a delight factor would be a box of chocolates on the bed when you check-in. And the point is, if the bed is not clean or if you are charging me $5000 a night for the room, the fact that you put a box of chocolates on the bed is not gonna impress me much.

With that in mind, in my opinion you should think about what are the "must haves" of each product category, and keep them simple and limit them to basic functionality and features such as output voltage, output impedance, channel balance etc and avoid creating standards for "the more the better" metrics such as SINAD and SNR. That way, you would not only create clear expectations for what a product must be able to do at minimum (which should evolve over time), you would also avoid potentially dividing discussions about what matters, what is audible etc and perceptions of "approved by ASR" and such.

My two cents.

Speaking of sending products for measurements, may I also suggest maybe instead of them sending you samples, you buy one yourself from a retailer and they compensate you for the invoice instead to avoid "golden samples"? I realize this might not work for "just launced" products maybe, but it might be possible to find a solution to that as well.