Standards for Audio Electronics Measurements?

[Ricardus]

This is nothing new, Amir has stated for quite some time now that when a company send him a product, he does them the courtesy of giving them the choice to not publish the review if it's not what they expected. Personally I agree, and hopefully the company will take some steps to remedy the situation.

I didn't know that either. Hmmmm...

 

[peniku8]

An example for a dongle:

Output voltage: at least 2 volts on unbalanced, 4 volts on balanced.
SINAD: 100 dB or better, 1 kHz, 22.4 kHz bandwidth
SNR at 50mv: 85 dB (?)
SNR at full 2/4 volt output: 110 dB (?)
Output impedance < 1 ohm

Example for Amplifier:
SINAD >= 80 dB
SNR >= 110 dB (?)
Channel balance < 0.5 dB
Crosstalk > 70 dB @20 kHz

About SINAD I have a personal train of thought:
Distortion audibility is best when

• testing with pure sine waves
• the order of the harmonic is very high, which minimizes the chance of the distortion being masked by the fundamental
• the distortion product(s) land(s) in the region where our hearing is most sensitive
• the level is sufficiently high that the distortion product also reaches a meaningful level
• the level is NOT so high that equal loudness contours take "flatten" our perception, which means the relative difference in level between a low frequency fundamental and a high order distortion becomes small again

Looking at the distortion audibility level graphs here (you need to pay to view all), the absolute worst case of the shown examples and orders comes down to 12th order harmonic at a 300Hz fundamental (3400Hz harmonic) at 100-110dB and falls to 0.03%, which equates to 70dBr.
Based on that information, that means a device with more than 70dB SINAD will be audibly transparent at all times.
For reference, I encourage anyone who reads this to take the Klippel listening test for distortion audibility.
Now, several devices with similar distortion in series will quickly worsen that figure. But realistically how many devices will someone have in series? DAC, EQ, Preamp, Amp? +3dB on an uncorrelated addition on each, which would net us 79dB of SINAD in this case. I don't see any benefit in a device having a SINAD better than 80dB based on these conclusions, as long as DNR is on point.

About SNR@50mv I think a simple noise level in volts would be a bit more convenient to work with. Just take the sensitivity of the IEM, multiply it by the figure and see what level the idle noise will reach with the IEM in question. But SNR@50mv is fine too, I guess. Just a bit more abstract for absolute comparison, but it works perfectly well for relative comparison.

Channel balance sounds fine, I wouldn't want a device with 1dB channel imbalance.

Output impedance I don't know. I'd say take the lowest impedance headphones/IEMs with the highest reactance (impedance swing) and calculate the impact of amplifier output impedance and make sure that the impact stays below 0.5dB? Somebody else can do the maths here, maybe @staticV3 or @oratory1990 feels like doing that

SNR I think is the main thing for me when it comes to DACs. If it's sufficiently high, you don't have to worry about gain staging, running lots of negative preamp gain or having a long chain of devices.. 110dB sounds like good to me.

Crosstalk highly depends if you're listening on sterophonically (speakers) or binaurally (headphones/IEMs). The more crosstalk, the more 'Mono' the signal becomes. 20dB crosstalk is audible but didn't bother me in my tests, 30dB I found inaudible (on speakers). Idk about headphones, I did not test that so I can't really comment any further on that, but 70dB sounds excessively high to me, if we're talking about audibility only. Maybe 50dB? It's not a tough bar to clear, but why cast off a device that is otherwise flawless if it has a crosstalk of 60dB?

 

[Galliardist]

In an ideal world, I'd agree with you. If what people are buying the most are soundbars and TWS earphones, a consumer audio review site should review those as well. But that requires a team, well defined protocols some of which needs to be invented on the go and would probably be not very "data" backed, and most importantly, it probably requires a business model different than voluntary donations and members sending in products for reviews. Applying science to audio reviews is not in the monopoly of ASR. There are quite a few other review sites that do that as well. I think trying to expand ASR to cover more of the market than it currently does will not work in the way which ASR is structured today.

On the guidelines, I do agree. I understand what Amir wants to do is to provide manufactureres with some basic guidelines so that they don't end up sending ****** products to him, wasting his time reviewing products that he can not publish because of bad performance. But, the consequence of that would be creating a new set of standards. I shared my view on this earlier on the thread, I think guidelines would improve his time efficiency, however they should be limited to basics feautres and functionalities, among other things, to allow futher exploration of interesting devices or engineering excellence.

Or maybe manufacturers should be clear whether they are sending in the product for measurement or for review, and be prepared to hear what Amir has to say if it is a review.

My point is that other reviewers and the forum can discuss these other matters and it would still fit within the remit of ASR. People here still discuss other reviewers' work on a regular basis. I don't expect Amir, say, to suddenly turn his attention to soundbars or really old historical products.

It seems that whatever guidelines are given to manufacturers would be open to question on their part. The response "did you blind test our product before deciding that it didn;'t meet a real standard for audibile difference" is one possibility. We can see from this thread that there are many different answers to the question "what is audible?". So for example the Naim Uniti Atom, which just fails our audibility article's lenient standard is used by members here with a claim that it's fine: do we know that it is audibly different/worse in practice?

 

[tmtomh]

No not at all. The change the market wants to see is a form-factor change. If the audiophile manufacturers (objectivist or subjectivist) don’t want to adapt, great news for those who are. It means little competition in the actual market the current industry currently ignores.

I don't think anyone is disputing the importance of user experience and form factor. Both aspects of music listening have already changed significantly over the past 5-20 years and no doubt will continue to do so.

And no one disputes that the audiophile market - that is, consumers primarily or at least significantly concerned with sound quality - is a minority of the larger audio market.

But a minority or niché market is still a large, viable market in today's world, across most sectors of the economy and consumer goods. So if a site like this can have a positive impact - which by all indications it has - on the engineering quality and prioritization of sound quality in the audiophile market, and if that impact has some spillover effect, however large or small, beyond self-proclaimed audiophile gear, then that's an unambiguously good thing.

There need not be - and isn't - a zero-sum relation between ergonomics/form factor on the one hand, and good sound on the other.

 

[staticV3]

Output impedance I don't know. I'd say take the lowest impedance headphones/IEMs with the highest reactance (impedance swing) and calculate the impact of amplifier output impedance and make sure that the impact stays below 0.5dB? Somebody else can do the maths here, maybe @staticV3 or @oratory1990 feels like doing that

I chose 21 IEMs from @csglinux's database with especially low impedance and high variability and created this spreadsheet:

Output Impedance Frequency Response Deviation

Results Output Impedance [Ω]:,50

docs.google.com

To use it, login with your Google account and click on File->Make a copy:

Then in that copy, you can change the output impedance and the graph will update the frequency response impact for each IEM:


You can click on the graph, then hover over the lines to highlight one of them:


The speadsheet assumes flat output impedance response

 

[peniku8]

I chose 21 IEMs from @csglinux's database with especially low impedance and high variability and created this spreadsheet:

Output Impedance Frequency Response Deviation

Results Output Impedance [Ω]:,50

docs.google.com

To use it, login with your Google account and click on File->Make a copy:
View attachment 337133

Then in that copy, you can change the output impedance and the graph will update the frequency response impact for each IEM:
View attachment 337135

You can click on the graph, then hover over the lines to highlight one of them:
View attachment 337136

The speadsheet assumes flat output impedance response

This is rad! Looks like 0.5Ohm keeps the variability within ± 0.5dB for all in the selection bar the "M3".
That sounds like a reasonable thing to aim for, then.

Looking back at my comments on SINAD, @Mad_Economist pointed me at an AES paper by Louis Fielder (9841), which somewhat contradicts the data/charts I have been working with. I'll give that paper a read and see if I can come up with a new "SINAD requirement for best-case distortion audibility".

 

[French Meloman]

Je pense que vous devrez peut-être considérer les amplis à lampes comme un cas particulier. De toute évidence, ils ne vont pas franchir l’obstacle des 80 dB SINAD en raison de l’objectif exprès de leur conception. Compte tenu de cela, la question est alors de savoir s’il existe un niveau de SINAD en dessous duquel même un dispositif conçu pour produire des distorsions euphoniques ne devrait plus être considéré comme performant ?

Translated by moderator. Please post in English moving forward.

I think that indeed tube amps, no matter how well designed, will probably never be able to achieve a SINAD of 80 dB due to the fact that tubes are inherently non-linear. On the other hand, a performance target of 50 dB could be a minimum performance criterion below which the product could be considered defective. And also assign a target of at least 60 dB of SINAD for a tube amp to be considered reasonably effective. This can be a technological challenge for amateur or professional electronics engineers.



From a more general point of view, I think that performance objectives should be modulated depending on the type of technology. In addition, certain types of defects could be considered major non-conformities, such as a difference in performance of several dB between the 2 channels, or significant instabilities or degradation of performance when the devices heat up.



It should also be kept in mind that the majority of content is natively 16-bit, which means that transparency > 96 dB should not provide any significant benefit to listening. In this regard, the multi-tone test seems to me to be the most revealing about the real transparency of a device. Additionally, I'm not sure the speakers, regardless of their performance, are capable of reproducing HD content properly.

This, of course, is just my opinion.

 

[Fleuch]

My job is to judge performance. With no goalposts, this industry has sunk to as low as they can possibly get. We have been able, through numerous reviews, managed to move this titanic of a ship to correct its heading a few degrees. Guidelines for performance may act as an auto-pilot for the industry to correct itself. And give targets to engineers and marketing people to follow when they have none currently. Once the whole industry is engineering proper products, then you can sort them based on other factors. Right now you have all of those problems and not knowing how something performs.

The influence ASR is having (present tense so important) on the industry is demonstrated by the increasing number of performance graphs and charts that accompany advertising of a product new to the market, but sadly only rarely from established and more traditional "names". The extreme case is where Fosi appears to be extracting a large number of ASR posts and copying to its own web pages. There is a difference between using the test results of a product sent to ASR, when there is interaction between Amir and the manufacturer, and using copies of posts to a forum thread, where the former is based on repeatable measurements and the latter is usually the opinion of an individual, similar to subjective reviews on YouTube or in print. Careful thought has to be given to how an "ASR seal of approval" is currently used and how future use can be influenced. There is also the very delicate questions of the trust placed in the test equipment itself and the conditions under which the ests are carried out, similar to how industrial instrumentation is calibrated. As important is this leads to how testing is supervised, whether results from testing by a manufacturer are as acceptable as those carried out by an independent organisation such as ASR, even if the same test equipment is used.

There are a number of instances where hallowed names in the industry have used performance specifications that are completely indequate; for example showing a splendidly flat frequency response between -3dB points complete with an exceptional SNR, but not really showing much of what happens between those 3dB points. ASR is shining a light on the real world performance of equipment over the actual bandwidth of the product. The information required by a customer in a dealer's shop may be no more than a visible "seal of approval" from a recognised authority, moving through to technical data that allows product comparison before visiting a dealer, up to the engineering data required by design engineers. Perhaps the objective is to provide the data that is sufficient for the greatest number of people who are guided by technical data. Personally my requirement is that I want to know that what is inside the delivery box actually performs in the way that the manufacturer claims it does, and possibly a little more.

With this in mind, I completely agree with Amir that we need, as a community of interest, to" know how something performs". The schedule of minimum performance criteria must not be exhaustive but it must cover the "basic necessities"; too many criteria and the objective of providing accessible information is lost.