Why SINAD is not important (article)

[Mad_Economist]

The issue was/is that old analogue THD meters had no way of separating the THD and the noise from the residual after the notched out fundamental. So much of the single 'THD' figures in the past, were really THD+N figures. Now of course, it's trivial with computer based FFT to measure the harmonics alone.

I wanted to include a bit about this in the article, actually! It ended up getting edited down to the point of unrecognizability, but initially I was talking about notch filters and Wien bridge oscillators.

We're blessed to live in an era where such refined signal generation and analysis can be done so easily.

One can only assume that is why many other measurements are conducted and posted in ASR measurement reviews.

Would that assumption hold, however? What measurements, bar frequency response, are we typically inferring sound quality from?

I think it suits the purpose of a metric for ranking products tested in a graph format.

Let me be clear, I'm not averse to measurements being done which have poor or no correlation with audibility in and of itself, but why are we ranking audio devices using that metric?

 

[PierreV]

THD(+N) has been expressed either as a percentage or x dB down forever. My vintage THD meters express S/N in dB and the distortion ranges/meter scales are marked in both dB and percentage.

Yes, I am a bit confused about what AP invented here...

1985

 

[amirm]

Yes, I am a bit confused about what AP invented here...

1985
View attachment 148855

They "invented" the notion of SINAD being same as THD+N as used in audio. Prior to this, SINAD was used in RF world and was not a metric that was used in Audio. So the measurement is not the invention. The invention was use of the term SINAD in this context.

Just a few weeks ago I had the CTO of an audio company writing to me complaining that I was using an ancient RF term -- SINAD -- for audio when no one does this. I had to explain to him that AP has made this synonymous with THD+N so the measurement is the standard and one that has been used for decades.

 

[Mad_Economist]

SINAD being same as THD+N as used in audio.

Putting aside evidence that Signal/N+D terms appear to exist in audio technology dating earlier, given that SINAD in your usage is identical to THD+N, would you please frame the century-old comment in the context of "a calculation of the ratio of signal to the sum of noise and distortion, which has historically been referred to as THD+N, and presently is also referred to as SINAD"? In that context, do you have an issue with characterizing signal/N+D or N+D/signal terms as a century old?

 

[restorer-john]

Putting aside evidence that Signal/N+D terms appear to exist in audio technology dating earlier, given that SINAD in your usage is identical to THD+N, would you please frame the century-old comment in the context of "a calculation of the ratio of signal to the sum of noise and distortion, which has historically been referred to as THD+N, and presently is also referred to as SINAD"? In that context, do you have an issue with characterizing signal/N+D or N+D/signal terms as a century old?

I think semantics are getting in the way of what you and @Amir are trying to discuss. Amir has hung his hat on SINAD and that's fine. Personally, I'd like to see at least residual noise added to enable more information to be gleaned. But if he adds (as he often does) S/N or Dynamic Range plots, we've pretty much got what we want, more or less. Wouldn't you agree?

He could add a giant process report dump to the reviews but it would turn off 99% of the readers. It's a balancing act and you cannot please all of the people, all of the time.

 

[Mad_Economist]

I think semantics are getting in the way of what you and @Amir are trying to discuss. Amir has hung his hat on SINAD and that's fine. Personally, I'd like to see at least residual noise added to enable more information to be gleaned. But if he adds (as he often does) S/N or Dynamic Range plots, we've pretty much got what we want, more or less. Wouldn't you agree?

He could add a giant process report dump to the reviews but it would turn off 99% of the readers. It's a balancing act and you cannot please all of the people, all of the time.

I would agree that semantics are getting in the way - I'm less copacetic with hanging a hat on SINAD or THD+N, though. I agree that the SNR and DR plots full the role I'd like on the noise side, but I'm really not a fan of front-and-center-ing unweighted distortion sums, no matter how you do it. I nattered at RTings about this in 2017 with their headphone reviews, I nattered at Amir about it on Reddit long ago, and I'll keep on squeaking about it, because the lit is conclusive: it's not a good metric for evaluating audio performance.

If anything, I'd actually prefer that THD and all its derivatives be removed, if a more perceptually relevant alternative isn't possible or acceptable, just leaving the FFTs - I don't think that doing the "dump the whole APX report" approach helps readers any, but I also am not a big fan of giving easy numbers that don't lead people towards sound quality.

 

[JSmith]

why are we ranking audio devices using that metric?

You seem to be missing the point somewhat though... there is one signal being analysed and displayed using different metrics by the analyser. One broad metric that is positive and uses whole numbers is easier to use when comparing the signal analysis for each product over a certain threshold of same.

What metric would you like to see used, or what new metric would you like to create and based on what aspects of the signal?



JSmith

 

[amirm]

Putting aside evidence that Signal/N+D terms appear to exist in audio technology dating earlier, given that SINAD in your usage is identical to THD+N, would you please frame the century-old comment in the context of "a calculation of the ratio of signal to the sum of noise and distortion, which has historically been referred to as THD+N, and presently is also referred to as SINAD"? In that context, do you have an issue with characterizing signal/N+D or N+D/signal terms as a century old?

I think the whole schtick needs to go. The notion that it is "old" seems to be used as a lay argument that because it is old, we should not use it. This has nothing to do with anything. We have a lot of old laws in physics. Do we put them down because they are old?

If you want to use it anyway, then the article should be about THD+N. The fact that it is called SINAD is a modern adaptation of the term/value and nothing more.

 

[Mad_Economist]

I think the whole schtick needs to go. The notion that it is "old" seems to be used as a lay argument that because it is old, we should not use it. This has nothing to do with anything. We have a lot of old laws in physics. Do we put them down because they are old?

If you want to use it anyway, then the article should be about THD+N. The fact that it is called SINAD is a modern adaptation of the term/value and nothing more.

The "old" bit mattered before my section on the history of summed noise and distortion measures died in editing Specifically, the fact that once upon a time a twin-t was your best friend on metrology is the only reason that it ever made sense to use THD+N, nobody would intuitively claim that all spuria have precisely equal audibility, and one of the cites - Shorter 1950 - was specifically proposing an analog-implementable frequency weighting for greater perceptual relevancy in the pre-computer era.

When you have work failing to find consistent thresholds for (in)audibility for a measure of nonlinearity going back to the 30s - which is an inevitability of its construction, THD just can't predict audibility because the ear is neither flat in frequency response nor equally discriminating regardless of proximity to the signal - and continuing to this day, it should be pretty clear that it's faulty. The inclusion of its age was more to explain why a faulty metric ever came to be used at all.

 

[amirm]

He credits him in the very first minute of the video and also posts links to all his references in the video description.... (at about 35 seconds in...)

So? That doesn't address the point I made that he is talking as if he knows/owns this material. He doesn't. He is reading the article and as such, he needs to be clear that this is not his subject matter expertise. It is like a news report talking about some study. The anchor doesn't pretend to be a peer of the people doing said study. They simply say, "there is a new therapy for cancer published in the Journal of XYZ."

Furthermore, that sort of video goes completely against some of the other stuff he has written such as DACs and headphone amps having "house sounds" and such. If he has gotten religion around this, then he needs to come out and say all of his prior assertions would be readily dismissible if one follows the research. To the extent he is not doing this, then it looks like a cheap shot against measurements and objective analysis.

 

[Mad_Economist]

You seem to be missing the point somewhat though... there is one signal being analysed and displayed using different metrics by the analyser. One broad metric that is positive and uses whole numbers is easier to use when comparing the signal analysis for each product over a certain threshold of same.

It's easy from a comparison standpoint, but also doesn't predict subjective audibility - I really have to emphasize how uniform this finding is in every single study of distortion audibility I've been able to find, the summed form is not a consistent differentiator of what we hear from a nonlinearity standpoint.

What metric would you like to see used, or what new metric would you like to create and based on what aspects of the signal?

As referenced here to John, I'd actually be fairly happy with simply removing the SINAD/THD+N dashboard entirely, leaving only the FFT. Failing that, I'd probably start somewhere like Brunet or Pahomov, but I suppose if someone wanted to start something either from scratch (as GedLee) or from PEAQ (as Brunet and Pahomov), it wouldn't be that time consuming to do a lit review. The subjective validation part would kind of suck, but ASR definitely has enough forum volunteers to make it viable if that was the route someone desired to go.

 

[amirm]

When you have work failing to find consistent thresholds for (in)audibility for a measure of nonlinearity going back to the 30s - which is an inevitability of its construction, THD just can't predict audibility because the ear is neither flat in frequency response nor equally discriminating regardless of proximity to the signal - and continuing to this day, it should be pretty clear that it's faulty. The inclusion of its age was more to explain why a faulty metric ever came to be used at all.

So that led you to say it is useless? You rather plow ahead in the dark with no metric of non-linearity and noise?

It is this kind of sensationalism which threw out any merit the rest of the article had. Andrew makes it even worse in his video.

In measurements, we are getting clues to the performance of a device and how well it aspires to produce a transparent high-quality experience. It is never a case where we get a measurement that comes out and tells us exactly what we want to know. We use a combination of measurements to build confidence in what we are testing. To run off and say THD+N has no value is just wrong. It opens the door to any piece of garbage claiming to be transparent now. I am able with the measurements I make to give a high confidence answer as to whether a device is transparent or not. If you think that is wrong, then you need to present your own set of measurements. If you can't, then you are just creating FUD here, helping companies that product poorly designed gear.

 

[amirm]

As referenced here to John, I'd actually be fairly happy with simply removing the SINAD/THD+N dashboard entirely, leaving only the FFT.

That is nonsensical. What the heck would you do with that graph? How would you memorize what is in it?

And why is it necessary when we can achieve and prove transparency using THD+N?

 

[Mad_Economist]

That doesn't address the point I made that he is talking as if he knows/owns this material. He doesn't. He is reading the article and as such, he needs to be clear that this is not his subject matter expertise.

To be clear, while I was the "subject matter expert", this article and video was actually a collaboration with Andrew, it isn't just something I handed him. We've spent quite a few hours discussing these topics, and while I wouldn't say he's eminently knowledgeable about the lit, he's both willing to take the time to read it and able to process it, and that is all one needs to engage with science.

So that led you to say it is useless?

No, the lack of correlation between THD scores and subjective results across over half a dozen separate listening tests, spread across multiple decades, which represent only a small sample of the body of work in this area did.

You rather plow ahead in the dark with no metric of non-linearity and noise?

No, I'm quite happy with your use of SINAD if you literally just dropped the distortion term; @restorer-john is correct, the SNR usage is not problematic in and of itself when it is noise-dominated, although I'd prefer a normal SNR dashboard instead.

I would specifically like you to stop using a dark age measurement like THD however, yes. As far as options for a superior weighting for nonlinearities, I'd be game to discuss that if you'd like (and have alluded to where I'd start, e.g. with PEAQ and its derivatives), although I will note that my personal stance matches John Vanderkooy's: Present-day electronic systems are sufficiently linear, and a window in the form of an FFT of either a single or multi-tone device that actually pushes past potential nonlinearity audibility will make that glaringly obvious without any form of scoring.

Edit:

And why is it necessary when we can achieve and prove transparency using THD+N?

To reduce type I errors.

 

[DonH56]

SINAD is used in the IEEE Standards for DACs and ADCs, transient recorders, and numerous other places including RF/mW standards. For data converters, SINAD is used to define ENOB (effective number of bits), which is a general figure of merit. AFAIK the definition in Wikipedia is wrong, or at least outdated, as SINAD is the ratio is of signal (power, voltage, current, whatever) to noise+distortion. That is, noise+distortion does not appear in the numerator of the SINAD definition I used throughout my career (for data converters or RF systems).

SNR and THD are defined independently with SNR excluding harmonic distortion and THD excluding noise except for what is in the bins (and there are ways of averaging that out though it is rarely worth it).

I'm too old to change and not doing data converter design these days so will stay out of this particular debate.

 

[amirm]

No, the lack of correlation between THD scores and subjective results across over half a dozen separate listening tests, spread across multiple decades, which represent only a small sample of the body of work in this area did.

You looked at the wrong studies. Acoustic analysis papers all use untrained listeners and generally are not subject matter experts for non linearities in general.

 

[Mad_Economist]

You looked at the wrong studies. Acoustic analysis papers all use untrained listeners and generally are not subject matter experts for non linearities in general.

I am willing to revise my analysis - can you point me to a body of lit which reflects a consistent correlation between THD and sound quality/audibility? Edit: Separately, I remain extremely interested in any research on altering the level of the masking effect, as you alluded to before - that would change quite a lot, so I'd be very appreciative of something that can give me some context on the order of effect we'd be talking about.

 

[amirm]

I am willing to revise my analysis - can you point me to a body of lit which reflects a consistent correlation between THD and sound quality/audibility?

What are you talking about? I didn't tell you THD is a predictor of audibility. I have shown you in the clear where I said such is not. What I said is that your evidence that you say points to it being "useless" is incorrect because they did not use trained listeners in this specific field. Study the research for audio coding where trained listeners are used and you can see the power they bring to bear in hearing non-linear distortions. For example, trained listener's ability to hear through forward masking is some 40 times better than untrained listeners. Backward masking is even more impressive in the way I can hear pre-echo that so many people can't.

So to be clear, you created your own headline and then went to war against it. This is not proper and is going to create misinformation and license for companies to produce garbage and no one measuring them to show otherwise. If you want this to be your mission, go ahead. I am done engaging in the topic when the good intentions were the first thing that was thrown out.

 

[Mad_Economist]

Study the research for audio coding where trained listeners are used and you can see the power they bring to bear in hearing non-linear distortions.

Before you quit this topic, I legitimately am very interested in any citations - or even researcher or institution names - to look for in this regard. I've tried to make a fairly thorough study of the AES's work on distortion, as well as a body of research from some European universities, but I don't claim to know every paper ever published, and I'm always happy to read a new one.

For example, trained listener's ability to hear through forward masking is some 40 times better than untrained listeners. Backward masking is even more impressive in the way I can hear pre-echo that so many people can't.

That's a very interesting result - did the work it comes from include work on frequency, in addition to temporal masking?

So to be clear, you created your own headline and then go to war against it.

For the record, I didn't write the headline - my original draft just said "SINAD is bad", which isn't a very catchy title, but is a headline I would fight in support of. "Not important" I would be less keen to endorse because of the cases where SINAD is commensurate to SNR, which is inarguably useful.

Edit: Also, it's a bit vexing to be accused of bad faith both while I'm actively defending this community (and yourself) in the (predictably vile) Youtube comments on the video, and while you appear to be trying to "gotcha" me about proving a negative. I'm earnestly open to revising or retracting the article (and strongly urging Resolve, who I believe will listen to me, to do the same with the video) in response to any new citations regarding distortion audibility. I want this to be a useful resource to people, and I'd hate to spread misinformation, so anything you can point me to is helpful.

 

[Chocomel]

SINAD/THD+N and THD are poor quality metrics, that's just a fact, look at all the science/papers outlining their issues and poor correlation with sound quality. Are they better than nothing, yes. But we have way more than nothing thanks to amir and others doing good measurements of equipment. Considering the issues with the aforementioned metrics and the quality of the measurements we have nowadays the oversimplification is misleading, rather than revealing of the actual product performance. And if there's want for a single number metric there are, again scientifically proven, better metrics available.

It seems some people here are more concerned with the title and (one of) the writers than the content of the article, let alone the numerous citations. The article isn't some hit piece on Amir or measurements as a whole, it's a want for better metrics.