Audibility thresholds of amp and DAC measurements

[Jimster480]

Hidizs S8 multitone was one example I found a while back. But there's plenty. Almost every time Amir measures an affordable DAC/amp with multitone, the test tones come out with a visibly non-flat envelope, even at the bad zoom level he shows them (because he's only using them as noise tests), with spike heights that cross into potential audibility by having more than 0.1-0.2 dB of difference, yet this goes completely ignored in the results discussion.

In my limited explorations of this, the amplitude differences get even worse when you test with hundreds of tones and large FFT sizes. This requires some investigation, I only got to trying such measurements after hearing DAC/amp differences in (admittedly sighted) A/B comparisons. There could be something there, but I'm not seeing a lot of people discussing it to clarify, only to handwave it away.

This is interesting, do you have somewhere where you elaborate on this?

 

[donjoe0]

This is interesting, do you have somewhere where you elaborate on this?

Mods helped move this discussion to the Multitone thread (seems this forum doesn't leave any road-sign post behind in the old thread), and I got independent data there that strongly suggests the biggest defects I was seeing with multitone were from FFT vs. tone frequency interactions rather than the DAC/amp being evaluated.

 

[j_j]

Mods helped move this discussion to the Multitone thread (seems this forum doesn't leave any road-sign post behind in the old thread), and I got independent data there that strongly suggests the biggest defects I was seeing with multitone were from FFT vs. tone frequency interactions rather than the DAC/amp being evaluated.

If you look in the matlab thread, there's a way to design multitones so that as long as you maintain sample rate lock for DAC and ADC, and you use a specific analysis rate, you can avoid all that annoyance.

 

[pkane]

If you look in the matlab thread, there's a way to design multitones so that as long as you maintain sample rate lock for DAC and ADC, and you use a specific analysis rate, you can avoid all that annoyance.

The issue is not the multitone design, but the use of separate, unsynced clocks by the DAC and the ADC in the test set up resulting in clock drift. While it may look like the amplitude of the tones varies with frequency, Multitone Analyzer now computes the frequency response correctly, without these apparent variations that are due to spreading of a tone peak over multiple FFT bins.

 

[j_j]

The issue is not the multitone design, but the use of separate, unsynced clocks by the DAC and the ADC in the test set up resulting in clock drift. While it may look like the amplitude of the tones varies with frequency, Multitone Analyzer now computes the frequency response correctly, without these apparent variations that are due to spreading of a tone peak over multiple FFT bins.

Odd, I've gotten past this with a 2 channel in, 2 channel out USB system running at 96k, using the same clock. This seems an obvious approach, yes?

 

[pkane]

Odd, I've gotten past this with a 2 channel in, 2 channel out USB system running at 96k, using the same clock. This seems an obvious approach, yes?

It works, but not if you're using a separate DAC and ADC units that don't do a clock sync.

 

[j_j]

It works, but not if you're using a separate DAC and ADC units that don't do a clock sync.

Well, that's a mistake for starters!

 

[pma]

But all-in-one DAC/ADC interfaces like Focusrite 2i2 and similar have poor noise and distortion parameters to measure anything on SOTA electronics. The advantage of synchronous clock is neglected by high noise and distortion.

 

[j_j]

But all-in-one DAC/ADC interfaces like Focusrite 2i2 and similar have poor noise and distortion parameters to measure anything on SOTA electronics. The advantage of synchronous clock is neglected by high noise and distortion.

Well, I don't advertise equipment, but my loopback is quite clean. You better believe I check.

 

[Blumlein 88]

Well, I don't advertise equipment, but my loopback is quite clean. You better believe I check.

So is your interface similar to a Focusrite 2i2 or something of higher quality? I have a different Focusrite device which can be fed a clock signal or output one. Takes care of clock sync issues, but it isn't my best interface for noise or distortion.

 

[j_j]

So is your interface similar to a Focusrite 2i2 or something of higher quality? I have a different Focusrite device which can be fed a clock signal or output one. Takes care of clock sync issues, but it isn't my best interface for noise or distortion.

All I'm going to say is that running a loopback on what you have is the first step. Make sure you have good gain staging first, too.

 

[Jimster480]

Mods helped move this discussion to the Multitone thread (seems this forum doesn't leave any road-sign post behind in the old thread), and I got independent data there that strongly suggests the biggest defects I was seeing with multitone were from FFT vs. tone frequency interactions rather than the DAC/amp being evaluated.

Thanks for this, I went over there and read the thread!

 

[jkim]

Like with SINAD, it's not always possible to tell the distortion and noise apart in THD+N vs frequency plots, so we have to go with the lenient noise threshold again.

View attachment 25303

I believe this thread is the best place to ask this question. I'm having a discussion with an ASR member on the practice and interpretation of conventional single-tone THD+N vs frequency measurements. In my view, whenever we make and publish measurements, we should consider the audibility of any observed anomaly. We could measure any aspect of a device for some technical reason. But if any anomaly is due to an inaudible aspect---for instance, a noise-shaping-induced peak of -70 dBFS at 50 kHz---, the result would be misleading. Of course, people familiar with measurement procedures and settings will have no problem guessing / interpreting things behind each plot. But we should consider quite a few people can be easily misled by what is shown.

As for the best practice for measuring THD+N vs frequency, I believe Amir normally uses BW = 90 kHz. But when ultrasonic noise dominates THD+N and causes poor results, he measures the device again with BW = 45 kHz, like this:

But I do not think Amir chose the BW of 90 kHz or even 45 kHz because he believes distortion & noise > 20 kHz is audible. He must have chosen the BW because THD resulting from fundamentals in the treble region cannot be properly measured otherwise, i.e., without the wide BW including harmonic products > 20 kHz. Sure, really strong signals > 20 kHz could destroy a tweeter, but although it is possible due to strong fundamental tones, such damage is unlikely with harmonics or noise of a device unless there's some defect.

So, my question is, in making THD+N vs. frequency measurements of hi-fi audio devices, is it okay to use even narrower BW like 20 Hz to 20 kHz? Here, what I mean by BW is not simply half of DUT's Fs. Actual DUT Fs can be set to anything like 96 or 192 kHz. By using 20 kHz BW for THD+N calculation, THD+N in the treble region will not be technically accurate since it does not include higher-order harmonics above 20 kHz. Still, I think it can be a better representation of the device's intended function, i.e., hi-fi audio serving human hearing.

Here's an example:

Of course, I do NOT mean we should ignore any response above 20 kHz, since it can indicate some issue related to the device's audible aspects (e.g., IMD). I also do not suggest only one setting should always be used. Multiple settings should be used to check if there's any anomalous behavior. But if alternative results have been examined (and found ok) and only one plot is to be presented to readers for simplicity purpose, which should be the default one?

I vote for 20 kHz BW being used to produce THD+N vs. frequency measurements to be informative by covering cases in which a device's (inconsequential) ultrasonic response corrupts THD+N in the audible frequency range.

I wonder about ASR members' opinion on this topic.

EDIT. I once showed THD+N vs. frequency plots in which 20 kHz BW was used but the sweep was done only up to 10 kHz because the results above 10 kHz simply reflect noise levels with no harmonic components. But my current practice is to use 20 kHz BW and make measurements up to 20 kHz as well (even if results in the treble range do not properly reflect harmonics).

 

[IgorD]

Totally agree. THD+N should be measured exclusively within a bandwidth of 20 Hz to 20 kHz.

Amir has reported unusually poor THD+N results on devices using the CS43131 DAC, which employs noise shaping. He needed to clarify that these devices are essentially transparent (or very close to it), and that the results are simply a measurement artifact.

JCALLY JM20 Headphone Dongle Review

This is a review and detailed measurements of the JCALLY JM20 USB-C headphone adapter (DAC and Amp). It is on kind loan from a member and is on sale for US $24. Not much to say about the construction so let's get into measurements. Speaking of that, I finally have a fix for this class of...

www.audiosciencereview.com

SMSL DL100 DAC and Headphone Amp Review

This is a review, listening tests and detailed measurements of the SMSL DL100 balanced stereo DAC and headphone amplifier. It was sent to me by the company and costs US $179. The DL100 is a fair departure from standard SMSL offering in that it uses a stamped steel enclosure instead of machined...

www.audiosciencereview.com

Measuring THD+N using only the audible bandwidth was actually standard practice a long time ago. NwAvGuy's site:

ODAC Released

CAREFREE ENJOYMENT: I’ve been listening almost exclusively to the ODAC for several weeks now and I’ve been smiling a lot—not so much at the...

nwavguy.blogspot.com

THD+N VS FREQUENCY: Here’s the distortion performance at –1 dBFS from 20 hz to 20 Khz into a more challenging 10K load with a measurement bandwidth of 22 Khz. At 1 Khz the distortion is only 0.0027% and it remains around 0.003% over most of the audio band with only a slight rise up to 0.0048% at 9 Khz before the harmonics fall above the audible range. ...

(image)

 

[j_j]

Totally agree. THD+N should be measured exclusively within a bandwidth of 20 Hz to 20 kHz.

I agree in spirit, but what really should be PLOTTED in an image is a couple of things:

1) With a "buzz tone", show the original, output, and error spectra.
2) Show the error spectra with a line horizontal line showing the sum of all energy DC-22kHz. Units should be dB below input signal level.
3) Show the error spectra with another horizontal line showing the sum of all error energy up to 192kHz. Units dB below below input signal level.

I stop at 192khz for simplicity in capturing the error signal.

This would require at least 20 to 22 bit capture, so the ADC has to be really, really good.

 

[nick_l44.1]

THD+N should be measured exclusively within a bandwidth of 20 Hz to 20 kHz.

This cannot be done because the distortion from frequencies above 10 kHz does not fit within this band at all. In fact, this would be filtering out the measurement object. The band up to 20 kHz actually only allows adequate THD measurement up to ~1 kHz. It allows to capture up to 20 harmonics from 1 kHz. To measure the same 20 harmonics from 20 kHz, we would need a bandwidth up to 400 kHz...

Amir has reported unusually poor THD+N results on devices using the CS43131 DAC, which employs noise shaping. He needed to clarify that these devices are essentially transparent (or very close to it), and that the results are simply a measurement artifact.

Indeed. There is a dilemma here. In order to accurately measure distortion at high frequencies we need a wide measurement bandwidth, but at wide bandwidth the total noise of any DAC will be very high, constant for any measured frequency and will simply mask the results to its level. The CS43131 and CS43198 have very high total noise at wide bandwidth, and while the energy of this noise is high, its spectrum is high frequency and it is simply unbelievable that it can have an effect on THD+N@1kHz. So I don't see any other option than measuring THD [20..20000] over a wide bandwidth without considering the noise. Then for CS431** we can get some interesting results (at -12 dBFS):

High frequency noise can hardly be blamed for affecting low frequencies, but it is definitely the cause of distortion at frequencies from 3-4 kHz. The audible effect is a loss of transparency, but many people don't notice it or don't care.

 

[Hayabusa]

This cannot be done because the distortion from frequencies above 10 kHz does not fit within this band at all. In fact, this would be filtering out the measurement object. The band up to 20 kHz actually only allows adequate THD measurement up to ~1 kHz. It allows to capture up to 20 harmonics from 1 kHz. To measure the same 20 harmonics from 20 kHz, we would need a bandwidth up to 400 kHz...

Indeed. There is a dilemma here. In order to accurately measure distortion at high frequencies we need a wide measurement bandwidth, but at wide bandwidth the total noise of any DAC will be very high, constant for any measured frequency and will simply mask the results to its level. The CS43131 and CS43198 have very high total noise at wide bandwidth, and while the energy of this noise is high, its spectrum is high frequency and it is simply unbelievable that it can have an effect on THD+N@1kHz. So I don't see any other option than measuring THD [20..20000] over a wide bandwidth without considering the noise. Then for CS431** we can get some interesting results (at -12 dBFS):
View attachment 456530
High frequency noise can hardly be blamed for affecting low frequencies, but it is definitely the cause of distortion at frequencies from 3-4 kHz. The audible effect is a loss of transparency, but many people don't notice it or don't care.

maybe a plot that puts the distortion at the harmonic frequency in stead of the fundamental would give more insight here.

 

[nick_l44.1]

maybe a plot that puts the distortion at the harmonic frequency in stead of the fundamental would give more insight here.

Sorry, didn't catch what you're suggesting.

 

[Sokel]

This cannot be done because the distortion from frequencies above 10 kHz does not fit within this band at all. In fact, this would be filtering out the measurement object. The band up to 20 kHz actually only allows adequate THD measurement up to ~1 kHz. It allows to capture up to 20 harmonics from 1 kHz. To measure the same 20 harmonics from 20 kHz, we would need a bandwidth up to 400 kHz...

Indeed. There is a dilemma here. In order to accurately measure distortion at high frequencies we need a wide measurement bandwidth, but at wide bandwidth the total noise of any DAC will be very high, constant for any measured frequency and will simply mask the results to its level. The CS43131 and CS43198 have very high total noise at wide bandwidth, and while the energy of this noise is high, its spectrum is high frequency and it is simply unbelievable that it can have an effect on THD+N@1kHz. So I don't see any other option than measuring THD [20..20000] over a wide bandwidth without considering the noise. Then for CS431** we can get some interesting results (at -12 dBFS):
View attachment 456530
High frequency noise can hardly be blamed for affecting low frequencies, but it is definitely the cause of distortion at frequencies from 3-4 kHz. The audible effect is a loss of transparency, but many people don't notice it or don't care.

The plot above is not a nice sign of evolution after 20 years.
The below is poor EMU measuring itself at 192kHz I/O (so, good up to 96kHz) .
THD seems a lot lower up high.

and the last capture of the sweep who shows the shaping, etc :

 

[j_j]

This cannot be done because the distortion from frequencies above 10 kHz does not fit within this band at all. In fact, this would be filtering out the measurement object.

Of course,with any real signal, distortion over 10kHz with signals that have more than one frequency will show distortion different products at lower frequencies. Yes, that's IMD, not THD, but it doesn't matter, they are in a very real sense the same thing. They both result from the same nonlinearities in the device under test.

Why, pray tell, would I care of there was an 80dB down distortion product only at, say, 70kHz? That would not make any sense at all to be a "measurement object".

The band up to 20 kHz actually only allows adequate THD measurement up to ~1 kHz. It allows to capture up to 20 harmonics from 1 kHz. To measure the same 20 harmonics from 20 kHz, we would need a bandwidth up to 400 kHz...

You do understand that it's not only harmonics that are measured, yes? Unless distortions and noise above 20kHz (or maybe 25kHz just to be really, really safe) are powerful enough to cause equipment or drivers to do something nonlinear, what do you believe is the problem.

Now, of course, if you put in 19kHzand 20kHz and get the 1kHz error tone out, yes, that could be a problem. I don't think that's what you're saying, though. Still, if there were any really substantial distortions at 19-20kHz, you'd see that, and it would show up like obviously in measurements.

In the rest of the discussion it is unclear to me exactly what you are measuring.

As to "loss of transparency" I guess I'll have to ask how you demonstrate that.