Alternative method for measuring distortion

[Serge Smirnoff]

Continuing to test with real music recordings. I ran a few tests against recorded DAC/ADC loop files from the collection posted here. This is with non-linear amplitude/phase corrections enabled.

Thanks, will check them.

I think we should start with simple examples - one-channel test signals that do not require time correction. Here is one of them: program simulation noise encoded/decoded by aptxHD. Both non-warped and warped df measurements are equal for this signal (https://www.dropbox.com/s/9huswm66dnqt4hz/se_psn.rar?dl=0):

non-warped

-51.5337-50.4587-49.3000

warped

-51.5358-50.4590-49.3036

What I find is that the median DF value is well correlated with the overall RMS of the difference (shown on the bottom of each screen capture), so I'm not sure the metric is sufficiently different from a simple rms delta value.

As @Blumlein 88 correctly noted df level is RMS level of the difference vs RMS level of the original.

 

[pkane]

I've lost the story line here.

What are the graphs above displaying?

Sorry, my fault for dominating this topic. The graphs I posted are a test run computing DF metric using DeltaWave. I thought it might make it easier to evaluate the metric in DW, since it's already doing most of the work needed. The median value is the interesting part, the rest of the plot is the error distribution over the rest of the file.

 

[Serge Smirnoff]

Interesting approach, especially regarding the significant differences with white noise. Looks to me like a circuit or chip design issue (what do they do differently inside, or is it a measurement issue after all?).

This is how reality looks like )) Various signals distorted to different extent, including m-signal. Testing with white noise is sensitive to all kind of distortions. This is the worst case testing for any DUT. The meanings of different signals are here - http://soundexpert.org/portable-players#howtoreaddfslides

Measurement procedure is correct. It is easy to prove measuring identical white noise samples.

 

[Serge Smirnoff]

Sorry, my fault for dominating this topic.

We can create a new thread for the issue or even communicate privately if the latter is of low interest for the people (I prefer a new thread; up to you).

 

[bogart]

We can create a new thread for the issue or even communicate privately if the latter is of low interest for the people (I prefer a new thread; up to you).

For my part, it's interesting to watch the collaboration, although a new thread will help keep discussions of Df a bit more accessible.

 

[Blumlein 88]

We can create a new thread for the issue or even communicate privately if the latter is of low interest for the people (I prefer a new thread; up to you).

I also vote for keep it here. Though I might not be posting much about it, I'm reading all this with interest.

 

[rajapruk]

Very interesting Serge. This is kind of what I have been missing, and have been thinking of myself lately!

Can you see any correlation between Df and traditional steadystate measurements like THD+N, or have you seen a device measure bad Df but good traditional measurements? (Sorry if this has been already discussed, I have not had the time to read the whole thread yet)

 

[RayDunzl]

I'm not "getting" it...

Assumptions:

A digital file source is compared to a digital result captured by ADC.

The sampled digits won't be the same, even if the analog waveform is "perfectly" reproduced.

So, ???

What's being compared?

(excuse my density if this is all obvious and/or previously explained)

 

[Serge Smirnoff]

Can you see any correlation between Df and traditional steadystate measurements like THD+N, or have you seen a device measure bad Df but good traditional measurements? (Sorry if this has been already discussed, I have not had the time to read the whole thread yet)

According to:

- these early df-measurements - http://soundexpert.org/portable-players-beta
- these ones with different m-signal - http://soundexpert.org/portable-players
- and these ones measured with higher quality recorder - http://soundexpert.org/articles/-/blogs/audio-quality-of-high-end-portable-players

the dependency between df-levels with m-signal and THD+N (df level with Sine) can be safely considered as non-existent.

 

[Serge Smirnoff]

A digital file source is compared to a digital result captured by ADC.

The sampled digits won't be the same, even if the analog waveform is "perfectly" reproduced.

So, ???

What's being compared?

Short answer: in reality the errors caused by "won't be the same" digits are much much much lower than error caused by imperfection of waveforms.

 

[eliash]

This is how reality looks like )) Various signals distorted to different extent, including m-signal. Testing with white noise is sensitive to all kind of distortions. This is the worst case testing for any DUT. The meanings of different signals are here - http://soundexpert.org/portable-players#howtoreaddfslides

Measurement procedure is correct. It is easy to prove measuring identical white noise samples.

Thanks for commenting.
Thinking further why white noise causes so much variation, I thought about phase shifts in a DUT, these might be regarded as relatively harmless from audible perspective, but could generate large amplitude differences, when comparing both signals in a broadband measurement.
- Is there any measure embedded in the test to cope with them?

 

[Serge Smirnoff]

Thinking further why white noise causes so much variation, I thought about phase shifts in a DUT, these might be regarded as relatively harmless from audible perspective, but could generate large amplitude differences, when comparing both signals in a broadband measurement.
- Is there any measure embedded in the test to cope with them?

Correct. Low df levels with white noise are due to phase inaccuracy in DUTs. And they are really "relatively harmless from audible perspective". And comparison of df-measurements with m-signal (histogram) to df-measurements with white noise confirms this - some DUTs showing low (good) df levels with m-signal have awful df measurements with white noise (Apple devices in particular). So, there is no necessity to "cope" with this. In df-metric perceived audio quality is not measured with tech signals. They are for developers/manufacturers of audio equipment as they help to understand what can be improved in circuitry design.

All this misunderstanding originates from the popular false assumption - if Sine signal has lower distortion then all other signals will also have lower distortion. Df measurements clearly show that this is not true. This misconception is the main reason why traditional audio measurements correlate badly to perceived quality estimations.

 

[Serge Smirnoff]

And here is the last concept that I need to explain in regards to df-measurements.

The main beauty of df-metric is possibility to find/define the special point on df scale, which corresponds to some df level with real music signal. Achieving this low level of difference any audio device disappears from audio path and it's presence can not be discovered by means of listening tests. This is audio singularity level (s-level). At this level the amount of all possible distortions is so small that discovering/researching them makes no sense. Psychoacoustics does not work below this point. Thus the problem of delivering high quality audio can be turned into pure engineering task - to provide required level of accuracy for real music signal. And you need to control at consuming side only one easily measurable parameter - df level with that signal (or program simulation noise in many cases). All other audio measurements loose their power completely below s-level.

This concept has the reasoning of a higher ground. Music is created by author on production side. Warmth, harshness, openness, brightness, scene depth ... are the author's instruments. Playback devices are just a communication channel aimed to deliver all those characteristics of sound untouched. Having transparent audio path the hard work of both artists/musicians and audio engineers/producers will be clearly audible. Nothing prevents though to insert into this transparent audio path some DSP simulating all your favorite tube/vinyl distortions and performing thousand other processings for creative listening. But all this is only in addition to transparent path, on top of it.

Current level of technology in audio industry is more than sufficient for producing cheap consumer audio devices operating below the audio singularity level. Median of histogram on df-slides shows how far from this point a DUT is.

 

[eliash]

Correct. Low df levels with white noise are due to phase inaccuracy in DUTs. And they are really "relatively harmless from audible perspective". And comparison of df-measurements with m-signal (histogram) to df-measurements with white noise confirms this - some DUTs showing low (good) df levels with m-signal have awful df measurements with white noise (Apple devices in particular). So, there is no necessity to "cope" with this. In df-metric perceived audio quality is not measured with tech signals. They are for developers/manufacturers of audio equipment as they help to understand what can be improved in circuitry design.

All this misunderstanding originates from the popular false assumption - if Sine signal has lower distortion then all other signals will also have lower distortion. Df measurements clearly show that this is not true. This misconception is the main reason why traditional audio measurements correlate badly to perceived quality estimations.

One way to cope with phase shifts could be to split the measurement channels into "Critical Band" or "ERB" related frequency slices, which would reduce the phase influence to a minimum and allow for a later summation of the level (or better power) deviations. Also it relates better to actual human aural reception. The measurement results should be lower than for the broadband method above, but result in a better qualitative distinction, no matter if m- or t- signals used...

 

[rajapruk]

All this misunderstanding originates from the popular false assumption - if Sine signal has lower distortion then all other signals will also have lower distortion. Df measurements clearly show that this is not true. This misconception is the main reason why traditional audio measurements correlate badly to perceived quality estimations.

Any comment from @amirm on this?

 

[pkane]

And here is the last concept that I need to explain in regards to df-measurements.

The main beauty of df-metric is possibility to find/define the special point on df scale, which corresponds to some df level with real music signal. Achieving this low level of difference any audio device disappears from audio path and it's presence can not be discovered by means of listening tests. This is audio singularity level (s-level). At this level the amount of all possible distortions is so small that discovering/researching them makes no sense. Psychoacoustics does not work below this point. Thus the problem of delivering high quality audio can be turned into pure engineering task - to provide required level of accuracy for real music signal. And you need to control at consuming side only one easily measurable parameter - df level with that signal (or program simulation noise in many cases). All other audio measurements loose their power completely below s-level.

This concept has the reasoning of a higher ground. Music is created by author on production side. Warmth, harshness, openness, brightness, scene depth ... are the author's instruments. Playback devices are just a communication channel aimed to deliver all those characteristics of sound untouched. Having transparent audio path the hard work of both artists/musicians and audio engineers/producers will be clearly audible. Nothing prevents though to insert into this transparent audio path some DSP simulating all your favorite tube/vinyl distortions and performing thousand other processings for creative listening. But all this is only in addition to transparent path, on top of it.

Current level of technology in audio industry is more than sufficient for producing cheap consumer audio devices operating below the audio singularity level. Median of histogram on df-slides shows how far from this point a DUT is.

Hi Serge,

The promise of a simple metric like df is indeed enticing. In my mind there are two questions that I'd like to see answered:

1. Demonstrate that df metric is indeed correlated with perceived audio quality (I think you had some studies referenced on your site -- can you please link them and describe in more detail?) Is there a sufficient evidence that df is better correlated than, say THD or THD+N or other common metrics?

2. Is there a sufficient difference between the df metric and the RMS null difference as computed by DeltaWave (and, similar software like AudioDiffMaker)?

For #1, I can certainly add 'df' as a measurement to DeltaWave and make it available to let others run their own tests, once we determine that it's being computed accurately. But more important in my mind are the actual controlled studies that show real correlation. THD+N is indeed a poor metric for audibility of distortions, I've proven this to myself and others have as well. It seems THD+N at the levels measured using modern DACs and preamps, and even amps is low enough to not really be audible. Maybe other than for some specialized, single-ended triode or guitar amplifiers and similar devices where distortion is there by design.

For #2 it looks to me, after running only only a few tests that RMS difference, a global metric, is very closely correlated with df. The question is which is more accurate and better correlated with audibility. I see why df might be a better metric, but this needs to be properly tested to confirm since RMS difference has been measured and available for quite a while, as in the example of the DAC/ADC loop measurements on gearslutz website I linked previously.

 

[eliash]

I don´t want to comment on the above question to Amir but one remark regarding the issue with the sine wave measurement, which is well known.
That´s why the parallel measurement with 32 (or so) sine wave frequencies in parallel (as displayed in the DAC tests) has some charme for me, allowing to look at least deep into the interspaces and see what appearance any intermodulation might have...of course no ear related m-signal, but certainly a better t-signal...

 

[pkane]

I don´t want to comment on the above question but one remark regarding the issue with the sine wave measurement, which is well known.
That´s why the parallel measurement with 32 (or so) sine wave frequencies in parallel (as displayed in the DAC tests) has some charme for me, allowing to look at least deep into the interspaces and see what appearance any intermodulation might have...of course no ear related m-signal, but a better t-signal...

Simple sine wave is easier to generate and to measure than a complex signal, it isn't necessarily better. But there needs to be a better reason to state this than just the belief that complex signal somehow exercises the DUT differently than a simple sine wave. I've certainly been a believer in this and DeltaWave is a prime example of using complex music files for distortion analysis. But it's certainly not proof

 

[boXem]

I am supposed to be in stealth mode, but this is too interesting.

There are anyhow some points that a pure technician like me is not sure to follow:
1. as already mentioned, the correlation between df and perceived audio quality is not clearly explained, at least in this thread. I have to admit I didn't dig in further.
2. metric is based on a measurement device. How is this device assessed itself? What would be the equivalent of the loopback we are used to? How to make the metric universal if measurements device impacts it?
3. measurements themselves: the examples shown are recorded at 96 kHz. This could mean that more than half the analyzed bandwidth is outside the audible band. This brings back to point 1.
4. the very bad result that all DUTs have with square waves makes me wonder if the bandwidth limitation + DC removal is also applied to the reference signal. If not, this can be quite misleading.

@Serge Smirnoff and @pkane , I am sure these points have already been addressed, could you please clarify?

 

[pkane]

I don´t want to comment on the above question to Amir but one remark regarding the issue with the sine wave measurement, which is well known.
That´s why the parallel measurement with 32 (or so) sine wave frequencies in parallel (as displayed in the DAC tests) has some charme for me, allowing to look at least deep into the interspaces and see what appearance any intermodulation might have...of course no ear related m-signal, but certainly a better t-signal...

Here's a comparison between a sine wave with some distortion applied to it (harmonic and jitter) and below is the same distortion applied to a 32 tone signal. Which is better?