The full-featured diffrograms in audio research

[Serge Smirnoff]

Hi everybody,

During the last year the df-metric got its further development not the least thanks to partnership between SoundExpert and Hypethesonics projects. We successfully assembled the measurement device, capable of measuring degradation of music signal down to -80dB of Difference level. The device is based on TI PCM4222 evaluation board and shows accurate and stable performance. To this moment we measured 23 portable audio players from different consumer segments - http://soundexpert.org/vault/dpa.html


Another progress was achieved in visualization of df-measurements. The DF parameter can be equally computed in both time and freq. domains. Using freq. domain allows to compute DF levels separately for signal magnitudes and phases and for different freq. sub-bands as well. The full-featured diffrograms computed this way show degradation of a signal in two dimentions - with time using a time window and with frequency using a set of freq. sub-bands. Below is an example of such diffrogram, which uses 400ms time window (=1px) and 1/12 octave bands (1px each). The DUT is Chord Hugo 2, the test track is A Day In The Life, The Beatles:

Hugo2-05.wav(192)_ref5.wav(44)_mono_400_Tm-89.3474(-76.8128)-21.6638_v3.3.png

Median df level (dB) for this DUT/track is (in brackets). We can see that the whole audible band is accurately and in full reproduced with some visible degradation only above 15kHz and for the fragments of the signal with low energy.

Degradation of the signal magnitudes looks slightly better:

Hugo2-05.wav(192)_ref5.wav(44)_mono_400_Mg-92.3221(-80.4647)-24.1157_v3.3.png

The degradation of the phases is much higher:

Hugo2-05.wav(192)_ref5.wav(44)_mono_400_Ph-56.9230(-23.4586)-4.5759_v3.3.png

The examples below will help to understand such diffrograms better.

DF levels and energy of a signal are coded according to this color map (algorithmically defined) :

Max.brightness of colors (white) corresponds to the power of full scale square wave; min.brightness (black) corresponds to -150dBFS.

All computations are performed in the region of interest: 20Hz - 20kHz regardless of a signal sample rate.

As the energy is indicated not per frequency but for freq. sub-bands, the diffrograms with white nose are slightly brighter towards the top:

These are magnitude, phase and time diffrograms computed for the signal of white noise (20Hz-20kHz, -10dBFS, 30s, 44.1k, 16bit). Grey color indicates DF = -Inf. Diffrograms with pink noise have equal brightness along vertical dimension.

The diffrograms above were computed with the same signal as reference and output. In other words both signals were perfectly time aligned and DF levels were computed as is. In practice the output signals usually have small time and pitch shifts which are corrected with a time-warping mechanism of the diffrogram33 function. Below are the same diffrograms computed with the time-warping:

These diffrograms show inaccuracy of the time-warping mechanism, which increases with frequency. Being the most complex signal, white noise forces the worst case scenario for the time warping, all other signals are corrected with higher accuracy; the latter can be increased at the expense of computation time. Time warping allows to compare signals at different sample rates and with any arbitrary time/pitch shifts. All examples with analogue signals in this presentation were time warped.

Let's return to the portable players and compare Hugo 2 with three other devices using diffrograms - Apple dongle (A1749), Questyle QP2R, HiBy FC3:

FC3-05.wav(192)_ref5.wav(44)_mono_400_Tm-86.3888(-81.0615)-20.6228_v3.3.png

Hugo2-05.wav(192)_ref5.wav(44)_mono_400_Tm-89.3474(-76.8128)-21.6638_v3.3.png

QP2R-05.wav(192)_ref5.wav(44)_mono_400_Tm-59.7192(-42.9632)-6.5048_v3.3.png

A1749-05.wav(192)_ref5.wav(44)_mono_400_Tm-51.1570(-38.3077)-8.7829_v3.3.png

Corresponding magnitude diffrograms:

FC3-05.wav(192)_ref5.wav(44)_mono_400_Mg-89.2874(-83.5085)-23.0726_v3.3.png

Hugo2-05.wav(192)_ref5.wav(44)_mono_400_Mg-92.3221(-80.4647)-24.1157_v3.3.png

QP2R-05.wav(192)_ref5.wav(44)_mono_400_Mg-74.0361(-56.3325)-10.7843_v3.3.png

A1749-05.wav(192)_ref5.wav(44)_mono_400_Mg-70.6177(-53.3802)-11.8761_v3.3.png

And their phase diffrograms:

FC3-05.wav(192)_ref5.wav(44)_mono_400_Ph-61.4849(-23.8133)-3.7251_v3.3.png

Hugo2-05.wav(192)_ref5.wav(44)_mono_400_Ph-56.9230(-23.4586)-4.5759_v3.3.png

QP2R-05.wav(192)_ref5.wav(44)_mono_400_Ph-15.3690(-4.2380)+0.1039_v3.3.png

A1749-05.wav(192)_ref5.wav(44)_mono_400_Ph-13.5322(-3.3230)-0.2530_v3.3.png

According to the above diffrograms Chord Hugo 2 and HiBy FC3 provide almost the same accuracy of this signal reproduction; QP2R and the dongle are far behind.

Resulting df-slides with full-featured diffrograms for tech. signals (1/6 octave bands) and with histogram of DF levels for two hours of music:

- on average, DF levels of t-signals are pretty consistent to DF levels with the music signal;
- t-signals degrade not accordingly; it depends on particular processing/circuitry used in playback device.

Such set of test signals (the music one in particular) and their DF levels give comprehensive picture of a device performance and its playback accuracy. Program simulation noise IEC 60268-1 can be used for express audio testing due to its good (actually - best) correlation with a music signal.

... to be continued below

 

[Serge Smirnoff]

Two additional examples of diffrogram usage.


1. Equalization. The example shows the difference between the three implementations of multi-band equalizer - in Adobe Audition (20 Bands, 2k, +3dB), Foobar2000 (18 Bands, 1.8k, +3dB) and Windows Media Player (10 Bands, 2k, +3dB):

aa2k+3dB.wav(44)_ref5.wav(44)_mono_400_Tm-57.2019(-31.5307)-19.0551_v3.3.png

fb1.8k+3dB.wav(44)_ref5.wav(44)_mono_400_Tm-54.1656(-29.2048)-18.3303_v3.3.png

wmp2k+3dB.wav(44)_ref5.wav(44)_mono_400_Tm-41.6046(-26.1875)-18.5527_v3.3.png

Their magnitude diffrograms:

aa2k+3dB.wav(44)_ref5.wav(44)_mono_400_Mg-64.3792(-31.2471)-18.6101_v3.3.png

fb1.8k+3dB.wav(44)_ref5.wav(44)_mono_400_Mg-63.3194(-28.9505)-17.7845_v3.3.png

wmp2k+3dB.wav(44)_ref5.wav(44)_mono_400_Mg-61.5150(-29.3715)-19.3083_v3.3.png

And their phase diffrograms:

aa2k+3dB.wav(44)_ref5.wav(44)_mono_400_Ph-57.7057(-21.1547)-9.4257_v3.3.png

fb1.8k+3dB.wav(44)_ref5.wav(44)_mono_400_Ph-58.1009(-20.1790)-8.8762_v3.3.png

wmp2k+3dB.wav(44)_ref5.wav(44)_mono_400_Ph-22.2988(-12.7592)-7.5660_v3.3.png

The EQ from WMP is less accurate and mostly because of its phase inaccuracy, especially around the region of equalization. Foobar2000 is on a par with the Audition.

2. MP3@320 cbr. This example shows how psychoacoustic bit reduction looks on diffrograms (timeDF, magnitudeDF, phaseDF):

mp320-05.wav(44)_ref5.wav(44)_mono_400_Tm-45.9059(-37.3180)-14.8931_v3.3.png

mp320-05.wav(44)_ref5.wav(44)_mono_400_Mg-49.0217(-39.9871)-17.1780_v3.3.png

mp320-05.wav(44)_ref5.wav(44)_mono_400_Ph-14.8832(-6.1369)-1.1948_v3.3.png

- the degradation almost evenly spans the whole freq. band and it is less around the freq. sub-bands where the most energy of the signal is concentrated;
- phase accuracy degrades foremost

The picture of the degradation is pretty consistent with the features of psychoacoustic encoding, which cuts out fragments of signal according to peculiarities of human hearing and not just according to frequency/amplitudes.

Also this example reminds that DF level is a pure quantitative parameter, not qualitative. And a meaningful comparison of DF levels for different DUTs is allowed only in case of similar type of degradation - similar artifact signatures.

For example, overall timeDF of mp3@320 (-37.3dB) is close to timeDF of Apple dongle (-38.3dB) but the inference about their equal perceived sound quality is incorrect. They have very different artifact signatures which is indirectly indicated by their different diffrograms - time, magnitude and phase ones.

On the contrary, all the diffrograms for Hugo 2 and FC3 are almost identical both by DF levels and appearance. In such a case we can safely conclude that perceived audio quality for these devices will be identical as well. It will be impossible to tell them apart in controlled blind listening tests.

Matlab code for computing these diffrograms (v3.3) - http://soundexpert.org/articles/-/blogs/visualization-of-distortion#part3
Pretty informative overview of the function and description of its arguments is inside the code.

Your feedback is welcome, thanks,
Serge.

 

[pkane]

Hi @Serge Smirnoff ,

Good to see you're making more progress, especially on building specialized hardware. The inclusion of phase difference plots is also a welcome addition, as these can make a large measurable difference but, often, not a large audible one. We discussed this last time, IIRC.

A question that was also asked before: how are your determining the audibility of DF? Do you have any new studies or results beyond surveying the casual internet reports by an uncontrolled group of audio users to demonstrate what levels (and types) of DF or diffograms can be audible?

Also this example reminds that DF level is a pure quantitative parameter, not qualitative. And a meaningful comparison of DF levels for different DUTs is allowed only in case of similar type of degradation - similar artifact signatures.

That has been my understanding, as well, since our last conversation. But, to make that last statement, I think it would help if you could define "similar artifact signatures" more precisely. How does one decide if these are similar, and which similarities and difference levels result in a similar audibility and which don't, and how does one determine this?

 

[Serge Smirnoff]

A question that was also asked before: how are your determining the audibility of DF? Do you have any new studies or results beyond surveying the casual internet reports by an uncontrolled group of audio users to demonstrate what levels (and types) of DF or diffograms can be audible?

Actually, no, I don't have the new studies; except maybe the new DF measurements of the players, which can give additional clue about DF audibility. But it looks like such studies are less and less necessary now. The threshold of DF audibility is much easier to define from the bottom. As I already explained [https://www.audiosciencereview.com/...-of-amp-and-dac-measurements.5734/post-302605] audibility of DF levels depend on the dynamic range of listening environment. Based on the absolute thresholds of human hearing [139dB] the absolute DF level is -142dB. More practical value is -99dB, which corresponds to max dynamic range of 16bit audio. Even more practical DF level is -90dB; which is usually showed when converting 32bit audio into 16bit (PSN signal) with or without dithering. BTW, here is how this operation with Triangle noise looks on diffrograms:

Assuming that at such low levels of degradation [DF = -90dB] the type of difference signal is not important and the difference signal itself tends to white/pink noise, I would suggest -90dB as a reliable/practical DF level for audio transparency. The ESS DAC from HiBy FC3 ($70) provides -76dB (PSN signal) - already pretty close to that theoretical value. Its diffrograms:

Most current audio equipment (including studio gear) has worse DF levels.

 

[Serge Smirnoff]

That has been my understanding, as well, since our last conversation. But, to make that last statement, I think it would help if you could define "similar artifact signatures" more precisely. How does one decide if these are similar, and which similarities and difference levels result in a similar audibility and which don't, and how does one determine this?

The concept of artifact signatures was explained here [https://www.audiosciencereview.com/...od-for-measuring-distortion.10282/post-284457]. The practical implementation is here - http://soundexpert.org/vault/dpa.html#artifact-signatures

The artifact signature is a vector of DF values measured with some long music signal (2 hours in our case). If another DUT shows the same/similar vector of DF levels, their artifact signatures are similar. The measure of similarity is based on Mean Absolute Error (dB). Usually the distance between DF vectors of 1-2dB is close enough to say that they are similar. This point is still a bit moot for me and requires a lot of listening tests' results to be researched. But within 1-2dB I'm pretty sure about similarity.

 

[ElNino]

I don't understand how this is at all useful. Minor differences in digital filters can cause huge differences in the "DF levels" that are unrelated to either harmonic distortion or IMD. (In fact, judging from your results, digital filters are probably the dominant factor in determining the "DF level", in the absence of EQ or compression.)

If you could come up with a similar metric that is insensitive to linear distortion, maybe that would be useful as a means of communicating differences to people who don't understand other measurements.

 

[dc655321]

Usually the distance between DF vectors of 1-2dB is close enough to say that they are similar.

Do you mean to say angles between vectors as a measure of similarity, and not distances (i.e. acos(dot(AB) / AB) )?

Also, how are you constructing the vector of DF values and why does it take 2hrs to generate enough info?

 

[PierreV]

Hmmm, impressive work, even if I am not sure what to make of it.

on the minus side

- if the results and their meanings can't be explained in a few clear unambiguous sentences, I don't see them gaining any traction, ever, with the typical audiophile.

- it seems to suffer from the same fundamental flaw as many other scoring systems. Specifically, it is based on hypotheses, builds a complex system out of them and, displays nice complex charts before actually heading into audibility territory. I would love the measurement/model building approaches to work in the opposite direction: namely, convincingly prove some audibility at the first step then and only then try to explain/model it.

For example, begin by a large blind test that proves most users can hear a difference between the 'awful' Apple dongle and the HiBy FC3, then attempt to explain it.

To put things plainly, if there is such a big metric scoring difference, the difference should be obvious. If it isn't obvious, any metric showing a huge difference simply isn't even relevant to audibility imho.

on the plus side

- when anyone tells you "You should have bought a 9$ Apple dongle!" you'll be able to answer "You must be kidding! Have you seen that DF score?"

I don't mean to be harsh. It is very easy to get lost in some form of numerology and I am guilty of that fascination with measurements/numbers myself in other areas. But one should stay somewhat grounded in practical reality imho.

 

[pkane]

The concept of artifact signatures was explained here [https://www.audiosciencereview.com/...od-for-measuring-distortion.10282/post-284457]. The practical implementation is here - http://soundexpert.org/vault/dpa.html#artifact-signatures

The artifact signature is a vector of DF values measured with some long music signal (2 hours in our case). If another DUT shows the same/similar vector of DF levels, their artifact signatures are similar. The measure of similarity is based on Mean Absolute Error (dB). Usually the distance between DF vectors of 1-2dB is close enough to say that they are similar. This point is still a bit moot for me and requires a lot of listening tests' results to be researched. But within 1-2dB I'm pretty sure about similarity.

You are measuring a mathematical distance between two waveforms, rather than the distance in the audibility space, and then pronouncing them to sound the same, if they are within 1-2dB of each other.

As I see it, DF is a correlation-based metric that isn't clearly associated with any known measures of audibility (the -90dB DF level is also not based on any formal research, as you state). But you then generalize from this metric to conclude that two devices are audibly(?) similar if their DF vectors over a 2 hour music recording are within 1-2 dB of each other. That's a jump to a conclusion from a previously unproven premise. Either I'm missing something, or there are many claims and assumptions here without much evidence to support them.

I'm all for creating a better metric, but there must be at least some evidence to demonstrate that it really is better.

 

[Serge Smirnoff]

I don't understand how this is at all useful. Minor differences in digital filters can cause huge differences in the "DF levels" that are unrelated to either harmonic distortion or IMD. (In fact, judging from your results, digital filters are probably the dominant factor in determining the "DF level", in the absence of EQ or compression.)

If you could come up with a similar metric that is insensitive to linear distortion, maybe that would be useful as a means of communicating differences to people who don't understand other measurements.

The main usefulness of df-metric is possibility to define such level of difference that will be imperceptible by 95% of listeners (or 99.(9)%) with any music. Audio manufacturers can achieve any required DF value today, not a rocket-science anymore. No need to sort distortions according to audibility, just eliminate them all (or most of them). Cheap and perfectly transparent audio path is the goal.

 

[Serge Smirnoff]

Do you mean to say angles between vectors as a measure of similarity, and not distances (i.e. acos(dot(AB) / AB) )?

I mean distances, not angles. Out of curiosity I tried almost all popular measures of distance and ended up with the simplest and most easily interpretative - Mean Absolute Error in dB. The most vivid separation of DUT clusters is achieved with Correlation Distance (Székely) but its interpretation in case of artifact signatures is too complicated, I failed.

 

[Serge Smirnoff]

Also, how are you constructing the vector of DF values and why does it take 2hrs to generate enough info?

The basics of df-metric is in the post [https://www.audiosciencereview.com/...native-method-for-measuring-distortion.10282/]. DF is the level of difference signal; it can also be computed through correlation coeff. Before computing an output waveform is time warped in order to get the lowest (best) DF value. The time warping with high precision is computationally intensive procedure. If an output signal is known to be perfectly time aligned, the the time warping can be disabled ("NoWarp" option). For example quantization (with/without dithering) is time accurate, ADC/DAC conversions are not.

 

[Serge Smirnoff]

- if the results and their meanings can't be explained in a few clear unambiguous sentences, I don't see them gaining any traction, ever, with the typical audiophile.

I think the opening text on the page with df-measurements of portable players is intended exactly to this purpose:

The audio measurements below illustrate how accurately the various players reproduce the initial/recorded waveform. These devices are tested with both real music and technical signals. Waveform degradation is measured with the Difference level parameter, Df (in decibels). This shows how significantly the final studio mix is altered as a result of the playback device. A low Df level (blue area) implies less deterioration of the sound.

 

[Serge Smirnoff]

- it seems to suffer from the same fundamental flaw as many other scoring systems. Specifically, it is based on hypotheses, builds a complex system out of them and, displays nice complex charts before actually heading into audibility territory. I would love the measurement/model building approaches to work in the opposite direction: namely, convincingly prove some audibility at the first step then and only then try to explain/model it.

If I have possibility to organize a fully controlled blind listening test, I would start exactly in this order. But in reality I have to rely on the assumptions like "Hugo 2 has higher perceived sound quality than Apple dongle". Number of tested devices in such a case could also increase credibility of the measurement method.

 

[Serge Smirnoff]

You are measuring a mathematical distance between two waveforms, rather than the distance in the audibility space, and then pronouncing them to sound the same, if they are within 1-2dB of each other.

As I see it, DF is a correlation-based metric that isn't clearly associated with any known measures of audibility (the -90dB DF level is also not based on any formal research, as you state). But you then generalize from this metric to conclude that two devices are audibly(?) similar if their DF vectors over a 2 hour music recording are within 1-2 dB of each other. That's a jump to a conclusion from a previously unproven premise. Either I'm missing something, or there are many claims and assumptions here without much evidence to support them.

In case of artifact signatures I measure the distance between vectors of DF levels, not between waveforms. In other words, I compare the types of degradation of the the same music signal by various DUTs. This distance has relation to audibility; yes, I examined only 5 cases of HA listening tests in order to research this relation and I'm not ready to reliably quantify it but at the moment I'm sure that the relation exists and can be used in extreme cases when the distance is very high (~>10dB) or low (~<1-2dB). The latter cases often indicate copy-paste designs of DUTs: [

https://twitter.com/i/web/status/1406241599059906560

], [

https://twitter.com/i/web/status/1406925750901559296

]


The DF level of -90dB is remarkable. It helps to define some anchor point on subjective quality scale. Most operations of quantization from 32/24 to 16bit with or without dithering end up with this DF level. Thus, the types of degradation of sound at this level vary to the same extent as various types of dithering affect the sound during this operation (assuming the same level of quantization noise or DF). If such quantization is considered safe for preserving sound quality then Df = -90dB is a good candidate for the point of the audio transparency.

 

[pkane]

In case of artifact signatures I measure the distance between vectors of DF levels, not between waveforms. In other words, I compare the types of degradation of the the same music signal by various DUTs. This distance has relation to audibility; yes, I examined only 5 cases of HA listening tests in order to research this relation and I'm not ready to reliably quantify it but at the moment I'm sure that the relation exists and can be used in extreme cases when the distance is very high (~>10dB) or low (~<1-2dB). The latter cases often indicate copy-paste designs of DUTs: [

https://twitter.com/i/web/status/1406241599059906560

], [

https://twitter.com/i/web/status/1406925750901559296

]


The DF level of -90dB is remarkable. It helps to define some anchor point on subjective quality scale. Most operations of quantization from 32/24 to 16bit with or without dithering end up with this DF level. Thus, the types of degradation of sound at this level vary to the same extent as various types of dithering affect the sound during this operation (assuming the same level of quantization noise or DF). If such quantization is considered safe for preserving sound quality then Df = -90dB is a good candidate for the point of the audio transparency.

The distance between DF levels is still a distance between waveforms. You're moving up one level of abstraction, but at the base level, the metric is still just a mathematical distance. Unless you can demonstrate that DF metric itself can predict audibility of the difference between two waveforms, any conclusions about their similarity based on comparing multiple DF metrics is unproven, and likely not valid.

Serge, I'm not trying to give you a hard time. I want to help. But, if you can't convince ASR membership that DF is a useful metric, you'll have a much harder time outside of ASR, IMHO. Especially where measurements and metrics are often ignored and not taken seriously.

 

[Serge Smirnoff]

The distance between DF levels is still a distance between waveforms. You're moving up one level of abstraction, but at the base level, the metric is still just a mathematical distance. Unless you can demonstrate that DF metric itself can predict audibility of the difference between two waveforms, any conclusions about their similarity based on comparing multiple DF metrics is unproven, and likely not valid.

At least, the tweets above show that DF metric easily detects very similar designs, which will sound exactly the same (not accounting the output impedance issue). The only question is to find the distance when the similarity stops working and the sound becomes too different for reliable comparison by DF levels.

At small DF levels the question of relation to subjective perception of quality also becomes less important because at DF=-90dB the statistical characteristics of the difference signal are more-or-less Gaussian and the diff. signal itself is on the edge of human perception/sensitivity. Being the most aggregated audio parameter [it accounts all possible/meaningful degradations of a signal], DF level guarantees that below some low level of difference any quality issues are imperceptible and the DUT/audiopath is transparent. None of the traditional audio parameters can provide such guaranties because each of them examines only some special aspect of the overall audio performance and DF accounts them all.

 

[Serge Smirnoff]

Serge, I'm not trying to give you a hard time. I want to help. But, if you can't convince ASR membership that DF is a useful metric, you'll have a much harder time outside of ASR, IMHO. Especially where measurements and metrics are often ignored and not taken seriously.

Thanks, Paul for the help. Hardly I can convince the whole ASR membership that DF is a useful metric. Maybe just a few (one, actually is enough). But I promise to try hard until the success )). At least I will continue to inform this community about the progress in DF metric development/application; those who are interested can follow and support.

 

[ElNino]

The main usefulness of df-metric is possibility to define such level of difference that will be imperceptible by 95% of listeners (or 99.(9)%) with any music. Audio manufacturers can achieve any required DF value today, not a rocket-science anymore. No need to sort distortions according to audibility, just eliminate them all (or most of them). Cheap and perfectly transparent audio path is the goal.

This goal can be achieved just as easily with other, standard measures of linear and nonlinear distortion.

 

[Serge Smirnoff]

This goal can be achieved just as easily with other, standard measures of linear and nonlinear distortion.

You might be right; then, great! We'll move to the goal from “opposite” directions. My confidence in adoption of “overkill” accuracy for analog audio is based partly on the lossless case in digital audio. While psychoacoustic encoding can provide perceptually transparent storage/distribution of a master record, the industry and listeners prefer to use lossless formats if possible, spending extra storage space and communication bandwidth. Similarly it will be very hard to resist using transparent/lossless analog path, the more so as the latter can be designed/manufactured cheap. Those "other standard measures of linear and nonlinear distortion" look like an intellectual overkill for the purpose of audio reproduction. The transparency can be defined and provided in a simpler (engineering) way.