Beta Test: Multitone Loopback Analyzer software

[danadam]

First, not sure what you mean about a 'non-flat envelope'.

AFAIU he means the 1 pixel difference in the level of some tones in the multi-tone graph in Hidiz s8 review:

 

[j_j]

Dithering problems in the graphics, I bet. It would be enlightening to know.

 

[donjoe0]

I have no reason to believe that's a dithering problem, as I can find other examples where a dedicated FR plot was also provided, and those multitone spike heights correspond to the rolloff shown on the FR plot. Plus we would see it in a lot more places, like the text labels, the characters wouldn't all be the same height etc. My bet is that's a real difference in the measured data, not a dithering problem. Shouldn't be too hard to believe, as with the given resolution a 1 pixel difference hovers right around 0.2-0.3 dB, near the threshold of audibility. The more interesting ones are where it exceeds that, and especially that other case where I'm getting much larger wobbles in the FR with pkane's multitone tool, when using the 300-400 tone settings. The Hidizs S8 was just one I could find quickly to show it's at least questionable that all these affordable DAC/amps "measure perfectly", if one keeps in mind the known thresholds of FR defect audibility as quoted in this thread, and if one pays attention to all the available data, not just to what's familiar and convenient.

Since he needs to attenuate the tones, they might end up too low and enter a non-linear range, which would end up affecting the resulting response, is this right?

I mean even if that were the case, that would still reflect very poorly on the FiiO KA17 - what respectable modern DAC/amp is already showing easily audible nonlinearity (0.8 dB) at just -35 dB output? For now I suspect it's something else, could be the way the tones are interacting with the sampling rate, FFT windows, something like what j_j said. It just seemed odd to me in the first round of comparisons why it would look so bad for the KA17 but stay well outside of audibility for the HiBy FC3, which also correlated to the subjective quality differences between them in my listening comparison. Both tested with the same hardware and software.

 

[Sokel]

I have no reason to believe that's a dithering problem, as I can find other examples where a dedicated FR plot was also provided, and those multitone spike heights correspond to the rolloff shown on the FR plot. Plus we would see it in a lot more places, like the text labels, the characters wouldn't all be the same height etc. My bet is that's a real difference in the measured data, not a dithering problem. Shouldn't be too hard to believe, as with the given resolution a 1 pixel difference hovers right around 0.2-0.3 dB, near the threshold of audibility. The more interesting ones are where it exceeds that, and especially that other case where I'm getting much larger wobbles in the FR with pkane's multitone tool, when using the 300-400 tone settings. The Hidizs S8 was just one I could find quickly to show it's at least questionable that all these affordable DAC/amps "measure perfectly", if one keeps in mind the known thresholds of FR defect audibility as quoted in this thread, and if one pays attention to all the available data, not just to what's familiar and convenient.


I mean even if that were the case, that would still reflect very poorly on the FiiO KA17 - what respectable modern DAC/amp is already showing easily audible nonlinearity (0.8 dB) at just -35 dB output? For now I suspect it's something else, could be the way the tones are interacting with the sampling rate, FFT windows, something like what j_j said. It just seemed odd to me in the first round of comparisons why it would look so bad for the KA17 but stay well outside of audibility for the HiBy FC3, which also correlated to the subjective quality differences between them in my listening comparison. Both tested with the same hardware and software.

Did a little sanity check here about the FR obtained from multitone signals, I used two DAC.
(all test conditions visible, check it out)

First a loopback with E-MU 0204 doing a multitone400 :



normal as usual,below is the obtained FR:



Looks normal to me (note the range) and the same FR obtained with a Log-Chirp:




Nothing unusual

Same stuff with Khadas TB:




Yes,it does a little worst than the 20yo interface but still normal as viewed at this plot, but... :



Looks familiar with the debated plots, doesn't it? Super weird! Now let's do chirps:



Totally normal,unlike the above.

Definitely the measurement method along with the timing issues of the chain (loopback had no issue) the way I see it.

 

[DonH56]

<elided> a copy of base Matlab plus Signal Processing package. (Yeah, that costs a (*(&*(ing mint.)

They do have a "home use" plan now that is much cheaper, though still not cheap. I think it costs me about $300 a year for the updates and the original cost some years back was about that. Still expensive for just hobby use. Octave is free, works pretty well, and is mostly compatible (until it ain't).

Edit: $149 for just Matlab for Home, plus nominal fee per package. I have a number of packages that double the total. I have a ton of Mathcad files I need to print and save as they abandoned the home market (home version very crippled) and since PTC acquired them the price has skyrocketed.

https://www.mathworks.com/pricing-licensing.html?prodcode=ML&intendeduse=home

 

[j_j]

I have no reason to believe that's a dithering problem, as I can find other examples where a dedicated FR plot was also provided, and those multitone spike heights correspond to the rolloff shown on the FR plot. Plus we would see it in a lot more places, like the text labels, the characters wouldn't all be the same height etc.

Bluntly put, if the PIXEL height in the input data is just right, that's exactly what you'll see.

Looking at the actual numbers represented by the graph is the only way to know for sure. When you reduce height to 'n' pixels, after all, what happens to ones that are one LSB above and below?

So you don't know, and neither do I, until we have the data.

 

[j_j]

They do have a "home use" plan now that is much cheaper, though still not cheap. I think it costs me about $300 a year for the updates and the original cost some years back was about that. Still expensive for just hobby use. Octave is free, works pretty well, and is mostly compatible (until it ain't).

Edit: $149 for just Matlab for Home, plus nominal fee per package. I have a number of packages that double the total. I have a ton of Mathcad files I need to print and save as they abandoned the home market (home version very crippled) and since PTC acquired them the price has skyrocketed.

https://www.mathworks.com/pricing-licensing.html?prodcode=ML&intendeduse=home

Check in the general audio forum,I started the thread. Amir may change the name of it. Makes a probe tone (digital domain) that can be used to accurately deconvolve a 10 second signal into an impulse, and does so for both frequency response and impulse response. Demo on digitally filtered drylab signal included. I did not include the probe signal as a wav file, because I guess I can't do that. Also, it would be good to include the .m file as a file but not that, either.

 

[donjoe0]

Did a little sanity check here about the FR obtained from multitone signals
[...]
Definitely the measurement method along with the timing issues of the chain (loopback had no issue) the way I see it

Superb, thank you for this! Most helpful information to clarify my results so far: simply trying the same test on another setup.
So what I'm seeing with this data added on:

* Indeed it looks like the FR curve will always be tens of dB down from 0 when testing with hundreds of tones, as pushing any higher volume would mean clipping

* The 400-tone result does still correlate to DAC/amp quality as revealed by other measurements, it just seems to exaggerate how bad the not-great DACs are vs. the really great ones (unless... the 400-tone does match some results that would be seen with music better than sweep tests, which can only be clarified by controlled listening comparisons)

* The Khadas Tone Board joins the ranks of the many many DACs I personally can't accept to see called "audibly transparent", since even with the log-chirp it shows roll-off at both ends, roughly 0.2 dB tilt over both the highest and the lowest octave, which according to numbers quoted in this thread are FR defects within the realm of human audibility. There must be trained listeners out there who on a good day, with just the right songs, can ABX-differentiate between the Khadas and a more competently made DAC. That is not what "audibly transparent" or "measures as transparent" means to me.

So returning to my case of the FiiO KA17 vs. the HiBy FC3, the original question that pushed me down this rabbit hole remains: have *I* heard these differences, or was it lack of level matching, placebo etc.? I think I need to ABX with music recorded through both devices. Just need to make sure I can record something as close as possible to what I hear when plugging headphones into them directly. The troubling part of that is I know my consumer-grade capture equipment only has audio inputs with multi kOhm impedances whereas in the listening test I used 34-ohm Hifimans. Can't be completely sure I'm getting the exact same signal when recording like this vs. what I heard in headphones, and would not be able to completely trust the ABX result in this case. Is there any Average Joe accessible equipment that can get around that?

 

[Sokel]

Superb, thank you for this! Most helpful information to clarify my results so far: simply trying the same test on another setup.
So what I'm seeing with this data added on:

* Indeed it looks like the FR curve will always be tens of dB down from 0 when testing with hundreds of tones, as pushing any higher volume would mean clipping

* The 400-tone result does still correlate to DAC/amp quality as revealed by other measurements, it just seems to exaggerate how bad the not-great DACs are vs. the really great ones (unless... the 400-tone does match some results that would be seen with music better than sweep tests, which can only be clarified by controlled listening comparisons)

* The Khadas Tone Board joins the ranks of the many many DACs I personally can't accept to see called "audibly transparent", since even with the log-chirp it shows roll-off at both ends, roughly 0.2 dB tilt over both the highest and the lowest octave, which according to numbers quoted in this thread are FR defects within the realm of human audibility. There must be trained listeners out there who on a good day, with just the right songs, can ABX-differentiate between the Khadas and a more competently made DAC. That is not what "audibly transparent" or "measures as transparent" means to me.

So returning to my case of the FiiO KA17 vs. the HiBy FC3, the original question that pushed me down this rabbit hole remains: have *I* heard these differences, or was it lack of level matching, placebo etc.? I think I need to ABX with music recorded through both devices. Just need to make sure I can record something as close as possible to what I hear when plugging headphones into them directly. The troubling part of that is I know my consumer-grade capture equipment only has audio inputs with multi kOhm impedances whereas in the listening test I used 34-ohm Hifimans. Can't be completely sure I'm getting the exact same signal when recording like this vs. what I heard in headphones, and would not be able to completely trust the ABX result in this case. Is there any Average Joe accessible equipment that can get around that?

No one objects that KTB is crap, the hump alone is enough.

But as you said "music" let's experiment with REW's FSAF measurement (electrically) which allows us to see what we do with all kinds of signals, including noise and music.


(song is 10s of "Bird on a wire")

So,test rig exactly as before, results:


E-MU


Khadas



...and the FR reported by REW for both, magnified without mercy.

I think it would be really hard to tell.

 

[donjoe0]

I think it would be really hard to tell.

Never said it was easy. But "hard to tell" is not good enough if we're going to regularly be assaulted with extremist language like "audibly transparent". It has to be impossible to tell, or we have to use other words to describe it. Or let's say... I will accept a 99.9% threshold: if 99.9% of the people can't hear a device's problems despite their best efforts under ideal listening conditions, fine, it's "audibly transparent". But not 99%, not 95%, not 90%. Not good enough for such absolutist language.

 

[sergeauckland]

Never said it was easy. But "hard to tell" is not good enough if we're going to regularly be assaulted with extremist language like "audibly transparent". It has to be impossible to tell, or we have to use other words to describe it. Or let's say... I will accept a 99.9% threshold: if 99.9% of the people can't hear a device's problems despite their best efforts under ideal listening conditions, fine, it's "audibly transparent". But not 99%, not 95%, not 90%. Not good enough for such absolutist language.

The 99.9% figure is probably a good one to pick. There may well be 0.1% of people with much greater acuity, such that they actually can hear 0.1% distortion, or a 0.1dB change in level or whatever. My view is that these outliers don't detract from the basic premise that for all intents and purposes, it's transparent to the vast majority.

S.

 

[Sokel]

Never said it was easy. But "hard to tell" is not good enough if we're going to regularly be assaulted with extremist language like "audibly transparent". It has to be impossible to tell, or we have to use other words to describe it. Or let's say... I will accept a 99.9% threshold: if 99.9% of the people can't hear a device's problems despite their best efforts under ideal listening conditions, fine, it's "audibly transparent". But not 99%, not 95%, not 90%. Not good enough for such absolutist language.

Note that the above shows not total harmonic distortion but total distortion,so all of them.
We also have to note the unfair condition of 4 m cable from Khadas to ADC (I like measuring gear as they sit on the rack) and the low input impedance of it.

At the end of the day considering the above I would go higher than 99.9% if a crap DAC can go this close despite the punishing conditions.
IF there's a chance for something to be audible with it (KTB) would be after a combination of levels to its bad spot of hump (around -30dBFS), PCM signal (does better with DSD, has no hump) ,system's gain structure,etc.

 

[danadam]

AFAIU he means the 1 pixel difference in the level of some tones in the multi-tone graph in Hidiz s8 review:

Someone correct me if I'm wrong...

The multi-tone file has tones with frequencies selected so that they fall in the centers of the FFT frequency bins. This means that in the FFT of the file, the magnitudes of the tones are always exact, regardless of which window is used.

When the file goes through DAC and ADC then some tones will be ever so slightly off the center of the FFT frequency bins, because the clocks in the converters are not perfect (or maybe just not synchronized with each other?). And this means that in the FFT of the capture, the magnitudes of the tones will depend on the type of the window that is used.

AFAICT the multi-tone graph in reviews was never intended to measure the tone levels but rather the noise floor, so it probably doesn't use flat-top window.

Here's Tanchjim Space recorded with ADI-2 Pro and the file processed with 3 windows: flat top, Blackman-Harris and Hann:

 

[donjoe0]

in the FFT of the capture, the magnitudes of the tones will depend on the type of the window that is used

Nice! Looks like the most plausible explanation so far. Even if this doesn't explain 100% of the weirdness in my results, it still points to a necessary improvement in the multitone software, i.e. if the selectable tone sets are to stay the same, the FFT (or window type?) setting must not be left to the user but adapted specifically for each tone set in order to not distort the FR.

Note, though, that the Audio Precision's 32-tone set is probably matched to the FFT bins, as it does allow for the spike tips to align to flat for exceptional DACs and still show defects at various frequencies for lower-grade DACs. There wouldn't be any reason for the defect frequencies to vary if it was all from tones vs. FFT misalignment.

At the end of the day considering the above I would go higher than 99.9% if a crap DAC can go this close despite the punishing conditions.

Sure, I said 99.9 just to make a point. If I were aiming for the most useful number I'd be looking at something more like "99.9% of audiophiles", and restrict it to people with lots of gear and listening experience.

 

[pkane]

I'm getting much larger wobbles in the FR with pkane's multitone tool, when using the 300-400 tone settings.

Someone correct me if I'm wrong...

The multi-tone file has tones with frequencies selected so that they fall in the centers of the FFT frequency bins. This means that in the FFT of the file, the magnitudes of the tones are always exact, regardless of which window is used.

When the file goes through DAC and ADC then some tones will be ever so slightly off the center of the FFT frequency bins, because the clocks in the converters are not perfect (or maybe just not synchronized with each other?). And this means that in the FFT of the capture, the magnitudes of the tones will depend on the type of the window that is used.

AFAICT the multi-tone graph in reviews was never intended to measure the tone levels but rather the noise floor, so it probably doesn't use flat-top window.

Here's Tanchjim Space recorded with ADI-2 Pro and the file processed with 3 windows: flat top, Blackman-Harris and Hann:
View attachment 439209

FFT windowing and the spread of an individual tone energy over multiple bins is the reason for the strange looking amplitudes when plotted, especially when using larger FFT sizes. Smaller FFT size will result in wider bins that catch more of the tone per bin. Still, the energy of a single tone spread over multiple bins can be computed correctly, resulting in the correct magnitude of the frequency response
(look for an updated MTA version soon - I discovered a bug that affected frequency response computation from multitone)

Here's what a 400 tone multitone looks like with an actual test DAC + ADC setup, 1M FFT and Blackman-Harris 7 window:

And here's the frequency response from this multitone:

And the frequency response from a log-chirp:

 

[DonH56]

Check in the general audio forum,I started the thread. Amir may change the name of it. Makes a probe tone (digital domain) that can be used to accurately deconvolve a 10 second signal into an impulse, and does so for both frequency response and impulse response. Demo on digitally filtered drylab signal included. I did not include the probe signal as a wav file, because I guess I can't do that. Also, it would be good to include the .m file as a file but not that, either.

Awesome, thanks!

Link to the thread: https://www.audiosciencereview.com/...ry-matlab-programs-to-test-your-system.61736/

 

[kyuu]

I did not include the probe signal as a wav file, because I guess I can't do that. Also, it would be good to include the .m file as a file but not that, either.

Is this due to the forum not allowing the file format to be attached to a post? Generally the way to get around that is stick the files into a zip archive first and then attach the zip file.

 

[DonH56]

These are the formats allowed; changing the extension has worked for me, though I have also put everything into a zip file for convenience.

 

[AdamG]

FYI as requested by @abm0 about two pages of posts as the conversation shifted to multitone topic were merged here. If we relocated any posts in error please let us know. Thanks

 

[pkane]

As mentioned earlier, here's an updated version 1.2.10 of Multitone that should produce a more accurate frequency response curve from a multitone test signal even with large FFTs and different FFT windows: https://app.box.com/s/ue7ll9xmvwogst817x2l1xg09opvgy47

(@abm0, @Sokel - please test)