Beta Test: Multitone Loopback Analyzer software

[Sokel]

Looks a lot better:

and that's th spectrum:

 

[FrenchFan]

Someone correct me if I'm wrong...

The multi-tone file has tones with frequencies selected so that they fall in the centers of the FFT frequency bins. This means that in the FFT of the file, the magnitudes of the tones are always exact, regardless of which window is used.

When the file goes through DAC and ADC then some tones will be ever so slightly off the center of the FFT frequency bins, because the clocks in the converters are not perfect (or maybe just not synchronized with each other?). And this means that in the FFT of the capture, the magnitudes of the tones will depend on the type of the window that is used.

AFAICT the multi-tone graph in reviews was never intended to measure the tone levels but rather the noise floor, so it probably doesn't use flat-top window.

Here's Tanchjim Space recorded with ADI-2 Pro and the file processed with 3 windows: flat top, Blackman-Harris and Hann:
View attachment 439209

I confirm that the amplitude depends on the window.

Here's a simple multitone with 300 theoretical
frequencies processed with BH7 and HFT248D.

BH7 amplitude tolerance: ~5%
HFT248 amplitude tolerance: ~2/10000

Graphs attached.

 

[Rantapossu]

As mentioned earlier, here's an updated version 1.2.10 of Multitone that should produce a more accurate frequency response curve from a multitone test signal even with large FFTs and different FFT windows: https://app.box.com/s/ue7ll9xmvwogst817x2l1xg09opvgy47

(@abm0, @Sokel - please test)

The new smoothed view on the sweep lines looks so much better than before. Thank you very much!

V1.2.9:

V1.2.10:

 

[pkane]

The new smoothed view on the sweep lines looks so much better than before. Thank you very much!

V1.2.9:

View attachment 439589

V1.2.10:

View attachment 439593

Ha! I forgot I added this option

 

[Rantapossu]

Ha! I forgot I added this option

The interval sweep has been an invaluable help over the past few weeks as I've tried to minimize the H3 by investigating and changing the temperature of the devices.

I know you're busy, but I hope you'll find time at some point to fix the issues I reported earlier regarding the interval sweep...

 

[donjoe0]

(@abm0, @Sokel - please test)

OK, output looks much better now, even with the way worse noise I'm getting this year from this PC, but my curse of getting "interesting" results has not lifted it seems.

First thought was "OK, since things are being computed properly for all window sizes now, I don't really need millions in FFT size, let's do just a bit more than 1 bin per ERB, say 128k". So I set it to 300 tones, 128k FFT, frequency range 20-20k like last October:

OK, much better behaved, no longer deviating from flat by almost 1 dB, but what have we here? Did FiiO put a little 0.2 dB sub-bass bump at 20 Hz just to give people the feeling of "man this puts out so much power just like they advertised, the bass is so authoritative yadda yadda"? (Running it in high-gain here, 54/60 for its own volume control, Desktop Mode off, 92/100 Windows volume.)

That bump did occur right at one end of my specified range, what if it's yet another artifact of the math in the tool? Let's move that down a bit and do 15 Hz, and extend all the way to the 24 kHz that my 48k sampling allows up top. And let's get it to the higher values I used in most tests last year, like 400 tones, 1 million FFT:

Oops.
Well, that did something to the 20 Hz bump, but what is this up top now? A weakness of 0.1 dB right in the vocal presence and sibilance region? Just like what I keep hearing in A/B listening tests vs. the HiBy FC3, where for some tracks with stage-central vocals the KA17 makes them sound veiled and pulled back vs. the crisp front-row-seat presentation from the FC3?

What say you, FC3?

Considerably more noise susceptible, with its flimsier housing that I don't think is even metal all around (see TD+N but also the more zigzaggy response), but otherwise the same disciplined neutral top of the class teacher's pet response I always see and hear from it.

Wait, what if I subject it to those other weirdness generating values I tried in the first test?

Still noisy as hell, mayybe(?) has some weakness in the sub-bass but I wouldn't bet on that being real instead of just noise. And otherwise, it just won't let itself be fazed by any 'scary' test parameters, still a straight arrow as always.

Now there might still be interactions of FFT size and number of tones that produce unwanted stuff, but I'm reaaally starting to suspect FiiO is up to dirty tricks with this one. I look at these ever shifting responses depending on the exact input and planet alignment and I can't avoid the suspicion that they're changing their response based on song contents(?) and introducing little response quirks to give people artificial impressions of "hifi-ness" like stronger sub-bass or deeper stage or smoother "high quality" treble (from that recessed sibilance area).

 

[pkane]

OK, output looks much better now, even with the way worse noise I'm getting this year from this PC, but my curse of getting "interesting" results has not lifted it seems.

First thought was "OK, since things are being computed properly for all window sizes now, I don't really need millions in FFT size, let's do just a bit more than 1 bin per ERB, say 128k". So I set it to 300 tones, 128k FFT, frequency range 20-20k like last October:
View attachment 439730

OK, much better behaved, no longer deviating from flat by almost 1 dB, but what have we here? Did FiiO put a little 0.2 dB sub-bass bump at 20 Hz just to give people the feeling of "man this puts out so much power just like they advertised, the bass is so authoritative yadda yadda"? (Running it in high-gain here, 54/60 for its own volume control, Desktop Mode off, 92/100 Windows volume.)

That bump did occur right at one end of my specified range, what if it's yet another artifact of the math in the tool? Let's move that down a bit and do 15 Hz, and extend all the way to the 24 kHz that my 48k sampling allows up top. And let's get it to the higher values I used in most tests last year, like 400 tones, 1 million FFT:
View attachment 439731
Oops.
Well, that did something to the 20 Hz bump, but what is this up top now? A weakness of 0.1 dB right in the vocal presence and sibilance region? Just like what I keep hearing in A/B listening tests vs. the HiBy FC3, where for some tracks with stage-central vocals the KA17 makes them sound veiled and pulled back vs. the crisp front-row-seat presentation from the FC3?

What say you, FC3?
View attachment 439736
Considerably more noise susceptible, with its flimsier housing that I don't think is even metal all around (see TD+N but also the more zigzaggy response), but otherwise the same disciplined neutral top of the class teacher's pet response I always see and hear from it.

Wait, what if I subject it to those other weirdness generating values I tried in the first test?
View attachment 439739
Still noisy as hell, mayybe(?) has some weakness in the sub-bass but I wouldn't bet on that being real instead of just noise. And otherwise, it just won't let itself be fazed by any 'scary' test parameters, still a straight arrow as always.

Now there might still be interactions of FFT size and number of tones that produce unwanted stuff, but I'm reaaally starting to suspect FiiO is up to dirty tricks with this one. I look at these ever shifting responses depending on the exact input and planet alignment and I can't avoid the suspicion that they're changing their response based on song contents(?) and introducing little response quirks to give people artificial impressions of "hifi-ness" like stronger sub-bass or deeper stage or smoother "high quality" treble (from that recessed sibilance area).

If you want to see why the measurement showed a bump in sub-bass, take a look at the spectrum plot around that area, zoom in if you need to. Are there enough tones below 20Hz? Did you generate a multitone that starts at those frequencies? What other multitone settings did you have enabled? Do the tones overlap each other? -- if yes, increase FFT size or use (no window) setting.

If you want to remove the effect of the FFT window, use the rectangular window (no window) setting. If you have minimal clock drift between the DAC and the ADC, that is the ideal setting as all tones will fall on just one FFT bin. But, if you do have clock drift, you'll find the whole multitone measurement will be wrong, with elevated, uneven noise floor, and wildly overlapping tones.

 

[donjoe0]

If you want to remove the effect of the FFT window, use the rectangular window (no window) setting. If you have minimal clock drift between the DAC and the ADC, that is the ideal setting as all tones will fall on just one FFT bin. But, if you do have clock drift, you'll find the whole multitone measurement will be wrong, with elevated, uneven noise floor, and wildly overlapping tones.

It's "no window" in all of them, so the 20 Hz bump has to be from something else, though not necessarily device specific, as I'm also seeing it with the JCally JM6 Pro:

So for the rest of the weirdness then it has to be majorly influenced by clock drift. After all I am outputting through USB dongle DACs here, while recording through my motherboard's integrated audio chipset. Could be that a big part of the apparent "quality" differences are simply differences in clock drift between the different oscillators in these dongles vs. my ADC. But if that were the case... wouldn't that severely limit the usefulness of this kind of loopback testing? If it always has to be the same clock for output and input, you can only really test audio interfaces and sound cards with it, not standalone DACs and DAC/amps.

 

[pkane]

It's "no window" in all of them, so the 20 Hz bump has to be from something else, though not necessarily device specific, as I'm also seeing it with the JCally JM6 Pro:
View attachment 439740

So for the rest of the weirdness then it has to be majorly influenced by clock drift. After all I am outputting through USB dongle DACs here, while recording through my motherboard's integrated audio chipset. Could be that a big part of the apparent "quality" differences are simply differences in clock drift between the different oscillators in these dongles vs. my ADC. But if that were the case... wouldn't that severely limit the usefulness of this kind of loopback testing? If it always has to be the same clock for output and input, you can only really test audio interfaces and sound cards with it, not standalone DACs and DAC/amps.

Yes, there's clock drift as shown by 'Drift' result on the bottom: 57.7ppm is a fairly sizeable drift.

 

[donjoe0]

If you want to see why the measurement showed a bump in sub-bass, take a look at the spectrum plot around that area, zoom in if you need to. Are there enough tones below 20Hz? Did you generate a multitone that starts at those frequencies?

Well I can't say how many should be enough, but I do see tones down to 8.9 Hz when I set my starting value to 20 Hz where it says "Freq".
Zoomed view for a case where I do get the 20 Hz bump:

Wrt. clock drift, really for all the others except that one JM6 result, it's around 5 ppm or lower. Maybe these results are not that useless after all.

 

[donjoe0]

what is this up top now? A weakness of 0.1 dB right in the vocal presence and sibilance region? Just like what I keep hearing in A/B listening tests

Nope, just a coincidence. It seems there's still the potential to see "cloud bunnies" depending on the interaction of FFT size and number of tones selected. With 200 tones and 2 million FFT the sibilance area is boosted, not cut:

Yet there's still this annoying discrepancy where these things always show up with the KA17 but not the FC3:

Could a clock drift of -3.3 ppm create this kind of difference all by itself? I notice the FC3 got a perfect 0 ppm this time.

 

[pkane]

Nope, just a coincidence. It seems there's still the potential to see "cloud bunnies" depending on the interaction of FFT size and number of tones selected. With 200 tones and 2 million FFT the sibilance area is boosted, not cut:
View attachment 439753

Yet there's still this annoying discrepancy where these things always show up with the KA17 but not the FC3:
View attachment 439754

Could a clock drift of -3.3 ppm create this kind of difference all by itself? I notice the FC3 got a perfect 0 ppm this time.

Is there a reason you're trying to use multitones to measure frequency response? Noise, non-linearity, timing errors, etc. can all affect this measurement since it's based on measuring amplitude of individual tones. Did you try a log-chirp, instead?

 

[donjoe0]

Is there a reason you're trying to use multitones to measure frequency response? Noise, non-linearity, timing errors, etc. can all affect this measurement since it's based on measuring amplitude of individual tones. Did you try a log-chirp, instead?

Well yes, it's the suspicions of not enough similarity to music for tone sweeps and white noise. Coupled with the fact that I keep hearing treble presentation differences when comparing these two particular dongles, but "standard" FR measurements don't show anything that should be so different between them as to be audible.

Tried log-chirp as well, pretty much tracks with tone sweep and white noise based FR, though I'm not sure why the log-chirp shows a "fattening" of the plot toward the high end. (Should I ignore that and just imagine a thin line going through the middle of it, as being the "real response" it's showing me?)

Anyway, yeah, I think I've milked this multitone idea for all it could give me, I really need to return to testing for real audibility of these DAC differences vs. potential confounders I may have fallen victim to. I continued this discussion today primarily to provide you with test results for the new version of the tool.

 

[pkane]

The interval sweep has been an invaluable help over the past few weeks as I've tried to minimize the H3 by investigating and changing the temperature of the devices.

I know you're busy, but I hope you'll find time at some point to fix the issues I reported earlier regarding the interval sweep...

@Rantapossu: just for you! Multitone v1.2.11 should fix the issues with the interval sweep chart. Please test and let me know:

https://app.box.com/s/ue7ll9xmvwogst817x2l1xg09opvgy47

 

[Rantapossu]

@Rantapossu: just for you! Multitone v1.2.11 should fix the issues with the interval sweep chart. Please test and let me know:

https://app.box.com/s/ue7ll9xmvwogst817x2l1xg09opvgy47

Oh, I feel so special...

Here are the first impressions:

1) No circles, triangles and crosses for the first round (At 0 seconds) on the chart -> FIXED!
2) Color change of the lines and the missing the 3rd line (White) after visiting the other tab -> FIXED!
3) Forgetting the manually set scaling after every round -> FIXED!
4) Not showing the 3rd measurement text and symbol at the first round -> Still broken. They appear after the second round...

5) A new feature, but I would still call it a bug. The X-axis is huge by default. Here is 6 minutes of measurement and it barely shows:

Pressing the "Fit all data" -button fixes the issue for a second, but the scaling will revert back to incorrect after the next round. If you now press the "Fit all data" -button and immediately manually change the scaling to correct before the next round, it will remain correct.

 

[pkane]

Oh, I feel so special...

Here are the first impressions:

1) No circles, triangles and crosses for the first round (At 0 seconds) on the chart -> FIXED!
2) Color change of the lines and the missing the 3rd line (White) after visiting the other tab -> FIXED!
3) Forgetting the manually set scaling after every round -> FIXED!
4) Not showing the 3rd measurement text and symbol at the first round -> Still broken. They appear after the second round...

View attachment 439899

View attachment 439900

5) A new feature, but I would still call it a bug. The X-axis is huge by default. Here is 6 minutes of measurement and it barely shows:

View attachment 439901


Pressing the "Fit all data" -button fixes the issue for a second, but the scaling will revert back to incorrect after the next round. If you now press the "Fit all data" -button and immediately manually change the scaling to correct before the next round, it will remain correct.

View attachment 439906

OK, one more try Hopefully items 4 and 5 are fully resolved (version number is unchanged, so please reinstall old v1.2.11):

https://app.box.com/s/ue7ll9xmvwogst817x2l1xg09opvgy47

 

[Rantapossu]

OK, one more try Hopefully items 4 and 5 are fully resolved (version number is unchanged, so please reinstall old v1.2.11):

https://app.box.com/s/ue7ll9xmvwogst817x2l1xg09opvgy47

Everything seems to be working fine regarding these previously reported issues.

Thank you very much!

(I do have a couple of smaller bug reports regarding graph behavior, though. I'll report more on those shortly...)

 

[Rantapossu]

This is a global (But minor) bug regarding the "Measurement hovering".

(This feature

... )

This may be a feature of the graphics library and perhaps unfixable.

If you keep your mouse pointer over the black graph area during the measurement and Multitone completes the run and shows the result, hovering doesn't work anymore. It shows nothing.

You need to move the mouse pointer away from the black area and back to it to make the hovering work again, like below:

If you keep the mouse pointer outside the black area throughout the measurement, hovering will work normally when you move the mouse pointer to the black area and over the line.

 

[Rantapossu]

This is perhaps an even smaller bug, no need to fix it if it requires any effort...

The interval sweep shows sometimes the X-axis time a little bit funny when hovering:

For example:
2 mins = 1.60 mins
3 mins = 2.60 mins
4 mins = 3.60 mins (Like below)

 

[pkane]

This is perhaps an even smaller bug, no need to fix it if it requires any effort...

The interval sweep shows sometimes the X-axis time a little bit funny when hovering:

For example:
2 mins = 1.60 mins
3 mins = 2.60 mins
4 mins = 3.60 mins (Like below)

View attachment 439941

This one is an easy fix, the previous one having to move the mouse... you guessed it: it's how the graphics library handles it.