DAC listening test files - updated with new filters files

[Thomas_A]

Sorry, I didn't mean to single you out. Yours was just the last message in the discussion. And AFAIU the discussion, it is about 9/10 happening by chance and 16-trials test helping with that:

@Thomas_A wrote
"It would be better if everone made 16 or 20 trials. I made 100 random tests with 10 trials and got 9/10 two times and 0/10 one time."
@huw wrote:
"As mentioned several times a single 9/10 result can be random chance if it was cherry-picked from multiple attempts."
you wrote:
"We’ve been asking for the more standard 16 trials for quite some pages, which is one of the reasons why."

I just don't see how 16-trials test helps with that, if p remains the same. It only helps when compared to 2 tests intended for summing, i.e. 2x 10-trials test vs 1x 16-trials test, because 2x10 requires trust that they happened one after the ohter.

As already mentioned, Josh made 16/16 for each of 5 pairs of DACs and 15/16 for two other combinations. 16/16 is very low chance to happen by random even more so for 5 pairs of DACs. FooBar has 16 trials as default, so why depart from that? Repeating 16/16 or 15/16 would be good to see.

 

[huw]

Josh made 16/16 for each of 5 pairs of DACs

Weren't these 2 x 8 trial tests?

I have to admit to being slightly confused by some of the discussion here. It seems obvious to me that any test for which a passing score can be achieved by random chance can be cheated by simply repeating the test over and over until the desired score is obtained and ignoring the failed attempts. The only question then becomes: Is the motivation to be able to demonstrate a passing score sufficient to reward the effort required? The more trials in each test the more motivation is required.

 

[Thomas_A]

Weren't these 2 x 8 trial tests?

I have to admit to being slightly confused by some of the discussion here. It seems obvious to me that any test for which a passing score can be achieved by random chance can obviously be cheated by simply repeating the test over and over until the desired score is obtained and ignoring the failed attempts. The only question then becomes: Is the motivation to be able demonstrate a passing score sufficient to reward the effort required? The more trials in each test the more motivation is required.

Yes they were 2x8 trials in a row according to the description of the article. So effectively it would correspond to 16 trials as default in FooBar.

 

[RandomEar]

I just don't see how 16-trials test helps with that, if p remains the same. It only helps when compared to 2 tests intended for summing, i.e. 2x 10-trials test vs 1x 16-trials test, because 2x10 requires trust that they happened one after the ohter.

You can get p < 0.05 using a 5 trial test with all 5 being correct guesses. I think we all agree that while the p-value for such a test is technically significant according to the commonly used threshold, this simply isn't enough repetitions to get a reliable result in any form, right?

I can't really explain what the specific statistical problem is in this case. But I would imagine that the error bars on your estimator (-> p-value) grow exponentially for smaller sample sizes. Or in other words: The p-value itself does not tell you how reliable the p-value is. Happy to be corrected in case I misunderstood this, though.

 

[pjug]

You can get p < 0.05 using a 5 trial test with all 5 being correct guesses. I think we all agree that while the p-value for such a test is technically significant according to the commonly used threshold, this simply isn't enough repetitions to get a reliable result in any form, right?

I can't really explain what the specific statistical problem is in this case. But I would imagine that the error bars on your estimator (-> p-value) grow exponentially for smaller sample sizes. Or in other words: The p-value itself does not tell you how reliable the p-value is. Happy to be corrected in case I misunderstood this, though.

So p < 0.05 meaning <5%. I think we do only need the p value (for the "at least" number). Setting aside the idea of double sided scoring, a coin flip calculator gives:

5/5 p = 3.15%
at least 9/10 p = 1.07%
at least 14/16 p = 0.21%
at least 15/16 is p= 0.026%

 

[RandomEar]

Even if it wasnt clipping that ringing is huge (10khz/1v on a 3 v signal) and probably audible for some.

I took another look at the delta spectrogram of the D10s vs. DacMagic+ test files:

It's clear that the difference emerges above 21.5 kHz. Looking at both reviews, the D10s reaches -30 dBrA at that point. For the DacMagic+ to match this, the active filter has to be the "steep" one, which is a linear phase fast roll-off type and does not clip in Amir's test. So I'm pretty sure my former concerns about possible clipping are unwarranted based on this analysis.

 

[RandomEar]

Concerning another topic: I have matched the two files ('dactest_comp.flac' & 'dactest_ref.flac' from D10s vs. DacMagic+) in DeltaWave again. Since Josh posted here that he heard an immdediate difference in the high hats in the first seconds of the song, I listened to the delta in that region. At elevated listening levels, it's inaudible to me up to second seven, where the singing begins. I need to push it +10 dB to detect anything reliably in that region (< 7 s) using either speakers or IEMs and even then, it's barely there. The differences after 7 s are much bigger.

In any case: Josh claims to have identified the difference between these two test tracks in the region where there is the least audible difference, as supported by the delta track and the PK metric:

Maybe his wording was imprecise and he refered to a later timestamp, but - and I'm kinda sorry to say it - the way I understand it right now, this is just making me more and more sceptical. If you listen to the delta, it is very, very clear that the perceptible differences begin the second the singers voice appears. And every scepticism aside, logic dictates that to hear a difference between two files, the difference has to exist and has to be audible in itself. If you reject this last proposition ("audible in itself" = audible in the delta track), we're quickly getting into the CHORD "I can hear noise at -300 dB" territory if you ask me.

Nontheless, I encourage you to test this yourself using DeltaWave and report back here.

 

[danadam]

Repeating 16/16 or 15/16 would be good to see.

If you want the probability of getting the result by chance to be so low, that's fine, but the way to communicate it is to state the p instead of the number of trials. A given p value will anyway require some minimum number of trials in a test. If 15/16 (p = 0.026%) is good, then 12/12 (p = 0.024%) should be similarly good. It's also better for the person taking the test to know beforehand what kind of rigor is expected

The more trials in each test the more motivation is required.

The more trials but with the same p value requires only slightly more motivation. If someone can do 100 tests with 10 trials to get 9/10 (p = 1.075%) by chance, then 100 tests with 16 trails to get 13/16 (p = 1.063%) by chance is only 60% more work. If you want to increase "motivation" requirement, then this is better done by asking for lower p value.

You can get p < 0.05 using a 5 trial test with all 5 being correct guesses. I think we all agree that while the p-value for such a test is technically significant according to the commonly used threshold, this simply isn't enough repetitions to get a reliable result in any form, right?

I can't really explain what the specific statistical problem is in this case.

Below is a simulation on jsfiddle.net. You can set number of tests and number of trials per test. The output says how many instances of each result there were. I run it a few times with:

number_of_tests = 1000;
trials_per_test = 5;

and

number_of_tests = 1000;
trials_per_test = 16;

Comparing 5/5 (p = 3.125%) and 12/16 (p = 3.841%) the results seem similar to me.

Untitled fiddle - JSFiddle - React, Tailwind, and code Playground

JSFiddle - Test your React, Tailwind, JavaScript, CSS, HTML online with JSFiddle.

jsfiddle.net

 

[RandomEar]

@danadam I did the same quick test in MATLAB and, of course, the p-value is a perfect predictor for the result of a large number of trials. I know intuition is a bad predictor when it comes to statistics, but it just feels like tests with a smaller number of total trials are more error prone? As in: Your chance of randomly guessing correctly is 50% for each trial, but in a 5 trial run, each single trial carries a much higher weight in the final result.

So if you expect on average 2.5/5 correct guesses for an infinite number of test runs using 5 trials per run, that means you are on average just 2.5 "lucky guesses" away from significance in each run. From those on average 2.5 correct guesses, that would be a 17.7% chance. Whereas for the 16 trial test case, your average is 8/16, so you are 4 "lucky guesses" or a 6.25% chance away from significance. You also do have proportionally more trials and if you run either the 5 trial or the 16 trial test variant until you reach the same total (very high) number of trials, the results will, of course, converge. However, for a single run, it feels like the 5 trial test has a higher chance of producing outliers.

There may be some logical error in this train of thought. As I said, working with stats = often not intuitive. That being said, as I understand it 16 trials is the default in foobar2000 and @pma did recommend to use that number more than a week ago and advised Josh to use more than 10 trials a couple days before that.

EDIT: Corrected the number of guesses. Also realized that if we are dealing with fractional guesses, we would technically need to do the same for the significance threshold. Which probably levels the differences for the 5 vs 16 trials comparison...

 

[huw]

In any case: Josh claims to have identified the difference between these two test tracks in the region where there is the least audible difference, as supported by the delta track and the PK metric:

That is very interesting! It wouldn't have surprised me if a trained listener with experience of using DeltaWave might have run exactly this analysis before attempting the ABX tests from @pma. EDIT (original was a bit more accusatory than I intended)

 

[huw]

If someone can do 100 tests with 10 trials to get 9/10 (p = 1.075%) by chance, then 100 tests with 16 trails to get 13/16 (p = 1.063%) by chance is only 60% more work

I'm glad you're not my boss: "It's only just over half as much work again, what're you complaining about?!"

 

[pjug]

Concerning another topic: I have matched the two files ('dactest_comp.flac' & 'dactest_ref.flac' from D10s vs. DacMagic+) in DeltaWave again. Since Josh posted here that he heard an immdediate difference in the high hats in the first seconds of the song, I listened to the delta in that region. At elevated listening levels, it's inaudible to me up to second seven, where the singing begins. I need to push it +10 dB to detect anything reliably in that region (< 7 s) using either speakers or IEMs and even then, it's barely there. The differences after 7 s are much bigger.

In any case: Josh claims to have identified the difference between these two test tracks in the region where there is the least audible difference, as supported by the delta track and the PK metric:

View attachment 553339

Maybe his wording was imprecise and he refered to a later timestamp, but - and I'm kinda sorry to say it - the way I understand it right now, this is just making me more and more sceptical. If you listen to the delta, it is very, very clear that the perceptible differences begin the second the singers voice appears. And every scepticism aside, logic dictates that to hear a difference between two files, the difference has to exist and has to be audible in itself. If you reject this last proposition ("audible in itself" = audible in the delta track), we're quickly getting into the CHORD "I can hear noise at -300 dB" territory if you ask me.

Nontheless, I encourage you to test this yourself using DeltaWave and report back here.

I would not dismiss the idea that Josh can hear the bandpass ripple, which is the same as a true pre and post echo (again, something different from Gibbs phenomenon) that might be audible. That is the main difference in the files.

 

[RandomEar]

I would not dismiss the idea that Josh can hear the bandpass ripple, which is the same as a true pre and post echo (again, something different from Gibbs phenomenon) that might be audible. That is the main difference in the files.

The Gibbs phenomenon is in the time domain and, as far as I know, has not been shown to be audible in filters. But even if it were, it would sound and look nothing like the passband ripple which is in the frequency domain. The plots look similar, but they show totally different things. In addition, the ripple is on a scale (<0.1 dB) which is at best right on the edge of audibility even for full spectrum level differences in ideal scenarios using broad band noise [1, 2]. You can test it using pure tones here - that's likely easier to pass than using music, but harder than using noise. In any case, I would be very surprised if the passband ripple ends up being detectable.

In addition, I would ask you to give the delta track a quick listen in DeltaWave: There's just nothing to hear in the first seven seconds unless you crank the SPL to uncomfortable levels. How could anybody hear something which is inaudible in itself with an additional loud music signal masking that inaudible stuff being present in the sample file? From my perspective, a lot of things are not adding up right now. But I hope we may soon learn why.

 

[pjug]

The Gibbs phenomenon is in the time domain and, as far as I know, has not been shown to be audible in filters. But even if it were, it would sound and look nothing like the passband ripple which is in the frequency domain. The plots look similar, but they show totally different things. In addition, the ripple is on a scale (<0.1 dB) which is at best right on the edge of audibility even for full spectrum level differences in ideal scenarios using broad band noise [1, 2]. You can test it using pure tones here - that's likely easier to pass than using music, but harder than using noise. In any case, I would be very surprised if the passband ripple ends up being detectable.

In addition, I would ask you to give the delta track a quick listen in DeltaWave: There's just nothing to hear in the first seven seconds unless you crank the SPL to uncomfortable levels. How could anybody hear something which is inaudible in itself with an additional loud music signal masking that inaudible stuff being present in the sample file? From my perspective, a lot of things are not adding up right now. But I hope we may soon learn why.

I have done ABX of the tracks in training mode including what Josh pointed to. Like you, I could not hear any difference. But the people who say they hear bandpass ripple claim the echoes are heard in a variety of ways (none having to do with sounding like echoes). Audibility may be related to the Haas effect (precedence effect). We should pretend Gibbs effect doesn't exist for discussing this subject. Bandpass ripple and the resulting echoes are a completely different effect. This was explained to me by one of the polite and math minded members in the other forum. But consideration doesn't seem to be getting traction in this thread, even though bandpass ripple is the big difference in the files.

 

[RandomEar]

Passband, band pass is a filter type This ripple is in the freqency domain and doesn't affect the phase, so there will be no echo-like effect. It is like an incredibly small EQ in a couple of places. You can even recreate it in EAPO using a dozen peaking filters. Maybe I'll do that some time and increase the ripple to a dB or two just to listen to it.

 

[pjug]

Passband, band pass is a filter type This ripple is in the freqency domain and doesn't affect the phase, so there will be no echo-like effect. It is like an incredibly small EQ in a couple of places. You can even recreate it in EAPO using a dozen peaking filters. Maybe I'll do that some time and increase the ripple to a dB or two just to listen to it.

Ha ha pre and post echoes so I figured you could say it either way!

 

[amirm]

These tests must be done in exclusive mode, bypassing the OS audio pipeline. Otherwise, both MacOS and Windows will resample the audio (to typically 48 kHz), and perform dithering, etc. These transforms could cause one sample to have artifacts and the other not. And at any rate, render any analysis of the original files moot as that is not what is being played.

 

[solderdude]

Ha ha pre and post echoes so I figured you could say it either way!

Pre and post echoes can be present in tape recordings using very thin tape and being stored over a certain time period.
Such a thing does not happen with digital files.

One should realize that the 'ringing' one sees in the familiar plots is always near Nyquist and above audible frequencies for most people.
They are only triggered by the frequencies near Nyquist and always lower in level than the signals that 'cause' them unlike pre-echos from tape.
Also, in the case of Josh's test that ringing would be in the 50kHz range so is not an issue at all.
The term 'pre-echo' is not really applicable and has a very different effect yet suggests a similar effect occurs and it doesn't.

Even when it all is resampled to 48kHz or 44kHz and the signals are very similar the filter used in the resampler would also react the same creating no difference when ABX ing.

And I would like to stress again that even if some person can reliably detect a difference by ear in certain specific parts of a recording with many training rounds and picking the best score does not equal a clearly audible difference between DACs similar as to what many people claim to hear in sighted not level matched comparisons.

Even when the box is measured and fully characterized in all kinds of conditions we still will be (my assumption) in the dark as to how Josh pulled off what he reported.
SImply because we are not Josh nor can what he heard never be confirmed nor repeated while witnessed. Speculations is all there can and ever will be... because there were no recordings of the actual test we simply can never know.

 

[pjug]

Pre and post echoes can be present in tape recordings using very thin tape and being stored over a certain time period.
Such a thing does not happen with digital files.

It does occur in digital audio if the filter causes passband ripple. Again, I can't hear the effect. I did not have a link to the math on hand but here it is from AI. First equation expresses the ripple and you can see how this then corresponds to having echoes.

Lagadec & Stockham, “Dispersive Models for A-to-D and D-to-A Conversion Systems” (AES Preprint 2097). The relationship is essentially:

H(f)=1+2αcos⁡(2πfT)

which corresponds in the time domain to

h(t)=δ(t)+αδ(t−T)+αδ(t+T).

And once again, I believe that if Josh can hear a difference in those files, this is most likely what he is able to hear. Because removing the ripple with DW pushes the PK metric down to better than -110dB.

 

[Sokel]

Disclaimer: I don't even know if its a thing to compare two files the way I did, but just for the fun of it.

What you see below is an FSAF, acoustical measurement of the two 2_DAC_test, right channel only.
Identical conditions (as much as it can be for acoustic measurements) , etc, the works.

First 10 seconds, right channel only:

Red is the first file of the test, Blue is the second.
Low's difference is probably external and my room has plenty of gain there, I don't think the file does '30's .

...and the phase

As silly as it can be, that's it.

(I repeated it twice, same results)