• Welcome to ASR. There are many reviews of audio hardware and expert members to help answer your questions. Click here to have your audio equipment measured for free!

DAC listening test files - updated with new filters files

I don't like JND, but I think the p<0.5 criterion should apply to audio as well - or at least be the starting point for discussion, rather than p<0.05.
You seem to mix up statistical significance with outcome.
 
Thanks for the considered response, I do go along with most of it.
That's not because of the "large sample size", but rather because you reached p < 0.023. You could be equally (actually more) confident with 15/20 test (p < 0.021). What the "large sample size" gives you, is that you get more confidence in detecting very small effects when they actually exist.
Large samples are probably a bit of a red herring, as a "just noticeable difference" can go from being very low to very high confidence. A win rate that is high or low is also a red herring when taken in isolation. It's always about the win rate and sample size size together, so one may as well focus on just the p value.
AFAIU (and again, I'm not a statistician) there are 2 types of errors that can happen in a test:
  • false positive - you detect an effect when in reality there is none
  • false negative - you don't detect any effect when in reality there is one
I think this is quite important as ABX testing is usually expected for audio discrimination testing. ABX tests are distinguished by their low probability of false positives, which is achieved at the expense of a higher probability of false negatives. Neither are desirable of course, but if you can get a positive ABX result, it means that is reliable. Unfortunately it's difficult to do that with listening tests because our auditory memory isn't good. Coupled to that, a statistically significant result with very low probability of randoms coring is also expected, so that's three obstacles to getting useful test results. The odds seem to be stacked against.
Please, don't pick one sentence out of context. It also says:
I wasn't trying to misrepresent what you linked. I read the whole document and thought my quote was actually representative. I don't think it's important though. I said I don't like JND, it's also a bit of a red herring. It doesn't take random scoring and sample size into account. I was being facetious when I mentioned it, and probably shouldn't have. The thing that caught my eye, though, was the criterion that a difference should be detected more than 50% of the time, which is a low bar to pass. I wondered why audio testing had to pass a higher bar? In all the reading I did, JND was always discussed in the context of p>50%. If you want to set the bar at 75% then no arguments. You just have to decide to do that, but I don't think there's anything cast in stone to require it.

When I suggest that audio testing should beat p=0.5 instead of p=0.05, it's because when we want an ask a question where the answer isn't absolutely 100% yes or absolutely 100% no, then it's quite legitimate to ask whether the answer is just "probably" right. Of course it's better to have an answer that is "very likely" or "highly probable" or "almost certainly" correct, but those things are difficult to achieve, and we can actually consider answers given with a lower degree of confidence than that.
 
When I suggest that audio testing should beat p=0.5 instead of p=0.05, it's because when we want an ask a question where the answer isn't absolutely 100% yes or absolutely 100% no, then it's quite legitimate to ask whether the answer is just "probably" right. Of course it's better to have an answer that is "very likely" or "highly probable" or "almost certainly" correct, but those things are difficult to achieve, and we can actually consider answers given with a lower degree of confidence than that.
You appear to be trying to come up with some "new" relaxed version of what is statistically significant and what's not based on your expectation of what's critical and what's not. That's not how statistics works. Data is either significant or not, and it doesn't change with how important or unimportant field of study is. Once data passes the threshold of being statistically significant, you can then decide to require a more stringent p-value of, say, below 0.03. But you can't just arbitrarily move the threshold of what is statistically significant because you don't think the field of study is unimportant.
 
You appear to be trying to come up with some "new" relaxed version of what is statistically significant and what's not based on your expectation of what's critical and what's not. That's not how statistics works. Data is either significant or not, and it doesn't change with how important or unimportant field of study is. Once data passes the threshold of being statistically significant, you can then decide to require a more stringent p-value of, say, below 0.03.
I don't think it's black and white.
I don't think anything is black and white.
We've gotten so used to drawing the line at p<0.05 that we've forgotten the reasoning. This came home when Josh scored 8/10 in the ADC vs direct comparison. Because that's narrowly above 0.05 it was dismissed. And many more have also been dismissed, despite representing good data points. Not as good as p<0.05 of course, but certainly better than worthless. p<0.05 gives you a high degree of confidence, and I'm suggesting that there's no actual need for it, just a wish.
If we were start off at p<0.5 instead, there might be some who would accept that, which is OK, and some who would look for more, which is also OK (and quite likely).
But you can't just arbitrarily move the threshold of what is statistically significant because you don't think the field of study is unimportant.
Why not? Is it against the law? Will someone die? Will I ruin my life? In actual fact one can move the threshold if one choses to. Life will go on, and it might even get better rather than worse.
 
Last edited:
Why not? Is it against the law? Will someone die? Will I ruin my life? In actual fact one can move the threshold if one choses to. Life will go on, and it might even get better rather than worse.
You're making a mathematical, statistical decision a subjective one. Anyone can then pick a threshold they like that proves their point. If you do that, you may as well skip the whole statistical analysis, as it is then of no value.
 
We've gotten so used to drawing the line at p<0.05 that we've forgotten the reasoning. This came home when Josh scored 8/10 in the ADC vs direct comparison. Because that's narrowly above 0.05 it was dismissed.
Clearly Josh hasn't been dismissed. Could he have done 16 trials rather than 10? Sure. That's up to him. At this point @pma has looked at possible mechanisms and there is a candidate so further testing can look at than. Also @amirm has the box and can look at that. The only real purpose of p <0.05 is that certain publications want it, and the FDA to approve drugs etc but really just because a drug has been approved doesn't mean its both safe and effective. Same for any study publishing a p<0.05 result for whatever the study is trying to say. You can believe whatever you please for p<0.5 but not many people will believe you and you are highly unlikely to be able to recruit a community of people who believe things based on infinitesimally better than chance. This thread is about @pma tests and his criteria apply
 
I don't think it's black and white.
I don't think anything is black and white.
We've gotten so used to drawing the line at p<0.05 that we've forgotten the reasoning. This came home when Josh scored 8/10 in the ADC vs direct comparison. Because that's narrowly above 0.05 it was dismissed. And many more have also been dismissed, despite representing good data points. Not as good as p<0.05 of course, but certainly better than worthless. p<0.05 gives you a high degree of confidence, and I'm suggesting that there's no actual need for it, just a wish.
If we were start off at p<0.5 instead, there might be some who would accept that, which is OK, and some who would look for more, which is also OK (and quite likely).

Why not? Is it against the law? Will someone die? Will I ruin my life? In actual fact one can move the threshold if one choses to. Life will go on, and it might even get better rather than worse.
I do not think anyone will risk die regarding audio (unlike in medicine), but it still remains good to have strict limits and follow the practice in research. Once there are differences, you can add that to the equation along with features, looks and how much money you are willing to spend.
 
Anyone can then pick a threshold they like that proves their point.
Exactly.
That's what's happening already.
By setting arbitrarily high thresholds you're suppressing perfectly good data that contradicts one's opinion.
 
Exactly.
That's what's happening already.
By setting arbitrarily high thresholds you're suppressing perfectly good data that contradicts one's opinion.
That may be your impression of what’s happening, but it’s not.

Again, 0.05 p value is a “statistically significant” threshold. Above this, the data doesn’t reject the null hypothesis— it’s not significant enough for standard research purposes to consider the result any further. More stringent and lower p values may be necessary for important or critical research, where greater certainty is required.
 
Exactly.
That's what's happening already.
By setting arbitrarily high thresholds you're suppressing perfectly good data that contradicts one's opinion.
Eh, no? The threshold is commonly accepted in pretty much all fields of science. You are trying to pick your own one for this specific case to make a point. That is arbitrary.

As explained: You can argue for any threshold stricter than p<0.5 (but definitely not p<0.5 itself) and if the majority of people in that field agree that it looks reasonable, it can be adopted. That won't happen of course, because a "tried and tested" threshold already exists and is being used everywhere.
 
, so one may as well focus on just the p value.
Something I already said one month ago :)

if you can get a positive ABX result, it means that is reliable. Unfortunately it's difficult to do that with listening tests because our auditory memory isn't good.
Or because there is nothing to detect.

Coupled to that, a statistically significant result with very low probability of randoms coring is also expected, so that's three obstacles to getting useful test results.
I guess that depends on one's definition of "useful". p<0.5 is essentially random, so it doesn't seem useful to me.

If you want to set the bar at 75% then no arguments.
It's not that I want that, it's simply a necessary adjustment due to the mechanics of the test.

If I wanted the JND threshold at 75%, then in my earlier experiments I would require 75% yes-es in the yes/no experiment and 87 correct answers in the ABX experiment.

The thing that caught my eye, though, was the criterion that a difference should be detected more than 50% of the time, which is a low bar to pass. I wondered why audio testing had to pass a higher bar?
It's only a low bar if you don't do the necessary adjustment. With the adjustment the bar for JND is the same at about 16 trials and becomes higher at 24 and more trials.
 
Something I already said one month ago :)
Or because there is nothing to detect.
I guess that depends on one's definition of "useful". p<0.5 is essentially random, so it doesn't seem useful to me.
It's not that I want that, it's simply a necessary adjustment due to the mechanics of the test.
If I wanted the JND threshold at 75%, then in my earlier experiments I would require 75% yes-es in the yes/no experiment and 87 correct answers in the ABX experiment.
It's only a low bar if you don't do the necessary adjustment. With the adjustment the bar for JND is the same at about 16 trials and becomes higher at 24 and more trials.
No argument with any of that, or many other comments, only the over-arching notion that nothing less than 95% confidence will do.

Where did that number come from? It's not cast in stone or discovered on the tablets. It's nothing fundamental like pi or e or c that existed before we did. We happen to have ten digits, so we count in base ten. So one tenth (uncertainty) might have been a nice round number. Presumably there was a lot of debate over a period of time, and people thought "hmmmm, I really think we ought to shoot for a lower level of uncertainty. Lets go for half that." And that caught on. If we have eight fingers we might have ended up with a sixteenth instead of a twentieth. That's how I think we arrived at p<0.05. It's nothing precise or concrete, it's just stood the test of time.

0.05 is a high level of confidence. Calling it statistically significant is an academically lofty understatement, suggesting that anything less isn't significant. I think that's unfortunate to say the least. The objective audio world has embraced statistically significant as if nothing else will do. In the other forum I drew comparison with the levels of confidence needed for a conviction in a court of law. Most countries are similar, and set a criterion of "beyond reasonable doubt" for serious criminal convictions. The legal profession resists attempts to make that quantitative rather than qualitative, but when pressed, they tend to agree on "better than 90% confident". So if you're testing medicines that might have life changing side effects, or convicting an alleged murderer and risk sending an innocent person to prison, you had better be sure you're right.

Yet when it comes to choosing a DAC, a similarly high level of confidence is still expected. I think that attitude has harmed rather than helped audio science. There's lots of good test data out there (yes, like Josh's 8/10 for the ADC test) that's been disregarded because it didn't pass an artificially high hurdle. When somebody comes out with a genuinely better audio format or codec or converter or amplifier , it could be ABX tested and still score p>0.05. It could then be dismissed as not proven to sound any better, even when it does sound very slightly better. By setting a high pass standard, I think the community has stifled progress and development by contriving to prove that everything sounds the same. The idea that "this is the way we've always done it" is something I've been fighting all my life. For me it's been a common objection, most often heard in companies that went out of business.
 
110110111011010011000100110110101000111010000101110010010001100111001100100110111100010001110111001100101101111001101001010100100111100111110111
 
  • Like
Reactions: pma
I would start with this, the files can be downloaded from the link below:


20kHz brickwall added

The original 96/24 is from the source discussed, 30s sample. Spectrum shown in the post above. There are 4 files in the zip, the original and 3 filters. I am not using brickwalls. An example of spectra of the original file and with filters are shown below. So, good luck in your ABX testing! Based on the results, we may add another filters.

[...]

A hint: most of the ultrasonic content is in the right channel, time 25.48 s. The trumpet. Much more than for the cymbals. Below a spectrum after 8kHz HPF.

[...]
Results for all filters:

FilterCorrect guessesp-value
F116/160.0%
F26/1685.5%
F316/160.0%
F419/3218.9%

Equipment: SMSL SU-9n -> SMSL SH-9 -> 7Hz x Crinacle Zero:2

I tried a bit harder for F4 in honour of @Thomas_A, but sadly, there was no significant result for me ;) I wonder if his result could be related to IMD caused by ultrasonic content, like it has been speculated for some positive high res ABX results or if it may be related to the sharply increasing phase difference above roughly 12 kHz:

Original_vs_F4_Phase.jpg


The passband ripple getting mentioned over and over in the thread is below 0.02 dB for this file pairing (and actually below 0.01 dB for most of the spectrum):

Original_vs_F4_Delta Spectra.jpg


According to the relevant paper (de Gevel (2017), The Valve DAC; itself citing Lagadec (1984), Dispersive Models for A-to-D and D-to-A Conversion Systems), 0.02 dB of ripple correspond to an echo magnitude of slightly less than -58 dB. If anybody is interested, you can calculate the echo magnitude based on the ripple magnitude using:

Code:
ECHO_MAG_dB = 8.70434*ln(RIPPLE_dB) - 24.6326
 

Attachments

....... or if it may be related to the sharply increasing phase difference above roughly 12 kHz:
View attachment 559757
That's an interesting chart, with a lot of phase shift within the pass-band. It's a bit like what some speakers do at the cross-over, or at the LF roll-off.

I never found any rules about desirable phase linearity limits or targets, but over the last few years I looked at all the speaker phase and time domain responses I could find, before & after phase compensation, and tried to correlate them together. My best estimate was that phase response needs to be linear within roughly 45 degrees to give good time domain response (assuming amplitude vs frequency was also linear within a couple of dB).

That chart looks like nearly 500 degrees excess phase variation in the audio band. Amplitude linearity is more important, but that will screw things up. My feeling is that might explain a lot of what's being heard and discussed in these threads. It's a shame that more DAC tests don't show phase response. It's a really simple objective way to get a handle on transients etc.

What DAC and filter does that chart relate to?
 
Results for all filters:

FilterCorrect guessesp-value
F116/160.0%
F26/1685.5%
F316/160.0%
F419/3218.9%

Equipment: SMSL SU-9n -> SMSL SH-9 -> 7Hz x Crinacle Zero:2

I tried a bit harder for F4 in honour of @Thomas_A, but sadly, there was no significant result for me ;) I wonder if his result could be related to IMD caused by ultrasonic content, like it has been speculated for some positive high res ABX results or if it may be related to the sharply increasing phase difference above roughly 12 kHz:

View attachment 559757

The passband ripple getting mentioned over and over in the thread is below 0.02 dB for this file pairing (and actually below 0.01 dB for most of the spectrum):

View attachment 559760

According to the relevant paper (de Gevel (2017), The Valve DAC; itself citing Lagadec (1984), Dispersive Models for A-to-D and D-to-A Conversion Systems), 0.02 dB of ripple correspond to an echo magnitude of slightly less than -58 dB. If anybody is interested, you can calculate the echo magnitude based on the ripple magnitude using:

Code:
ECHO_MAG_dB = 8.70434*ln(RIPPLE_dB) - 24.6326
I can't say what of the mechanisms that could be in play; a very slight brighter sound was what was heard. My primary guess will be IMD; I've nailed such tests in the past (IMD) but at the time, due to poorer playback system.
 
You appear to be trying to come up with some "new" relaxed version of what is statistically significant and what's not based on your expectation of what's critical and what's not. That's not how statistics works. Data is either significant or not, and it doesn't change with how important or unimportant field of study is. Once data passes the threshold of being statistically significant, you can then decide to require a more stringent p-value of, say, below 0.03. But you can't just arbitrarily move the threshold of what is statistically significant because you don't think the field of study is unimportant.
I'd reframe this slightly in that when a study is designed the p outcome is part of the predefined success criteria, so typically p<0.05 is selected. What you don't do is try to massage data after the fact and use a different p criterion. You **could** try and use p<0.5 but no one will be interested. In a certain field, you could also predefine your objective at p<0.001, so yes bottom line you can't just arbitrarily move the p-value to try and massage the data. To be simple a hypothesis would be "This study hypothesizes that A and B will be measured to be statistically different to p<0.04..."
 
It seems our hearing drops 100-150 dB/octave at the human brick wall point (being around 14-15k for me). So any filter that should cross that drop at higher f must be a brick wall (at 1000-5000 dB/octave). Anything lower slope would be impossible to have any effect on frequency energy. Remaining effects is the echo smearing/ripple/phase, and IMD. I note that I do not hear any difference in the f2 filter. Nothing.

So again it would be nice with an increasing 12-14-16-18k brick wall but then perhaps not a linear phase but minimum phase.
 
Last edited:
0.05 is a high level of confidence.
It's nothing compared to 3σ or 5σ in physics :)

When somebody comes out with a genuinely better audio format or codec or converter or amplifier , it could be ABX tested and still score p>0.05. It could then be dismissed as not proven to sound any better, even when it does sound very slightly better.
Somehow I doubt that anybody decided not to create some audio gizmo just because they couldn't show p<0.05. I'd be surprised if there even was a non negligible number of them that tried the test in the first place. Ok, maybe not for audio codecs, but I don't think that people on hydrogenaudio.org would consider that stifling progress.

By setting a high pass standard, I think the community has stifled progress and development by contriving to prove that everything sounds the same.
Sounds a bit over-dramatic, imo.

The idea that "this is the way we've always done it" is something I've been fighting all my life.
I'd say that ABX tests, p-values and all that is exactly the opposite of "the way we've always done it" in audiophilia world, so congratulations, you've won (or maybe start winning) ;)
 
Last edited:
NEW TEST FILES

Hello all, after a short break that was needed to prepare and test new files, there is a link to new set of test files.


The test files were prepared by @j_j from my original file dchesorig1.wav, which is also included in the test. Zipped is this original file + 4 files with strange names (names generated randomly), and those 4 files use various types of digital filters. Your task is to test (ABX) the dchesorig1.wav file against any of the remaining 4 files, and post the result, if you are willing to participate. My 71 years old ears are able to tell only one of the filter files against the original, however with high success rate 38/48 (p = 0.00003085). I believe that you who are younger will be able to tell the difference in more files. Have fun and looking forward your results.

Note: Please use higher trial count than 10. If only 10 trials were used, please repeat the test and we use the cumulative result.
 
Last edited:
Back
Top Bottom