You seem to mix up statistical significance with outcome.I don't like JND, but I think the p<0.5 criterion should apply to audio as well - or at least be the starting point for discussion, rather than p<0.05.
You seem to mix up statistical significance with outcome.I don't like JND, but I think the p<0.5 criterion should apply to audio as well - or at least be the starting point for discussion, rather than p<0.05.
Large samples are probably a bit of a red herring, as a "just noticeable difference" can go from being very low to very high confidence. A win rate that is high or low is also a red herring when taken in isolation. It's always about the win rate and sample size size together, so one may as well focus on just the p value.That's not because of the "large sample size", but rather because you reached p < 0.023. You could be equally (actually more) confident with 15/20 test (p < 0.021). What the "large sample size" gives you, is that you get more confidence in detecting very small effects when they actually exist.
I think this is quite important as ABX testing is usually expected for audio discrimination testing. ABX tests are distinguished by their low probability of false positives, which is achieved at the expense of a higher probability of false negatives. Neither are desirable of course, but if you can get a positive ABX result, it means that is reliable. Unfortunately it's difficult to do that with listening tests because our auditory memory isn't good. Coupled to that, a statistically significant result with very low probability of randoms coring is also expected, so that's three obstacles to getting useful test results. The odds seem to be stacked against.AFAIU (and again, I'm not a statistician) there are 2 types of errors that can happen in a test:
- false positive - you detect an effect when in reality there is none
- false negative - you don't detect any effect when in reality there is one
I wasn't trying to misrepresent what you linked. I read the whole document and thought my quote was actually representative. I don't think it's important though. I said I don't like JND, it's also a bit of a red herring. It doesn't take random scoring and sample size into account. I was being facetious when I mentioned it, and probably shouldn't have. The thing that caught my eye, though, was the criterion that a difference should be detected more than 50% of the time, which is a low bar to pass. I wondered why audio testing had to pass a higher bar? In all the reading I did, JND was always discussed in the context of p>50%. If you want to set the bar at 75% then no arguments. You just have to decide to do that, but I don't think there's anything cast in stone to require it.Please, don't pick one sentence out of context. It also says:
You appear to be trying to come up with some "new" relaxed version of what is statistically significant and what's not based on your expectation of what's critical and what's not. That's not how statistics works. Data is either significant or not, and it doesn't change with how important or unimportant field of study is. Once data passes the threshold of being statistically significant, you can then decide to require a more stringent p-value of, say, below 0.03. But you can't just arbitrarily move the threshold of what is statistically significant because you don't think the field of study is unimportant.When I suggest that audio testing should beat p=0.5 instead of p=0.05, it's because when we want an ask a question where the answer isn't absolutely 100% yes or absolutely 100% no, then it's quite legitimate to ask whether the answer is just "probably" right. Of course it's better to have an answer that is "very likely" or "highly probable" or "almost certainly" correct, but those things are difficult to achieve, and we can actually consider answers given with a lower degree of confidence than that.
I don't think it's black and white.You appear to be trying to come up with some "new" relaxed version of what is statistically significant and what's not based on your expectation of what's critical and what's not. That's not how statistics works. Data is either significant or not, and it doesn't change with how important or unimportant field of study is. Once data passes the threshold of being statistically significant, you can then decide to require a more stringent p-value of, say, below 0.03.
Why not? Is it against the law? Will someone die? Will I ruin my life? In actual fact one can move the threshold if one choses to. Life will go on, and it might even get better rather than worse.But you can't just arbitrarily move the threshold of what is statistically significant because you don't think the field of study is unimportant.
You're making a mathematical, statistical decision a subjective one. Anyone can then pick a threshold they like that proves their point. If you do that, you may as well skip the whole statistical analysis, as it is then of no value.Why not? Is it against the law? Will someone die? Will I ruin my life? In actual fact one can move the threshold if one choses to. Life will go on, and it might even get better rather than worse.
Clearly Josh hasn't been dismissed. Could he have done 16 trials rather than 10? Sure. That's up to him. At this point @pma has looked at possible mechanisms and there is a candidate so further testing can look at than. Also @amirm has the box and can look at that. The only real purpose of p <0.05 is that certain publications want it, and the FDA to approve drugs etc but really just because a drug has been approved doesn't mean its both safe and effective. Same for any study publishing a p<0.05 result for whatever the study is trying to say. You can believe whatever you please for p<0.5 but not many people will believe you and you are highly unlikely to be able to recruit a community of people who believe things based on infinitesimally better than chance. This thread is about @pma tests and his criteria applyWe've gotten so used to drawing the line at p<0.05 that we've forgotten the reasoning. This came home when Josh scored 8/10 in the ADC vs direct comparison. Because that's narrowly above 0.05 it was dismissed.
I do not think anyone will risk die regarding audio (unlike in medicine), but it still remains good to have strict limits and follow the practice in research. Once there are differences, you can add that to the equation along with features, looks and how much money you are willing to spend.I don't think it's black and white.
I don't think anything is black and white.
We've gotten so used to drawing the line at p<0.05 that we've forgotten the reasoning. This came home when Josh scored 8/10 in the ADC vs direct comparison. Because that's narrowly above 0.05 it was dismissed. And many more have also been dismissed, despite representing good data points. Not as good as p<0.05 of course, but certainly better than worthless. p<0.05 gives you a high degree of confidence, and I'm suggesting that there's no actual need for it, just a wish.
If we were start off at p<0.5 instead, there might be some who would accept that, which is OK, and some who would look for more, which is also OK (and quite likely).
Why not? Is it against the law? Will someone die? Will I ruin my life? In actual fact one can move the threshold if one choses to. Life will go on, and it might even get better rather than worse.
Exactly.Anyone can then pick a threshold they like that proves their point.
That may be your impression of what’s happening, but it’s not.Exactly.
That's what's happening already.
By setting arbitrarily high thresholds you're suppressing perfectly good data that contradicts one's opinion.
Eh, no? The threshold is commonly accepted in pretty much all fields of science. You are trying to pick your own one for this specific case to make a point. That is arbitrary.Exactly.
That's what's happening already.
By setting arbitrarily high thresholds you're suppressing perfectly good data that contradicts one's opinion.
Something I already said one month ago, so one may as well focus on just the p value.
www.audiosciencereview.com
www.audiosciencereview.com
Or because there is nothing to detect.if you can get a positive ABX result, it means that is reliable. Unfortunately it's difficult to do that with listening tests because our auditory memory isn't good.
I guess that depends on one's definition of "useful". p<0.5 is essentially random, so it doesn't seem useful to me.Coupled to that, a statistically significant result with very low probability of randoms coring is also expected, so that's three obstacles to getting useful test results.
It's not that I want that, it's simply a necessary adjustment due to the mechanics of the test.If you want to set the bar at 75% then no arguments.
It's only a low bar if you don't do the necessary adjustment. With the adjustment the bar for JND is the same at about 16 trials and becomes higher at 24 and more trials.The thing that caught my eye, though, was the criterion that a difference should be detected more than 50% of the time, which is a low bar to pass. I wondered why audio testing had to pass a higher bar?
No argument with any of that, or many other comments, only the over-arching notion that nothing less than 95% confidence will do.Something I already said one month ago
Or because there is nothing to detect.
I guess that depends on one's definition of "useful". p<0.5 is essentially random, so it doesn't seem useful to me.
It's not that I want that, it's simply a necessary adjustment due to the mechanics of the test.
If I wanted the JND threshold at 75%, then in my earlier experiments I would require 75% yes-es in the yes/no experiment and 87 correct answers in the ABX experiment.
It's only a low bar if you don't do the necessary adjustment. With the adjustment the bar for JND is the same at about 16 trials and becomes higher at 24 and more trials.
Results for all filters:I would start with this, the files can be downloaded from the link below:
dchestest.zip
drive.google.com
20kHz brickwall added
dchesF4.zip
drive.google.com
The original 96/24 is from the source discussed, 30s sample. Spectrum shown in the post above. There are 4 files in the zip, the original and 3 filters. I am not using brickwalls. An example of spectra of the original file and with filters are shown below. So, good luck in your ABX testing! Based on the results, we may add another filters.
[...]
A hint: most of the ultrasonic content is in the right channel, time 25.48 s. The trumpet. Much more than for the cymbals. Below a spectrum after 8kHz HPF.
[...]
| Filter | Correct guesses | p-value |
| F1 | 16/16 | 0.0% |
| F2 | 6/16 | 85.5% |
| F3 | 16/16 | 0.0% |
| F4 | 19/32 | 18.9% |
ECHO_MAG_dB = 8.70434*ln(RIPPLE_dB) - 24.6326
That's an interesting chart, with a lot of phase shift within the pass-band. It's a bit like what some speakers do at the cross-over, or at the LF roll-off........ or if it may be related to the sharply increasing phase difference above roughly 12 kHz:
View attachment 559757
I can't say what of the mechanisms that could be in play; a very slight brighter sound was what was heard. My primary guess will be IMD; I've nailed such tests in the past (IMD) but at the time, due to poorer playback system.Results for all filters:
Filter Correct guesses p-value F1 16/16 0.0% F2 6/16 85.5% F3 16/16 0.0% F4 19/32 18.9%
Equipment: SMSL SU-9n -> SMSL SH-9 -> 7Hz x Crinacle Zero:2
I tried a bit harder for F4 in honour of @Thomas_A, but sadly, there was no significant result for meI wonder if his result could be related to IMD caused by ultrasonic content, like it has been speculated for some positive high res ABX results or if it may be related to the sharply increasing phase difference above roughly 12 kHz:
View attachment 559757
The passband ripple getting mentioned over and over in the thread is below 0.02 dB for this file pairing (and actually below 0.01 dB for most of the spectrum):
View attachment 559760
According to the relevant paper (de Gevel (2017), The Valve DAC; itself citing Lagadec (1984), Dispersive Models for A-to-D and D-to-A Conversion Systems), 0.02 dB of ripple correspond to an echo magnitude of slightly less than -58 dB. If anybody is interested, you can calculate the echo magnitude based on the ripple magnitude using:
Code:ECHO_MAG_dB = 8.70434*ln(RIPPLE_dB) - 24.6326
I'd reframe this slightly in that when a study is designed the p outcome is part of the predefined success criteria, so typically p<0.05 is selected. What you don't do is try to massage data after the fact and use a different p criterion. You **could** try and use p<0.5 but no one will be interested. In a certain field, you could also predefine your objective at p<0.001, so yes bottom line you can't just arbitrarily move the p-value to try and massage the data. To be simple a hypothesis would be "This study hypothesizes that A and B will be measured to be statistically different to p<0.04..."You appear to be trying to come up with some "new" relaxed version of what is statistically significant and what's not based on your expectation of what's critical and what's not. That's not how statistics works. Data is either significant or not, and it doesn't change with how important or unimportant field of study is. Once data passes the threshold of being statistically significant, you can then decide to require a more stringent p-value of, say, below 0.03. But you can't just arbitrarily move the threshold of what is statistically significant because you don't think the field of study is unimportant.
It's nothing compared to 3σ or 5σ in physics0.05 is a high level of confidence.
Somehow I doubt that anybody decided not to create some audio gizmo just because they couldn't show p<0.05. I'd be surprised if there even was a non negligible number of them that tried the test in the first place. Ok, maybe not for audio codecs, but I don't think that people on hydrogenaudio.org would consider that stifling progress.When somebody comes out with a genuinely better audio format or codec or converter or amplifier , it could be ABX tested and still score p>0.05. It could then be dismissed as not proven to sound any better, even when it does sound very slightly better.
Sounds a bit over-dramatic, imo.By setting a high pass standard, I think the community has stifled progress and development by contriving to prove that everything sounds the same.
I'd say that ABX tests, p-values and all that is exactly the opposite of "the way we've always done it" in audiophilia world, so congratulations, you've won (or maybe start winning)The idea that "this is the way we've always done it" is something I've been fighting all my life.