DAC listening test files - updated with new filters files

[welwynnick]

It depends what your purpose is. With enough samples you can show that statistically a 55% success rate is unlikely to be pure luck, so demonstrating some small degree of audibility. That may be useful from an academic viewpoint. As a guide for a purchasing decision, being able to tell which is which only slightly more reliably than a coin toss when listening intently under ideal conditions means that for everyday use the difference is effectively inaudible.

You can show that 51% success rate gives you p>0.05.
Or would you rather someone tested 5/5? But look what happened when someone did something similar recently.
If someone scores p>0.05 but you don't like the result, do you demand p>0.005?
If someone scores p>0.005 but you don't like the result, do you demand p>0.0005?
Where does it end?
Rigorous scientific research typically draws the line at p>0.05, and with audio testing by ABX, it's a tough line to cross.

 

[pjug]

Rigorous scientific research typically draws the line at p>0.05, and with audio testing by ABX, it's a tough line to cross.

I think p better than 0.05 is easy if the differences are clearly heard (I think 0.05 much too easy). And I don't think asking for something like 38/48 is some kind of unbearable listening test. Nothing that resembles endless.

 

[pma]

5/5

There is simply too high probability that you’ll get it by chance. 16 trials at least, or cumulated result from at least 2 runs of 10 trials. One 8/10 is nothing. This is not about liking the result or not, nonsense.

BTW, @j_j is preparing test files from my original. We are testing it, it might be interesting.

 

[welwynnick]

There is simply too high probability that you’ll get it by chance.

That was kinda my point. I promise to be more explicit in future.

BTW, @j_j is preparing test files from my original. We are testing it, it might be interesting.

What do you hope that the test files will show?

 

[danadam]

There is simply too high probability that you’ll get it by chance. 16 trials at least

Is 12/16 also going to be dismissed? It's about the same probability. Actually, slightly higher.

 

[Thomas_A]

There is simply too high probability that you’ll get it by chance. 16 trials at least, or cumulated result from at least 2 runs of 10 trials. One 8/10 is nothing. This is not about liking the result or not, nonsense.

BTW, @j_j is preparing test files from my original. We are testing it, it might be interesting.

If it is more coming with brickwalls I would like to test 12-14-16-18-20k. I could increase the number of trials as it gets harder eg 20-25-30-35-40.

 

[pma]

By now there is quite strong indication that it is rather a passband ripple that is audible, than the Fc limit >=20kHz. In a properly designed filter. At least for us old guys.

 

[Thomas_A]

By now there is quite strong indication that it is rather a passband ripple that is audible, than the Fc limit >=20kHz. In a properly designed filter. At least for us old guys.

Wouldn’t that throw all known JNDs in the waste? What about IM effects in the ear itself? I probably have read something in the past regarding the slope of the curve at our hearing limits and I guess a brickwall or very steep filter is needed. 8th order at those f is too shallow. Are minimum phase filters coming?

 

[somebodyelse]

You can show that 51% success rate gives you p>0.05.
Or would you rather someone tested 5/5? But look what happened when someone did something similar recently.
If someone scores p>0.05 but you don't like the result, do you demand p>0.005?
If someone scores p>0.005 but you don't like the result, do you demand p>0.0005?
Where does it end?
Rigorous scientific research typically draws the line at p>0.05, and with audio testing by ABX, it's a tough line to cross.

Rigorous scientific research often goes very much further than p>0.05 - that's more of a hurdle to see if it's worthy of further research than a level for a conclusive discovery, and the follow up research . That's especially true if the result is unexpected. Take the OPERA experiment at CERN whose results proved controversial even with 6 sigma confidence. Four independent experiments subsequently failed to reproduce the result with similar confidence, and the OPERA scientists suspect there was a faulty element in their experiment.

Having said that, my argument isn't about the p-value, but what the success rate says about the degree of audibility of the difference, especially for a high p-value. "Clearly audible" by definition has no room for error, so requires 100% success irrespective of p value. You might argue a little wiggle room for the odd incorrectly recorded result. Anything less is going to move through grades like 'often audible', 'sometimes audible', 'occasionally audible' and finally inaudible. We can argue about the labels for given success rates and p-values.

 

[welwynnick]

Rigorous scientific research often goes very much further than p>0.05 - that's more of a hurdle to see if it's worthy of further research than a level for a conclusive discovery, and the follow up research . That's especially true if the result is unexpected. Take the OPERA experiment at CERN whose results proved controversial even with 6 sigma confidence. Four independent experiments subsequently failed to reproduce the result with similar confidence, and the OPERA scientists suspect there was a faulty element in their experiment.

Yes, but this is just HiFi. Turn on the TV and look at what's happening in the World. It's just a hobby. It's not important.
If anyone wants to argue for more confidence, I will argue for p<0.5. Yes, 0.5.
In other words, if you perform a properly controlled test between two devices, and the results say that you PROBABLY can tell the difference between them, that's a good result.
If you want to argue that you want better than p<0.5, then fine, argue away.
Without a blind test, you wouldn't even be able to say "probably", so it's actually a step forwards from subjectivism, and good enough for many people.

 

[krabapple]

You can show that 51% success rate gives you p>0.05.
Or would you rather someone tested 5/5? But look what happened when someone did something similar recently.
If someone scores p>0.05 but you don't like the result, do you demand p>0.005?
If someone scores p>0.005 but you don't like the result, do you demand p>0.0005?
Where does it end?
Rigorous scientific research typically draws the line at p>0.05, and with audio testing by ABX, it's a tough line to cross.

It's p<0.05 , that is, a p less than 0.05 that is an indicator that we are observing something other than randomness/chance. IOW, a difference was 'really' heard.
Though there is still a small (5%) possibility that it's still randomness. That % is considered 'acceptable'.

p<0.05 is the threshold that's hard to cross, for small differences.

A result with a p>0.05, on the other hand, is common for small or nonexistent differences. It indicates that a difference was not 'really' heard.

 

[danadam]

If anyone wants to argue for more confidence, I will argue for p<0.5. Yes, 0.5.
In other words, if you perform a properly controlled test between two devices, and the results say that you PROBABLY can tell the difference between them, that's a good result.

That's 6/10, 9/16 or 11/20. I don't see how that can be described as "probably". And I don't see what would be the point of doing the test at all.

 

[welwynnick]

That's 6/10, 9/16 or 11/20.

That's exactly what I'm saying.

If you score 50% success rate in a blind test, p>0.5, so your result probably comes from random scoring. Therefore the difference isn't audible.

Taking an extreme example, if you score 51% in 1000 tests, p<0.05, and you could conclude that it's vanishingly small but a Just Noticeable Difference.
EDIT: Correction - should read "10000 tests"

Somewhere in between, there are lots of tests where the probability of random scoring is between 5% and 50%.

Therefore the probability of true scoring is greater than the probability of random scoring.

In those cases, there are varying degrees of confidence. You can't say its definitely audible or inaudible, but you can say it's probably audible.

When you get to p<0.05, you're on safer ground. It's still not definite, but you have a high degree of confidence in the result.

There are many cases where you do have to have a result or a decision beyond all reasonable doubt, better than p=0.05, or even better than that.

However HiFi isn't one of those cases. Nobody will lose their life or their livelihood. It's only a hobby.

So it's still useful to get a blind test result that merely shows you can probably hear the difference. It doesn't actually have to be beyond all reasonable doubt.

 

[Thomas_A]

That's exactly what I'm saying.

If you score 50% success rate in a blind test, p>0.5, so your result probably comes from random scoring. Therefore the difference isn't audible.

Taking an extreme example, if you score 51% in 1000 tests, p<0.05, and you could conclude that it's vanishingly small but a Just Noticeable Difference.

Somewhere in between, there are lots of tests where the probability of random scoring is between 5% and 50%.

Therefore the probability of true scoring is greater than the probability of random scoring.

In those cases, there are varying degrees of confidence. You can't say its definitely audible or inaudible, but you can say it's probably audible.

When you get to p<0.05, you're on safer ground. It's still not definite, but you have a high degree of confidence in the result.

There are many cases where you do have to have a result or a decision beyond all reasonable doubt, better than p=0.05, or even better than that.

However HiFi isn't one of those cases. Nobody will lose their life or their livelihood. It's only a hobby.

So it's still useful to get a blind test result that merely shows you can probably hear the difference. It doesn't actually have to be beyond all reasonable doubt.

It's all about power and probabilities. The numbers <0.05, <0.01 and <0.001 are just generally agreed limits for saying "it's probable at that level" vs. "inconclusive result". Then there is a limit of what is practical; 53/100 is inconclusive but 530/1000 is not.

 

[danadam]

Taking an extreme example, if you score 51% in 1000 tests, p<0.05

For 510/1000 p = 0.274. You need 527 for p < 0.05.

, and you could conclude that it's vanishingly small but a Just Noticeable Difference.

No, that's not what JND is. JND is a difference that you notice at least half the time. And because ABX is 2 Alternatives Forced Choice (2AFC) test and "no difference" means 50% success rate, the threshold for JND moves to 75%. So for 1000 trials you need 750 successes to consider it JND. Which kind of makes sense. If you missed it more than 250 times, then it's not really that noticeable, isn't it

You can't say its definitely audible or inaudible, but you can say it's probably audible.

I guess we have different definitions of "probably". For me, slightly above "definitely inaudible" there is "possible but improbable". And at some point, as it moves towards "definitely audible" , it becomes "possible and probable".

When you get to p<0.05, you're on safer ground. It's still not definite, but you have a high degree of confidence in the result.

Yes, some would call this safer ground "probable"

 

[Thomas_A]

I would not call it "probably" but on a stretch "still possible" when there is an inconclusive result with low power. If power is high I would change it to "probably not".

 

[welwynnick]

For 510/1000 p = 0.274. You need 527 for p < 0.05.

It was a typo, and should have read 51% in 10000 tests give you p<0.05. Because of the large sample size in that academic scenario, that means you would be confident that the test result was not achieved with random scoring, so you have successfully detected a difference. I said just noticeable because the success rate was only just over 50%, so the difference must be small and on the borderline of detectability.

No, that's not what JND is. JND is a difference that you notice at least half the time. And because ABX is 2 Alternatives Forced Choice (2AFC) test and "no difference" means 50% success rate, the threshold for JND moves to 75%. So for 1000 trials you need 750 successes to consider it JND. Which kind of makes sense. If you missed it more than 250 times, then it's not really that noticeable, isn't it

I think that's exactly what JND is - where a small difference is subjectively perceivable more than 50% of the time.
Your first reference says:

In psychophysics, a just-noticeable difference (JND) is the amount something must be changed in order for a difference to be noticeable, detectable at least half the time........
The JND usually reported is the difference that a person notices on 50% of trials.......
If a different proportion is used, this should be included in the description—for example one might report the value of the "75% JND".

Your second reference says:

Threshold values may now be determined from the plot by arbitrarily selecting, say 75%, as the threshold value.

It says they MAY be determined, not that they have to be determined. All discussion of JND says more than 50%, meaning p<0.5.
In my extreme example it was actually p<0.05 because the probability of random scoring was low.
I don't like JND, but I think the p<0.5 criterion should apply to audio as well - or at least be the starting point for discussion, rather than p<0.05.

 

[jabbr]

By now there is quite strong indication that it is rather a passband ripple that is audible, than the Fc limit >=20kHz. In a properly designed filter. At least for us old guys.

This is important: a clear and sensible mechanism

 

[danadam]

Because of the large sample size in that academic scenario, that means you would be confident that the test result was not achieved with random scoring, so you have successfully detected a difference.

That's not because of the "large sample size", but rather because you reached p < 0.023. You could be equally (actually more) confident with 15/20 test (p < 0.021). What the "large sample size" gives you, is that you get more confidence in detecting very small effects when they actually exist.

AFAIU (and again, I'm not a statistician) there are 2 types of errors that can happen in a test:

• false positive - you detect an effect when in reality there is none
• false negative - you don't detect any effect when in reality there is one

You adjust your sensitivity (how much you care/want to avoid) to false positives by lowering p and to false negatives by increasing the number of trials. In your academic scenario of p < 0.05 and 10'000 trials you:

• really, really want to detect even the smallest effect when the effect actually exists, i.e. you want to avoid false negative,
• and you care much less (comparatively) whether you detect an effect when no effect actually exists, i.e. you care less about false positives.

I suspect that for p < 0.05 and 10 trials it's the opposite:

• you don't want to "accidently" detect an effect when no effect actually exists, i.e. you want to avoid false positive,
• and you care less (comparatively) whether you miss an effect when the effect actually exists, i.e. you care less about false negatives.

Note that I wrote "comparatively". As p is the same in both scenarios, you care about "false positives" the same in scenario 1 as in scenario 2. It's only your care about "false negatives" that is different, either in one direction or the other.

I presume there is some number of trials that gives you the same "care" about false positives as about false negatives, but I don't know what that number is.

Your second reference says:

Threshold values may now be determined from the plot by arbitrarily selecting, say 75%, as the threshold value.

It says they MAY be determined, not that they have to be determined.

Please, don't pick one sentence out of context. It also says:

In the 2AFC procedure, the percentages range from 50% to 100% as compared to 0% to 100% in the regular MCS psychometric function. That is because as the stimulus becomes too dim to detect the subject is guessing which interval the flash was in. The guessing rate is 50% for two alternatives.

and

Because guessing will lead to a percentage of correct responses equal to the reciprocal of the number of alternatives, the level at which threshold is defined must be adjusted for chance. In the curve below, plotting the psychometric function for a 2AFC experiment, the threshold is defined as 75% correct. This corresponds to a 50% threshold in a psychometric function derived from a yes/no detection experiment.

Let's say we have an experiment where we play a sound and ask a subject whether they hear it. Or another, where we play a reference sound and another sound and ask the subject whether they sound the same. We repeat that 100 times. These are YES/NO experiments. If the subject truly doesn't hear the sound or the difference, we get 100x NO. If the subject has no problems hearing it, we get 100x YES. Somewhere in between there is the sound level or the difference level where we get 50x YES and 50x NO. That's the level where it is heard half of the time and this is JND.

Let's say we repeat this experiment but using ABX, i.e. 2 Alternatives Forced Choice (2AFC) test. This time, if the subject truly doesn't hear the difference, we get 50 correct answers because they have to choose something and by chance will choose correctly half the time. If the subject has no problems hearing it, we get 100 correct answers. The level where they heard the difference half the time (and not just randomly chose) is halfway between 50 and 100, so at 75 correct answers.

In my extreme example it was actually p<0.05 because the probability of random scoring was low.
I don't like JND, but I think the p<0.5 criterion should apply to audio as well - or at least be the starting point for discussion, rather than p<0.05.

I just don't see what would be the difference between that and a single coin toss (beside saved time ).

 

[RandomEar]

I don't like JND, but I think the p<0.5 criterion should apply to audio as well - or at least be the starting point for discussion, rather than p<0.05.

But that would be random chance. If you score 51/100 in an ABX trial your p-value would be 0.46. Clearly, nobody can realistically argue that you "just noticed a difference" if you are only one coin flip away from perfect randomness, right?

Some threshold needs to be crossed to make the plausible assumption that you're not looking at purely random outcomes. At p<0.5, that assumption would never be plausible because for a large number of tests, you can get infinitely close to perfect randomness (p=0.5). If you accept one standard deviation, it would be p<0.32. But that is clearly still very lenient because a typical trial being judged significant actually wouldn't be significant in about one out of three cases. The next "nice round number" is two sigma or p<0.05.

You can argue for any other number in-between, but it depends on whether the majority of people is willing to accept a detection if it were wrong in K out of N cases. There is no correct threshold, but there are wrong ones and p<0.5 is definitely one of those.