I was doing graduate-level statistical analysis and statistical software development back in the early 1980s. I think I can keep up.
The fact that a patient reports a change to their condition doesn't mean that some recently undertaken therapy is the cause of the change -- or even that there really was a change. With no treatment at all, how many of the plantar fasciitis sufferers would have reported that their condition got better, worse, or remained unchanged over the that same time period? If a patient was given a "special shoe" that the doctor said should reduce the pain, how many of them would believe that they were dealing less pain, even if the shoe was secretly just some random sneaker bought at Walmart? That's why medical researchers rely on blind studies and statistics.
It's not about predicting any individual's responses. It's about determining whether differences you think that you hear 'mysteriously' vanish when you don't know which DAC you are listening to.
You're free to evaluate audio components by whatever criteria you want. If you want to switch between two DACs and then pat yourself on the back for hearing the differences that you hoped or expected to hear, go for it. But it's still bad science.
Many people think that doing statistics is same thing about knowing statistics.
But it isn’t.
On average we know that we cannot change global scores on plantar fascitis, but we are completely sure that insoles or shoe adaptations can change things: the problem is that for many people pain became worse, temporarily.
Not knowing which patient can obtain benefits from insoles, we usually tell the patient what we know and let to decide by itself.
A statistical test on a population about DAC audibility will show a correlation: if is 0 or low enough correlation between some variable (give a number between 0 and 10 for the quality of sound for example)and some different dacs we can conclude that no correlation have been found, thats all. If you take the conclusion that no one can feel a change of quality on the dacs, is another conclusion that is out of the possibilities of your trial.
You can also made another test with two different dacs and giving some samples to ask if the subject is listening for different things or it thinks that are the same.
Again, your statistical conclusion can be low correlation between dacs and influence on the sound (which is more focused on the primary question of the post, don’t asking for quality just differences).
Once more, a conclusion that no one can distinguish between two dacs is outside the statistics: only you have some probability strength asserting that different dacs don’t change perception on a population. Letting you with a strong argument if someone ask you for some DAC for a public event to answer “no dac will provide the public better experience”. On an individual you can carefully explain the obtained statistics and let him tray by itself. As I mentioned before the tests don’t tell you anything about this particular person.
I let you a funny problem that is given again and again to students to begin learning which knowledge can be obtained from probability and statistics:
You have a circle of radius 1 m, and another of radius 2 m with the same center (the little inside the bigger).
Which is the probability of finding a point inside the smallest circle?
