I don't know why people are dismissive of 'argumentum ad populum'. Surely it's the listening test advocate's wet dream: a massive sample size; subjects not being influenced by the knowledge they are taking part in an experiment; products being tested in the environment in which they are going to be used; subjects confirming their true preferences with hard cash.
Sure there may be some distorting factors like marketing, but no different from the psychological pressures that any scientific test subject is under (e.g. wishing to please the tester, saying they prefer the product they think they should like, etc.).
It is the basis of the entire science of economics after all: the rational actor.
I am being devil's advocate of course: I think that all preference-based evidence is unscientific because by definition it is subjective, but I just wonder why people would be dismissive of one subjective 'test' but think that another is hard science. Just because it uses the language of science and maybe people put white coats on doesn't make a test scientific. If it were so, science could produce the next hit record, but as we have seen, it fails
totally.
The human being is a moving target, with preferences that change all the time. Only the very basest biological mechanisms are fixed; the conscious brain being modified by the environment, and modifying itself constantly. The listening test subject may already have audiophile Stockholm Syndrome, and introductory verbiage at the start of the paper doesn't make that go away.
N.B. I am not saying that there is not truth, nor genuine timeless preference; just that it cant be ascertained scientifically. Audio must be engineered, based on logical criteria - and that is what happens: listening tests just oscillate around central core engineering principles, like linearity and flatness of frequency response. Those characteristics weren't arrived at by listening: someone realised what an audio system was supposed to do and drew it out on paper.