Am I reading this right? The various tweeters were recorded and then played back on a headphone to the test subjects?
I don't want to come across like an idiot, just in case there is something I am not aware and ignorant about, but at face value this experiment sounds as stupid as recording a system playback and then playing it back on YouTube.
Someone please tell me that I am missing something here.
Yes you are missing something here.
Videos and recordings of physical phenomena are not the problem. Most of the experiments I do professionally are recordings of phenomena.
At home, I use a microphone all the time to get valid results under controlled conditions. People on YouTube could too, but they typically don't, instead we get either amateur attempts at 'science', with mics and speakers changing position, the supposed investigator talking over the top of the sound, telling us what we are supposed to hear, etc. Or we get slickly curated marketing videos (one of which you offered as evidence above). Many of the videos are outright manipulated.
The problem is controls. As stated in the second paragraph of the paper, binaural recordings are often used to apply controls to an audio experiment. In this case, there is need to control for a variety of factors and was discussed from the outset of the paper. It seems you didn't read or understand the paper. Or maybe think you could get test subjects to all repeatedly orient their ears near tweeter, or maybe believe all of the research into head and ear shapes don't matter. Or maybe you are just obsessing on materials, which this industry fosters. Or possibly you reject controlled observations if they don't align with your sighted opinions.
Interestingly, this result is aligned with research by Toole and others, where frequency response is a prime factor in perceived sound.
I wonder what happened to that one guy who got it right but couldn't on the next day. Was it just pure luck?
This was explicitly discussed in the paper, is part of sample statistics on a study of this size, that 1 out of 20 trials will report a significant difference even while guessing. You don't understand sample statistics? Are you serious or are you sea-lioning?
People have failed to identify materials used in drivers for ages. Marketing drags us back into confusion. so they can sell exotic materials. Driver integration in the speaker system is way more important, which members on this forum are trying to tell you.
edit: fixed a sentence and typo