I feel Stereophile nailed it in their assessment of the problems of controlled testing.
There is no way I could NOT notice the differences between my Cambridge Audio integrated, Peachtree 220 and my new March Audio P252. But, that is in my own listening room, with familiar music and with my familiar speakers, etc. Put me in a room where any of these variables are changed and I concede that I may have a problems telling them apart. And, of course, the resolving power is hugely important. I may not be able to notice the depth of bass available on the March compared to my Cambridge if they're being played through some mini-monitors or lean electrostatics.
Back in the days of rec audio newsgroup there was a South Florida stereo dealer, Steven Zipser (real name). Anyhow, his organization, Sunshine Stereo, was his house. Tom Nousaine and Steve Maki challanged the Zipser to a little ABX. Here's Tom's write up, in
Audio Critic.
_____________
On Sunday afternoon, August 25th, Maki and I arrived at Zipser's house, which is also Sunshine Stereo. Maki brought his own control unit, a Yamaha AX-700 100-watt integrated amplifier for the challenge. In a straight 10-trial hard-wired comparison, Zipser was only able to identify correctly 3 times out of 10 whether the Yamaha unit or his pair of Pass Laboratories Aleph 1.2 monoblock 200-watt amplifiers was powering his Duntech Marquis speakers. A Pass Labs preamplifier, Zip's personal wiring, and a full Audio Alchemy CD playback system completed the playback chain. No device except the Yamaha integrated amplifier was ever placed in the system. Maki inserted one or the other amplifier into the system and covered them with a thin black cloth to hide identities. Zipser used his own playback material and had as long as he wanted to decide which unit was driving the speakers.
I had matched the playback levels of the amplifiers to within 0.1 dB at 1 kHz, using the Yamaha balance and volume controls. Playback levels were adjusted with the system preamplifier by Zipser. I also determined that the two devices had frequency response differences of 0.4 dB at 16 kHz, but both were perfectly flat from 20 Hz to 8 kHz. In addition to me, Zipser, and Maki, one of Zip's friends, his wife, and another person unknown to me were sometimes in the room during the test, but no one was disruptive and conditions were perfectly quiet.
As far as I was concerned, the test was over. However, Zipser complained that he had stayed out late the night before and this reduced his sensitivity. At dinner, purchased by Zipser, we offered to give him another chance on Monday morning before our flight back North. On Monday at 9 a.m., I installed an ABX comparator in the system, complete with baling-wire lead to the Yamaha. Zipser improved his score to 5 out of 10. However, my switchpad did develop a hang-up problem, meaning that occasionally one had to verify the amplifier in the circuit with a visual confirmation of an LED. Zipser has claimed he scored better prior to the problem, but in fact he only scored 4 out of 6 before any difficulties occurred.
His wife also conducted a 16-trial ABX comparison, using a 30-second phrase of a particular CD for all the trials. In this sequence I sat next to her at the main listening position and performed all the amplifier switching functions according to her verbal commands. She scored 9 out of 16 correct. Later another of Zip's friends scored 4 out of 10 correct. All listening was done with single listeners.
In sum, no matter what you may have heard elsewhere, audio store owner Steve Zipser was unable to tell reliably, based on sound alone, when his $14,000 pair of class A monoblock amplifiers was replaced by a ten-year old Japanese integrated amplifier in his personal reference system, in his own listening room, using program material selected personally by him as being especially revealing of differences. He failed the test under hardwired no-switching conditions, as well as with a high-resolution fast-comparison switching mode. As I have said before, when the answers aren't shared in advance, "Amps Is Amps" even for the Goldenest of Ears.