Following this thread closely... lots of golden nuggets!
You mentioned that 0.5 dB differences across a broad frequency range could be heard... especially when the listener was directed to listen for. The 0.5 dB figure corresponds to the acuity recognized by the medical community, which often uses pure tones whose frequencies are broadly spaced.
Since you've likely been involved in innumerable ABX tests, do you find a strong statistical correlation between your measured lab results and human auditions? Which lab measurements tend to produce the strongest effect in test subjects (you've mentioned phase sensitivity... are there others)? Are there measures that you've found are not good predictors of test subjects' likes/dislikes?
Someone once said; "try to do one simple experiment each day". As much as possible, I try to do that. It may be a test on an AP test station, a listening test, or a set of calculations. I am often surprised by the results. Sometimes I am stunned. I am stunned by our ability to hear a pin drop in a relatively noisy room. It has been said, "we are fearfully and wonderfully made". The more I learn about our auditory system, the more I am amazed. There is a trillion to one power ratio between the loudest sounds we can tolerate and the softest sounds we can hear. Our ears can detect a 3 kHz tone that is 30 dB lower than a white noise masking signal (I frequently demonstrate this to high school and university students who visit Benchmark). I do this particular demonstration using an AP test station.
In most cases, our measurement systems can pick up defects that are too small to hear. But, occasionally, we hear something that we did not measure. When this happens, we have to go back to the lab and figure out what we are missing with the measurements. It is always a case of not having done the right measurement. Sometimes the "right" measurement is something entirely new. For example, Amir makes a point of running power amplifier tests at 5 W as well as at full power. The low power tests can reveal defects that may be obscured in the high-power tests. Years ago, Dick Olsher said "the first Watt is the most important Watt". I have written a paper where I contend that "the first 10 milliwatts are the most important". It was an ABX test between two amplifiers delivering 10 milliwatts into a loudspeaker. This ABX test confirmed the audibility of zero-crossing distortion in a class-AB amplifier that had very good full-power measurements. Here is the link:
https://benchmarkmedia.com/blogs/application_notes/power-amplifiers-the-importance-of-the-first-watt
All of our tests tend to be focused on detecting audible defects. We are attempting to build transparent products and this means that we should not be able to hear a change when the product is inserted into the signal chain. We also want to know if we can hear a difference when a measured defect is turned on and off.
In most cases, it is easier to fix a measured defect than it is to prove that it is audible or inaudible. For this reason, we are highly dependent upon measurements. If all goes well, the listening tests confirm that we have not missed something in the bench measurements.
If we were attempting to make products that produced a euphonic effect, we would need to make subjective judgments about what sounds "better". "Better" is in the ear of the beholder and it can and should take you in a variety of different directions. Effects by definition add distortion, frequency response, or dynamic changes to the levels. Effects must be evaluated with listening tests while bench measurements will have a more limited role.
Studios use many devices to create sonic effects. They also use highly-transparent monitoring chains to evaluate what they are creating. Our equipment is often a part of that transparent monitoring chain. If you want to hear exactly what they intended to create, you will also need a transparent playback system. On the other hand, there are listeners that like to color the sound of their playback system to suit their own taste.
