Here is an article about phase distortion audibility that I was provided with by a kind forum member when I asked the same question some days ago.
Did You notice that back and forth an allpass filter was identified with lesser confidence, the bigger the phase lag was? I read the original publication ( yes, I can ):
http://mue.music.miami.edu/wp-content/uploads/2013/10/MastersThesisDaisukeKoya.pdf
It is public to the public, other than some AES internals.
Read section (5. Results and Discussion). That is what I would call a clear concession of a NULL result. The tests do not in any way lead to a substantial conclusion. All what is left are speculations regarding some other mechnisms in the "auditory system", take note, which were not considered when designing the tests.
"Conducting research of broad test-signal scope and experimental design practicality
resulted in limited results. Obviously,
more refined research utilizing a greater signal
variation base (i.e., more frequencies for sawtooth waves across frequency spectra)
and phase distortion levels are
necessary to ascertain more accurate permissible levels
of phase distortion for these signals."
"The human auditory system was found to be extremely tolerant of even gross
phase distortion effects. Although the impulse test signals. ... Improved irradiation methods,
such as the use of phase-equalized loudspeakers in an anechoic environment, may
also aid in ascertaining more accurate permissible levels."
Why is it so hard, seemingly, to give up an idealistic requirement regarding the
signal, despite the fact that even with compromises the
information is there completely?
Additionally it is impossible to maintain phase anyway. Spatially extended sources would change phase relations with angle, e/g a drum kit, piano, human voice. There is directivity, and so there is phase ondulations. Same with microphones, intrinsically. Same with the mixing guy at his desk, and his maybe "timed" speakers--he is going to move sometimes. That would necessarily spoil all the effort on "time". And the same at home.
The best "time alignment" one could have is to listen to music, when You don't think of possible imperfection of that holy device in front of You. Take a time off and just enjoy the music.