This is a very interesting observation, and it correlates somewhat with my experience. I have both a set of two-way speakers and a set of single-drive speakers in a bad room. I experience much less listening fatigue when listening to the single driver speaker.
Please note that this is my own reflections on the topic, but I am wondering if the step response is relevant here. The single driver speaker has a perfect step response by design, and the two-way speaker is, due to the crossover, first firing the tweeter and then the woofer, not producing a single wavefront to the listener. From psycho-acoustic research we know that humans are extremely good at handling and separating direct sound from speakers and reflections by "assembling" the music using the brain. It is believed that this process are one of the contributors to listening fatigue. However, the reflections from a two-way speaker will be more complex due to the time-delay and phase-issues from the two-way speaker. I know that this is controversial for several members of this forum, as some research indicates that step-response is not a big factor in speaker design. But, these tests are done for a very limited time-span, and listeners will for this reason not experience listener fatigue.
The soundstage and more "relaxed" sound from the single driver is something I appreciate, and I can separate them from the two-way speakers each and every time. However, for more complex music, I tend to choose the two-way speakers.