As endless tests and research has shown most important for the rating of a loudspeakers are on-axis and of-axis (directivity) responses, then come distortions and phase linearity is finally only the icing on the cake which has very small influence on the audible result (except on rare pathological loudspeaker designs which usually though show up also on frequency and directivity responses).
Yes, Toole is another of the gentlemen whose work and posts i regularly follow. Refer to his book often, and enjoyed the interview with Olive, that Erin recently posted.
I'm totally onboard with the idea smooth directivity is a prime component of satisfaction with speakers.
In fact, my last two years of speaker DIY have been mainly about trying to achieve the best total on-ax and off-ax response i can, measuring outdoors on a spinorama as reflection free as possible.
A little background info....
Like everything in audio, once a certain threshold is met, improvements in a particular aspect such as frequency response or directivity, don't bring much extra to the party.
I had run into that on my DIYs with on-ax frequency response (mag and phase). I'd become adept at correcting minimum phase variations, while leaving the rest alone. Near perfect mag and phase was easy, and getting even more perfect was going nowhere.
So directivity became my focus in search of SQ improvement. And after a couple of years effort, I now feel i've hit the threshold of 'diminishing returns' with directivity/polars.
In my simplified take, overall directivity is a function of the speaker's acoustic design. And as we know directivity can't be changed with processing, other choosing correct xover frequencies for smooth directivity transitions between driver sections, and minimizing lobing.
Imo, we kind get hung with having to make an upfront choice on what kind of directivity we want.....the speaker choice defines it.
Maybe all the cardioid and beam steering stuff will eventually change all that, but i'm not holding my breath....
So I've found my directivity choice in DIY conical MEHs/Synergies. And like said, feel i've taken their polars as far as i can.
And now i'm circling back around to mag and phase, looking at various xover strategies/orders.....mainly because i've moved from 4-way MEHs (counting sub) to 5-ways, and their is a lot more room to move xover points around. (5-way active, FIR processing driver-by-driver.)
Through all this, i must say i simply cannot live without phase linearity any longer;
because A, for however much it helps or not, it sounds awesome,
and B, it's honestly by far the easiest way to achieve great frequency response (if you can afford the latency.)
I've mentioned all that in the hope it lends some credence to my post(s).
Ok, return to topic...
To borrow a phrase from Toole, i think speaker building has it's own 'circle of confusion'.
Can we really assess good directivity until we have good response? Can we assess good response until we have good directivity? Can we do any of that without good bass? Do we need to listen outdoors to isolate speaker from room (i think so, best room in my house is clearly outdoors).
I think phase linearity fits strongly into this circle of confusion.
And yes, determining phase audibility has proven difficult at best as shown by many studies. that imho have failed to meet the necessary conditions on the '
speaker quality circle of confusion'.
Like you say, phase linearity may only be the icing on the cake....but hey, ain't icing the part that makes the cake taste so great !