@amirm made a couple of very good points, with which I agree.
Dr. Toole was actually saying two distinct things:
One, measurements of non-linear distortion do not do a good job of predicting audibly significant distortion.
Two, with "mainstream" speakers generally, distortion is not an issue.
To be completely honest I am not very comfortable with either of these statements. Measurements of distortion are motivated primarily by the desire to determine how loudly the speaker will play before distortion reaches the level that it matters. With any speaker, there is a volume level where distortion will be so bad that it obviously matters. As such, his second statement could reasonably be interpreted as a statement to the effect that all mainstream speakers can play as loudly as they need to play. Perhaps this is true, but when interpreted in this reasonable way, it is apparent that the statement is sort of silly. Distortion is definitely an issue if a speaker is cranked up higher than the SPL it can handle, and measurements of non-linear distortion do a pretty decent job of finding where this threshold occurs for a given speaker.
It is almost certainly true that with the great majority of speakers, the anomalies in the frequency response are much more audible than the non-linear distortion. It is for this reason that Dr. Toole and many others have made comments similar to the comments he made. This is understandable, and it is appropriate to say something along the lines of what he said, but in saying it, I would try to be careful not to suggest that speakers don't distort badly if you turn them up too loud, or to suggest that measurements aren't useful for finding the threshold where this occurs.