Headphones is a different experience and really not comparable, for many reasons.
Small driver is a simplification, because size in itself is not what matters, it is the radiation pattern. So a 6" in a horn with large mouth can be much better than a 15" direct radiator. But while most people can relate to a 6" vs. 15", few people have experience with such horns. So it is easier to just use driver size to tell the story.
It is the radiation pattern that makes the difference, and this difference matters from somewhere down in the bass range up to the very highest frequencies. And larger speakers typically have better directivity from midrange and down in frequency.
A better pattern in the low-mid range reduces early reflections and this gives better definition, better clarity, retains timbre better, better transient reproduction.
Having listened to the same speaker, where this radiation pattern can be changed, reveals that this difference affects the sound. Better directivity sounds different from the typical small omni-radiating box speaker. Transients have more realism and impact, rendering of images improves, clarity improves. You can not see this in the frequency response, but you can see it when looking at what happens in the time domain.
The interesting aspect of this is that it is not a matter of size, it is a matter of radiation pattern. So if a small speaker can be made to produce a similar pattern as a larger speaker has, it should be possible to relicate the sound of a large speaker in a small physical size. And indeed, it was possible.