It's the sharp filters that screws with phase, their benefit is better frequency response/lower noise/lower distorsion, slow filter affects phase less but frequency response is worse and digital noise/distorsion comes through. The optimal filter is a balance of all that, and as you say it can be beneficial or detrimental depending on what music is playing.
Early CD-players had extreme sharp filters and a perfect frequency response because of that, but people complained it sounded "digital", possibly because of f*ed up phase.
And yes, the differences are very subtle, but can be easily heard. I have had switchable filters in my CD-players already >30 years ago, I've never preferred the perfect measuring default filter and some of the other filters may almost sound bad so one has to choose careful depending on personal taste.