I think it's the combo of the altered frequency response, distortion, and stereo imaging. Also for certain genres, the non-brickwalled masters for vinyl compared to digital. Don't know about anyone else's setup but I was able to compare a release where I know the masters are the same between the two mediums, and I noticed the vinyl setup had warmer bass (distortion, bass bump) and rolled off upper midrange/treble delivering a smoother sound. It's certainly less accurate but I can totally see how this is pleasant for some genres. The stereo image is also much less stable, like slapping a stereo widener effect set to 10-15% on the master. Sounds that are 100% mono in digital seem a bit unnatural sometimes. That lack of precision in vinyl can make the sound more enveloping and organic. I do like this for older music because they can sound a bit sterile without this filter.
The problem I think is that nowadays artists are free to use any of the many distortion/stereo imaging plugins to make songs sound exactly how they want—adding the additional filter of vinyl playback on top only serves to place the reproduction further away from the artist's intent. If artists want to add that analog vibe digitally they easily can (exaggerated case in point, lo-fi hip hop! This might come as a shock to boomers but nowadays people are seriously adding vinyl crackle/pop sounds and wow/flutter to their songs.)
I make liberal use of distortion and EQ in my music productions so it's not that I think they are bad. But when it comes time to reproduce the artist's work it should be through a calibrated, accurate, and low distortion system to best convey what they wanted you to hear. No different than viewing photos/art/movies through a calibrated screen.