At the risk of entering yet another sample rate flame war....
The idea is that higher sample rates allow better filtering of ultrasonic noise by raising the Nyquist frequency (half the sampling frequency). If you want to pass a 20kHz sine wave cleanly and the recording's sampling frequency (rate) is 44.1kHz, you have to construct a filter that lets 20kHz through with no attenuation but that attenuates over -60dB at only 22.05kHz. That's a very steep filter, which is more likely to 'ring' (create short duration oscillations not present in the original signal).
Many DACs already upclock the sampling frequency internally to avoid this; they take a data set with 44.1kHz sample rate and upsample it to 352.8kHz before applying the filtering. I can see this at work by looking up the MPD audio tab in Moode Audio on my Raspberry Pi with TI PCM5122 DAC, for instance.
I think this is more of an issue in music production rather than reproduction, but it could be an issue in certain reproduction (playback) setups. For instance... Back around 1998 or so, I used to master demo CDs for musicians. They'd bring me their stereo mixdown DAT tapes (16bit/48kHz) or WAV or AIFF files and I'd master them to CD so they could take the master CD to a duplication service. I was using a whiz-bang PC with a fancy RME soundcard, running Steinberg WaveLab and Waves VST plugins for digital signal processing. I noticed that if I could work with 24-bit 48kHz (or 96kHz) files to begin with, I'd end up with a smoother and more easily listenable result than if I was given a 16-bit 44.1kHz file to begin with. I would do all my processing in 24 bits (or 32 bit float) and only downsample to 16 bit/44.1kHz when I was ready to print the final mastered file for CD. It sounded noticeably better that way. Why would that be?
Ever since then, my attitude has been that if you can transfer a precious analog master tape to 24-bit/192kHz PCM using great analog gear and a high quality ADC, then why the heck not do that? If you want to make that transfer to DSD128 or even DSD512, well why the heck not? More data is better, because you never know what that extra information might be used for in 10 or 20 years.
As for straight up playback, I noticed that the latest 24/192 transfer of John Coltrane "A Love Supreme" captured the bias frequencies of the tape deck used to play back the master tape. The bias frequencies (one was 38.4kHz) are captured in the digital file. It's useless information since it gets filtered out in the D-to-A process, but it was there in the analog domain when you played the tape, right? We're talking about one of the great recorded masterpieces of the 20th century, so it should be digitized with as much fidelity to the source as we can possibly attain. If you want the ability to play it back that way too, you'll need high sampling rates like 96kHz, 192kHz or whatever. It's technically possible, so why not do it? I would rather hear "A Love Supreme" just like it came from the 4-track tape deck than I would want to listen to a data-reduced copy, especially if it costs me nothing extra to attain that level of fidelity to the source.
Sorry, I got a bit carried away.