People who say that, usually ask for examples where jitter audibility has been demonstrated using a commercial music release and/or hardware.
Jitter audibility is a case, again, of quantity as well as quality of distortion. As in: no one says X can't possibly be audible, X doesn't exist, X can't be measured. But measuring a difference doesn't mean hearing one. X *can* be audible, if it's high enough....but how often *is* X high enough, in practice, with the sources and 'samples' we use?
CD player jitter for example is typically measured in picoseconds: thousandths of a nanosecond.
Maybe as per B&G's 1998 work that you cite, the next time I'm listening to pure sine waves at 8 kHz on different gear (preferably 90s vintage), I'll watch out for jitter?