A little footnote from ESS on the sound of jitter:
Technical Details of the Sabre Audio DAC
Martin Mallinson and Dustin Forman, ESS Technology Technical Staff
https://6moons.com/audioreviews/wyred4/sabre.pdf -- footnote 15
(original ESS Company link no longer works - this one does )
"The noise that jitter induces is not easily described: it is not a harmonic distortion
but is a noise near the tone of the music that varies with the music: it is a noise that
surrounds each frequency present in the audio signal and is proportional to it.
Jitter noise is therefore subtle and will not be heard in the silence between audio
programs. Experienced listeners will perceive it as a lack of clarity in the sound
field or as a faint noise that accompanies the otherwise well defined quieter
elements of the audio program."
--
My thoughts on jitter:
Jitter means the sample is sampled "early or late" at initial digitization, or is decoded into the analog waveform "early or late". What happens in between, as long as the sample data is intact, shouldn't matter.
At the ADC, when the analog signal is digitized, the ADC clock can introduce jitter into the data itself. The sample is taken too early or too late compared to a "perfect" clock. The ADC sampled the wrong point in time on the waveform. Now, "jitter" is in the samples, the sample value is too high or too low, and can't be corrected (unless the last clock in the chain can somehow know the error of the original ADC clock and reproduce it).
Between digital interfaces, the receiving interface collects the data packet and moves it to a buffer for further processing. Transmission jitter is gone, a group of samples are "stationary" in the buffer The next clock in the chain introduces its own jitter as the data is clocked out to the next function.
At the DAC, the sample is returned to an analog voltage level, and, again, the sample value can be decoded to an analog level early or late, and be too high or too low compared to perfection (a sample created and decoded at the "correct" time).
Ultimate analysis:
With competent gear, the samples are off by nanoseconds and there's nothing you can do about it.
Don't worry about it unless you think you can hear it, then, good luck.
Benchmark DAC2, 10 years ago or so: