Why can't you play Hi-Res audio through the optical (S/PDIF) input of an AVR?

[little-endian]

The arrangement described above works fine for CD quality (16/44.1) audio.

Not the actual point, but just bear in mind that what is often loutishly called "CD quality" is actually pretty much "human ears quality". Anything more is technically nice and interesting, but effectively forlorn hope when it comes to audible improvements.

It works also for stereo audio with the following bit depth/sampling rate combinations: 16/48, 16/96, 24/44.1, and 24/48. It ceases to function properly at 24/96, in which case there are quite frequent (and very audible!) "hiccups" in the sound.

That is odd. If 96 kHz / 16 bit works, 96 kHz / 24 bit should do so as well as higher "bit depths" via S/PDIF don't impose a higher signal bandwidth which is determined by the sample rate only. The field for the bit depth for 16 bit gets stuffed up to 24 anyway as the further audio and "auxiliary" bits are set to zero with the clock speed staying the same. Since 24 bit per sample are correctly processed at lower bit rates, maybe it's an obscure limitation after the S/PDIF input or simply a software bug.