If the DAC device has no Dolby/DTS decoding capability at all, I would guess the output would be white noise or silence.
What can also happen is the TV has a Dolby decoder (modern ones do) , which if it senses the output will be stereo, downmixes incoming 5.1 bitstreams to 2.0 , which can go out via optical (probably as PCM). Such downmixing can be an iffy proposition, depending on the metadata in the source (mixing engineer's downmixing choices). But that's true of downmixing taking place in an AVR too.