I have my EX5 working with TOSLINK from a recent Asus motherboard and have gotten 24 bit / 96khz working.
Getting a new cable is a good idea, they are cheap and it will eliminate one source of failure.
Troubleshooting steps:
1. If you haven't already tried this, set 2 channel PCM at 16 bit with 44.1 or 48 khz sampling rate, don't try anything else until you get the basics working.
2. Make sure the cable ends are aligned correctly and FULLY seated in the sockets, sometimes the optical cable will falsely seem to be in but will still be 1-2mm out, usually the socket will have a soft click first and then a firmer second click before bottoming out.
3. Once you get CD quality audio working, then try higher bit depths and sampling rates.
You should be able to get 24bit / 96Khz but anything above that is out of spec. Though 96khz should be supported, I have encountered issues with using it before on some DACs. 32bit or 192khz will not work.