...I used to totally disable motherboard soundcard (so its drivers, then TOSLINK doesn't work anymore) and use ASIO drivers...Switching from USB to Toslink does not add drivers, meaningful amounts of jitter, or anything else that could limit sound quality.
In fact, the electrical isolation may reduce interference noise and actually improve sound quality.