Hi, and welcome
If you haven't already, it's worth looking around the site and reading some of the introductory articles and the recurring comments and themes that come up.
Consider what you are asking about: transfer of digital data (moving numbers around, not audio .. not until you get to DAC output). That doesn't have a sound. A USB => Optical adapter does not change the data, unless it's broken, so there is no impact. You get better ground isolation, but that only matters if you have ground noise, which is really obvious and you clearly don't.
The most likely, although unpalatable at first, explanation is expectation bias - we change something so we expect a difference so we hear a difference ... it's annoying, but true.
Connect your Z8 directly by USB and just listen to music for a week. After that time your brain will have adjusted and you'll be OK again.
It's worth thinking about what could, as you ask, be going on otherwise to make a difference. You will find that there are no credible explanations.
Welcome to ASR