OK, here is the test with D50. Seems like my memory was faulty as there is a difference:
The increased noise components persisted when using Toslink.
To see if that is created by X-USB or Topping D50, I tested the latter using my Audio Precision analyzer S/PDIF output and same noise was there.
In other words, the X-USB is as good as the usb interface inside D50. It is just that there is extra noise when S/PDIF is used in Topping D50 relative to its USB input, showing once again that async USB is a superior interface to S/PDIF.
Fortunately the levels of new distortion spikes over S/PDIF are close to -130 dB down so absolutely not an audible concern.
Bottom line is what is stated: if you have a good USB interface, use that instead of buying X-USB. It can't do better and might do a bit worse (measurable, but not audibly).