It is a very fair question about these various transmission media to the DAC. USB is easily the most popular and widespread. Many DACs support only USB, some maybe two of the choices and not many support 3 or more. So, objective comparisons via the same DAC are problematical in trying to answer the question completely of "which is best sounding?"
It is also a question of the specific implementation. As always, a specific piece of gear might not tell us what is possible in a best case scenario with a particular transmission protocol. Or, a specific piece of gear with a well implemented transmission protocol might mislead us about other, inferior implementations. Books/covers?
My impression is that plastic fiber Toslik is by far the least respected among those you mentioned. It is limited in bandwidth - 96k stereo, I believe - and seems to populate only mid-fi and lesser gear. Glass fiber optical, like AT&T and Theta's Single Mode, was once considered "top dog" in some of the best gear, but those have disappeared. I think there might still be glass fiber used in some pro recording setups.
Coax was once demonized by measurement gurus as high jitter in the early days of separate DACs. But, Amir's measurements of his Levinson DAC indicate it can indeed be pretty good, at least in that one implementation. However, Coax does not handle above 192k and it is stereo only on each wire. It does not seem to be the path to the future, and most PCs do not have coax out. It is more for legacy DACs with coax input.
USB is justly the most popular in consumer gear. It is cheap, reliable and has excellent bandwidth for most all hi rez audio available today, even in Mch. And, it is two-way, unlike coax. That has facilitated the asynchronous USB protocol, now almost universal, which effectively eliminates jitter by making the DAC clock, not the player clock, the master, as Amir points out. Its major downside in use is that it is limited to 5 meters in length without special extenders, some of which may rely on an Ethernet intermediary.
Yet, it is a favorite passtime of computer audiophiles to constantly bash USB with unproven claims of this and that deficiency, requiring this and that add on dodad to "fix" its "known" problems. Many think it the devil incarnate, "not suitable for audio", based on no or precious little objective data. But, I am happy to use it and it works great for me without tweaks or overpriced cables. Again, that might be implementation dependent. My DAC is claimed to be galvanically isolated, which might give me above average results.
I first got turned on to USB in playing ripped Blu-rays from a hard drive to an Oppo 93. Many friends and I tried this, and not one of us ever heard or saw the slightest audio/video difference between that high bandwidth playback via synchronous USB 2.0 and the same silver BD disk via the same player, all with very unexotic cables and gear. Asynchronous USB is purportedly better still than synchronous was. So, I am totally unmoved by the unsubstantiated USB bashing everywhere until such time as someone can show me why it is inferior.
There is not much in consumer gear yet for Ethernet directly to the DAC, and what is available tends to be pricey, except in HT AVRs, etc. But, it it is somewhat trickier in setup than USB. The HT and inexpensive versions generally require DLNA/UPnP software protocols which are not very robust. Objective data on Ethernet's superiority to USB is something I have not seen anywhere. But, Ethernet has no length limits in the typical home plus very high bandwidth, but it is not plug' play like USB typically is.
HDMI is not common for audio-only to a DAC. But, it is a decent, if typically higher jitter, protocol in HT environments.
There are also "I2S" types of transmission involving the transmission of clock pulses from a player to a DAC, both properly equipped for that, usually involving a DIY retrofit with PCs. Various signal cable scenarios are used with that, coax often, with a separate lead for the master clock pulses from the player. Allegedly, it can improve jitter performance, but there can still be jitter on the clock feed itself. I do not see what it offers over properly isolated asynch USB. Best to have the master clock very close to the DAC chip inside the DAC, I think, as with asynch USB.
For mysterious reasons and without proof, audiophiles seem to raise unfounded issues about data integrity all the time. Forget about it until you see proof positive, which after many years still has not materialized. Also, asynch USB has verifiably the lowest jitter, yet computer audiophiles everywhere (even a few in this forum) continue to talk about jitter as a prevalent, unsolved problem with USB. It 'taint so.
The esoterica about the "awful downsides" of USB relate mainly to noise transmitted via the power, not the signal, leads in the USB protocol. I, personally, do not feel that is an issue with proper implementation in the DAC.
But, of course, we have all the mystical, unmeasured, unproven claims by Swenson and many others about why their magic USB add-on box works wonders in overcoming the nasties in USB. Don't fall for that, unless, miraculously, Swenson can someday fulfill his promise to provide measured, objective evidence to back up the claims he seems to have come upon while listening to his 1950's Lowther drivers in cardboard tube baffles suspended by wires from his ceiling.
.
