It is the most convenient way, just to have the EX5 with its volume control as a "preamp" for all these. I have my HTPC connected via USB, a TV via Toslink, a CD player via Coax and a bluray player via the TV, the Topping's volume control being "master volume". No problems whatsoever.
Oh, and headphones too, plugged in all the time and switched on when needed. The EX5 stores the volume for the headphones and speakers separately.
The volume of the speakers (JBL 308P Mk2, in my case without sub at least yet) is set to maximum. BTW it seems to really be "volume" rather than "gain", because the hiss stays the same if you I them down. I didn't touch the default settings of the speakers yet, waiting for my carpets too arrive before I can make measurements.
One remark (obvious I know but easy to forget): Turn down the volume of the EX5 when connecting speakers and sub for the first time.