I thought there would be a simple way to have the USB only accessible to the Ultra. I might contact WiiM and make a request to see if it’s something they could do in a future update. Until then it looks like I’m going to have to have a separate “Daddies secret USB” stick.
I really like the feature though and it’s making me want to buy another system for the kitchen now I know I can easily access my tunes via the TV.
Maybe the solution is to submit a Feedback ticket through the Wiim app and see if they have a fix for you or would be willing to include this feature in a future release. Wiim has been more willing than any other audio equipment maker I've ever seen to make software improvements to already-sold audio equipment based on user suggestions.