Got the SPDIF working by switching to USB power supply only. This solution works actually pretty well now in my case. I have a Matrix X-SPDIF converter hooked to my PC, which has two outputs:
1. AES -> for my digital active speakers
2. Coaxial -> connected now to Khadas, which is connected to Atom amp
The PC doesn't now obviously recognize the Khadas, but it doesn't matter. I can have the X-SPDIF on all the time and when I want to listen with headphones, I just simply turn the Atom amp on. Voila!
Now I need to buy one of those 'USB condoms' which strip away all the data and allow power only, so I can get power to Khadas from the PCs USB port...