Thanks for the comments everyone! I can confirm that I tried digging for any PCM output level settings but nada on this TV :(
And yep, I reset all gains/settings, have the TV set for 2-ch PCM and everything else disabled. It could perhaps be some weird TV behaviour with ARC and fixed volume...