Good to know that the mystery has finally been solved.
I was under the impression that +4 dbu would mean higher volume..
Actually, no, as you found out.
What you are setting is
input sensitivity, i.e. input level required for a given output level (@ volume = max), 92 dB SPL @ 1 m in this case. At -10 dBV (300 mVrms), you will need 12 dB less than at +4 dBu (1.228 Vrms). So -10 dBV is in fact the quieter setting.
According to specs, the -10 dBV setting will accept levels up to +6 dBV = 2 Vrms or typical consumer DAC output level (including your STX II's line out), vs. up to +20.3 dBu (8 Vrms) for the +4 dBu setting.
If volume decrease doesnt translate to lower sound quality than I guess I am happy with playing them at 24-30 at+4dbu, instead of 8-12 at -10dbv?
It would certainly still seem adequate to me. Here (Xonar D1) 24-30% would translate to -21.6 dB to -18.2 dB, so you'd still have about the recommended 20 dB of headroom.
However, it would be better if you arrived at this input sensitivity by choosing -10 dBV and turning down the volume dial instead. I am very much assuming that the input sensitivity switch alters analog input stage gain to give input ADC full-scale input at either 2 Vrms or 8 Vrms, and the volume dial is just controlling internal digital level adjustment. A consumer line-level output combined with the +4 dBu setting would thus result in the top 12 dB of ADC dynamic range being wasted, and if memory serves that's only around 100 dB to begin with. In return, ADC noise would be brought up. So you may be seeing higher noise than necessary, and as we all know the JBLs aren't exactly setting any records in this regard to begin with.
For reference, I am using the same range of soundcard output volumes with
* ReplayGain enabled (album gain, avoid clipping), pre-gain = -3.2 dB,
WASAPI (shared mode) output (just as loud as DirectSound, but I figured it would be one less abstraction layer as DS sits on top of WASAPI in the sound stack anyway)
* YouTube volume = 50%
(BTW, get XonarSwitch if you don't have it, the tool has proven quite handy for its ability to set presets alone - I've got ones for 44.1, 48, 96 and 192 kHz @ 24 bit each. Note, if you ever need to set system and card sample rate different from one another, go with card > system. While hardware upsampling is very good, attempting to downsample has resulted in terrible quality for me.)