Either your Multimeter is faulty or your Power supply, maybe you also got some noise getting into your test leads somewhere. the psu isnt grounded, ac leakage is normal and can mess up your results.
i tested mine.
(!! some of it could be from my setup and equipment, im sitting on budget stuff. !!)
equipment used: Unit-T UT61E, Siglent SDS1204X-E, EastTester ET5420 Electronic Load, probes set to 10x. dodgy and long wiring all over the bench.
No load:
37.96V
pretty stable over a timespan of 14 seconds (2V/div)
Noise wasnt too bad either
20Mhz BW limit, 17mV at a few hundred hz.
FFT up to 200khz, ive seen worse.
Constant 4A Load:
Voltage was 37.67V,
also quite stable i dont saw any movement over 14sec (again 2V/div due to my offset range wasnt big enough on 1V/div setting)
noise and ripple increased. roughly 40mV at inner part. looks extreme but keep in mind this is a very low level signal.
And where is all that HF crap coming from? right, the SMPS switching frequency.
its at ~155khz with a bandwidth of 20khz.
the PSU got a little warm after a few minutes under 4A load, nothing to worry about. most notebook psu's doing much worse.