I got the popping! It's pretty quiet and it would probably not bother me given the noise floor in the Berwhale household.
To get the popping, I have Windows set to output to the DX5 II and I have MusicBee set to output to the DX5 II using WASAPI Exclusive - with this setup I get a small pop when starting and stopping music playing in MusicBee.
If I set MusicBee back to DirectSound, the problem disappears.
This explains why I have not encountered this issue before. For many years, I have been using eAPO to apply PEQ in Windows and have therefore not been using exclusive audio in any application. I don't see any need or benefit to running exclusive mode drivers for music listening with PEQ applied on the DX5 II, so i'll be leaving MusicBee on DirectSound.