Any device that uses a single supply with a capacitor-coupled output internally will have this issue to some degree, as the output has to move up to about half that supply when powering up. In order to avoid this fundamental problem, you'll need either
a) an output driver (or headphone driver) IC generating a negative supply of its own by means of a charge pump DC/DC, something that has become fairly common in the last decade and change
or
b) a split supply for the entire analog stage
or
c) extra circuitry for automatic output muting.
You'll recognize (a) by the ability to output 2 Vrms on a 3.3 V supply, although the datasheet will generally proudly proclaim a DC-coupled output anyway.
Now bargain basement audio interfaces tend to be hitting their price point by keeping circuit complexity to a minimum and using old, often outdated components. In case of the UM2 or UCA202, that's a trusty TI PCM2900 audio codec, which generates a +3.3 V supply from USB +5 V and generally performs roughly like a 20-year-old onboard codec.
If you want modern(ish) technology for cheap, it has to be in some sort of mass-market item, like a headphone dongle. Maybe not the $5 kind, but possibly a $10 one, and definitely some ES92xx, CS43131 or AK4376/77(A) based job.
The prices for USB audio interfaces with split analog supplies have come down noticeably, and there is hardly any single supply ones left around the 100€ price point. Behringer UMC2xxHD, Native Instruments Komplete Audio 1/2, M-Audio AIR 192|4, that's about it for the big names I think. I have even found a +/-5 V split supply in the 65€ Swissonic UA-2x2, although its 20-year-old Wolfson consumer audio codec only has single-ended outputs so you will get pops if you don't turn down the volume. I think the cheapest options that shouldn't have pop noise ought to be ESI Amber i1 (80€), Focusrite Scarlett Solo 3rd gen (89€), Arturia Minifuse 1 (96€), Tascam US-1x2HR (99€).
If this is too rich for your blood, look for a used CS4272-based deal with split power supply, like a 1st/2nd-gen Scarlett or something.
