Typically, noise gets into the analog-side through the USB power. USB power (form a computer) is usually noisy (no problem for the digital audio data) and some computers are worse than others, and some interfaces are better filtered than others.... So you don't know which to blame. It's usually a high-pitch whine. It tends to be more of a problem on the ADC side of audio interfaces because the high-gain mic preamps amplify any noise.
It's not usually "ground loop noise" but it can be.
An interface with its own power supply should eliminate this possibility.
The problem is - You can't simply "use a different power supply" with a USB-powered device because you need the data and power connections.
But, linear or switching can be good.
The noise ("ripple") from a switching supply is high frequency and that makes it easier to filter-out. (You want to pass DC, which is zero-Hz, so the higher the frequency the better.) Plus it's usually above the audio range, whereas 50 or 60Hz (or the 100/120Hz harmonics) is obviously in the audio range.
A linear voltage regulator (purely DC) can help in either case and I've seen a circuit that was "double regulated" with a 2nd (lower voltage) linear regulator in series for the preamp or other "sensitive" circuitry.
