Higher sampling rates tend to mean higher noise since the effective noise bandwidth is greater. I have not kept up with audio ADCs, but in general the faster you operate, the higher the distortion. But delta-sigma ADCs tend to obviate that issue since they (over) sample at a much higher rate in the front end anyway. Filter choice could influence the results but I have no experience (or no recent experience) playing with the filters. Higher rates, and correspondingly higher input bandwidth, can reduce aliasing from band-limited sources (as any input should be) but as noted above can lead to in-band IMD that may cause audible differences (I'd expect it would have to be pretty bad, however).
I guess to really dig deeper you'd want to know the exact ADCs involved (the chips inside the boxes) and their architecture (e.g. delta-sigma, SAR, etc.) as well as the buffers. If two different ADC boxes have the same ADC chips inside then something else is going on, whether input buffers and filters, or the way the ADC's output is (digitally, internally) processed before being passed out and down the line.
I may have missed it: what are "digital files" that you run through the ADC? There must be a DAC involved, so are you distinguishing an analogue source (e.g. tape) from a digital source file run through a DAC into the ADC? A number of analogue sources like records (LPs, vinyl), tape, and FM signals can have high-frequency content lacking in typical 44/48 kHz source files, and that HF content could be affecting the ADCs. That would also explain why lower rates sound better, assuming when you drop the rate, the input anti-alias filter is also dropped in frequency and so rejects the high-frequency noise and signals.
ADC overdrive (and related level matching) through the chain are big players and one of my prime suspects "from afar", though of course the input buffers of the ADC could be designed to intentionally color the sound, and it is true enough that different ADCs can have different noise and distortion signatures.
Interesting topic.