Ok, so I don't think that's the issue. I found a song that in the beginning of the song it was just a male singer and when he was loudest you could hear some slight distortion, and I don't hear it with XLR. I put the DAC in pre-amp mode and lowered the volume to -4.0dB and I still could hear it through RCA. A couple times I thought I heard it through XLR but then I would re listen and wouldn't hear it. Part of why I'm worried it's all in my head. But for the most part XLR sounds cleaner and clearer.
So I guess the clear solution is to use the XLR out, but it's annoying. I just ordered some Neutric XLR plugs, I want to make a nicer looking one then the one I was using. I made these out of leftover components and I'd like them to match the rest of the cables.
Any ideas would be appreciated!