That's pretty much what I was thinking, although if the turntable will be the primary source, I might still look elsewhere. One caveat is that I've also read that the volume level through the RCAs is lower, so I wonder if that could be a significant factor in why people think it sounds worse.
The Input sensitivity of the RCA inputs is : 0.775V. Many domestic sources may output lower than this. Professional gear tends to put out 2 volts in comparison.
The Digital inputs appear to be much louder than the RCA Input for "normal" domestic sources. I certainly have to increase the volume significantly when using an Amazon Echo as an analogue source compared to Bluetooth or Optical sources. I don't have a turntable but as you will almost certainly be using a pre-amp for the cartridge there may be an option to set the gain appropriately to match the input sensitivity of the amplifier.
The quality of the RCA inputs is probably determined by the quality of the ADC chip that is used. It is a fairly basic chip rather than some of the chips seen in high end gear. For most people this will be fine and certainly for the majority of users of this amplifier.
The Aiyima D03 is a great little amplifier and has quickly become the default for the various sounds systems we have scattered around the house - TV (Optical), Amazon Echo (Analogue RCA) and PC (USB). We use Mean Well LRS-200-24 Power Supplies with these units with KEF Q-series or Denon (made by Mission) bookshelf speakers.