+1 on the Alps, for a easy solution. A stepped attenuator would probably be better on channel balance, and other above reasons.
Point to note, staying with lower resistance values (like 10K) is a good idea. A little story here... I got one to use in front of a chipamp board, just happened to be handy, it was a 100K. Didn't worry much about the high R value. Much time goes by, and I do some measurements on the amp with the volume control at lower setting... and I find a roll off above 7khz! Where'd that come from?? Usually, I'd have the pot either all the way up, or all the way down when plugging/unplugging cables, driving it with some source that I'd control for the volume level.
So, I hadn't noticed a lack of highs, till I was using it at a DIY speaker event, with long cables. We were puzzled by dull highs on a speaker, eventually went to the host's AV amp and it sounded much better. I went home to check out my amp, (LM3886, should be a good performer) and found the volume control related problem.
What was going on, was the high resistance of the pot, combined with the longer RCA cables (higher capacitance), was creating a RC filter. When you were at partial volume, it was such that it was down in the audio range, and impacted the frequency response of the amp! I wasn't expecting that at all, started to measure various cables with a good LCR meter, and found that they can be very different, and a 10-12' RCA typically has more C than you might expect. So, keeping the cables short helps (less C) but having a lower input impedance (lower R) also makes whatever RC filter you make, above the audio range.
I wonder if this effect might have something to do with creating some cable mythology... Some amps have very high input impedance, and some fancy audiophile cables might happen to have a lot less C per foot than others? "Ahh, now this cable removes another veil between you and the music...."
Final thought... see if whatever chip/JFET solution you look at has distortion specs, or is measured by someone. I had seen chip "programmable potentiometer" parts years ago, think I remember they were noisier, or had more distortion than most average audio amps back then. I dont' know if there's more "audio high end" versions available now that wouldn't be a step down for your Hypex.