The best place for this kind of device is between pre & power amps or in an EQ loop (where present on an integrated amp), with some simple method of calibration. It would also benefit from a decent PSU and not running it off cheap switch mode power supplies.
Putting any additional resistance (cheap relays*) in the speaker chain is never a good idea.
*those relays might be cheap iron framed relays which Self has demonstrated introduce a lot of distortion because the signal is being passed through the iron frame.
I currently switch between two pairs of speakers with a Beresford TC-7210 - this has a couple of decent quality rocker switches on the front panel.
It's actually intended to switch 2x amplifiers to one pair of speakers, but works just as well the "wrong" way round.
View attachment 155358