I have a stupid idea to remove the need for a switch. Why not have a 3.5mm splitter feed the headphones from each amp. Then program two buttons on a universal remote control.
Button 1 - Mute amps A&B, restart mustic source, Unmute amp A
Button 2- Mute amps A&B, restart mustic source, Unmute...