At the risk of losing your discrete selection functionality, you could try "programming" one of the other buttons -- like B-1 -- to the toggle feature. My guess is it will set B-1 to toggle and remove all functionality from A-1 and A-2, but maybe you'll get lucky and get all three functions.
To "program" I just hold down the target button while pointing at the switch. After a couple seconds the Power light will blink twice indicating the program state has changed.
Another thing that might be annoying for you: Once you do this, the display only shows the selected channel for a few seconds. Then it jumps to show the programmed remote button. For example, in its current state, remote button A-1 toggles my inputs. I hit A-1, the device shows A-B, then after about 3 seconds it shows A-1. Hit A-1 again, it shows A-C for a few seconds, then A-1 again. A bit annoying. At this point, I just turned off the display (remote button B-6) and rely on the LED.
It's a shame this thing is so fiddley with the remote. The default functionality of discrete input buttons and constant input display would work great for my setup.