You have most likely, an oxidised/pitted set of speaker relay contacts. Very, common.
The 'burst' of volume is required to 'cut through' the oxide and then all is well until next time or a period down the track.
Prior to turning it up, one speaker is lower in level and/or distorted.
To confirm it is the relay, take the cover off the amp, turn it on and play at the level where the issue occurs. Tap the speaker relay firmly with a wooden chopstick (safety first) and see if it instantly solves the issue. All you need do is either replace (preferable) the relay or alternately, remove it, pop the cover and clean the contacts with some 2000+ grit wet and dry cut into a thin streamer-like strip, folded over and pulled through the NO contacts.