@sarieri : Just 2 cents: What CPU frequency governor do you use? I would suggest to try a frequency governor with fixed frequency, so that frequency switching is avoided. Switching frequency could cause delays which in the USB gadget driver are undesired (CPU must service every single USB microframe). The SSH access event could force the CPU to higher frequency, triggering the delay.
You can try the highest frequency your RPi allows, and if it helps, try lower values to decrease heat.
What is your gadget bInterval (microframe period)?