From years of taking such blind tests, the smaller the impairment, the shorter the delay needs to be. In the case of very small differences such as high bit rate lossy codecs, watermarks, generational loss, jitter audibility, etc., I have found that even half a second may be too long. The switching itself because an event that the brain notes and concentrates on vs what you are trying to find. Most powerful is when the switching is instant where this disturbance is not there and memory of the past event is quite fresh before and after the switch.
When I first took the blind test of speakers are Harman, I found the 3 to 4 seconds an infinity compared to what I was used to. But as you say, for speakers, the difference is usually quite large so it became a small nuisance and did not get in the way of making an assessment.
