As to why we need fast switching ... Here is article on how we our memory of auditory information works. The TL; DR version is that after half a minute, we will not remember the details of the sound we heard, only its abstract representations.
Some excerpts from the article ...
Preperceptual Auditory Storage
Preperceptual auditory storage retains the uncategorized representations of auditory inputs that have not yet been fully processed (Massaro 1975) and is also referred to as short auditory storage (Cowan 1984).
...
The duration of preperceptual auditory storage is very short. Most researchers agree that it lasts less than 300 ms.
Synthesized Auditory Memory
The auditory features stored in preperceptual auditory storage can be further analyzed to form integrated representations of sound.
...
The term “synthesized” refers to the process in which auditory features such as pitch, loudness, and aspects of timbre are analyzed and combined into integrated auditory representations. The duration of the synthesized auditory memory appears to vary from less than 1 s up to 30 s, depending on how it is measured, but it is most often found to be several seconds (Cowan 1984).
Generated Abstract Memory
The integrated representations in synthesized auditory memory can be further processed to form abstract representations in generated abstract memory (Massaro 1975). The abstract representations are considered to be domain general, meaning that they do not carry information about specific sensory details. Thus, abstract representations generated from each sensory domain (hearing, vision, touch, and so on) are all stored together in the generated abstract memory.
In more recent literature, generated abstract memory is often referred to as “the focus of attention” and is reported to have a core capacity of three to five items when various memory strategies are controlled (Cowan 2001). It is thought that information must be saved in generated abstract memory before high-level thinking about it can occur.