For example, would
this switcher or
this splitter add noise or distortion to my audio system? My intuition tells me no as the former just uses a switch to change outputs, so there's no room for it to add noise or distortion while the splitter shouldn't because it has to do is exactly copy the bits coming from the input and paste it to three different outputs.
Neither will add any noise nor distortion.
Either of them can add some very small jitter but the it is the DAC's task to remove that.
Optical splitters (with a power supply) will have 1 receiver that outputs a High/Low signal which then gets fed to 3 (or maybe even 1) LED driver which switches on/off the transmitters (LEDs). Nothing is added processed or anything. receiver -> driver -> transmitter
The optical switch (with a power supply) is doing it the other way around. It has 3 receivers and a 'switch' that lets the selected input go to the transmitter.
That switch can be a real switch, relays or digital ports depending on functionality. 3 receivers -> switch (in whatever form) -> driver -> transmitter
Also here everything is digital. No noise added, nothing is processed nor 'decoded and re-encoded).
Of course there will be a different optical waveform. The light intensity will have changed, the digital waveform may have changed slightly (signal width, duty cycle, jitter) but this is all digital and does not affect the encoded bits that represent data. That data is not just audio but also lots of other data.