I've never done comparisons, but agree that 320k mp3 is totally listenable, I'll use it when needed to send a work in progress to someone for a listen, to fit in email.
FWIW, I think Monty sets the bar on audio quality to "for 95% of people, 95% of music, 95% of the time" or something. In particular, he said "maybe...
maybe" dither would more important if CD had ended up 14-bit as originally planned. That's putting it too leniently. A digital fade at 14 bits will "tear", so you only need to be on a quality system, in a quiet room, monitoring reasonably loudly, to get routinely annoyed by non-dithered 14-bit. Though I'd agree it drops off quickly as you go to 15 and 16 bits. I consider that a minor quibble, since he's not advocating a change to 14-bit

So, I agree with what he says 100%, even if a fraction of a percent of it is "in spirit".