The best idea we have of the performance of the headphone amp that's built into the M2/4/6 (identical between all three acc. to Motu) is Julian Krause's spreadsheet (
Link).
We can use his data to predict that the M4 will output 0.4V with the RT Closed before clipping:
View attachment 273416
With 0.4V, you can drive the RT Closed to 90dB SPL Peak, so about 75dB Average. That's really not very loud.
If you don't use EQ and listen at moderate volumes, then you may just get by with the built-in Amp, but if you have some recordings with higher dynamic range, or tracks that aren't normalized to close to 0dBFS Peak, then you'll quickly run out of headroom. So yeah, I recommend you get an external Amp.
If you do get an Atom Amp, then connect it to the M4 via RCA, set the M4's Line out to 100% volume, and set the Atom Amp to the lowest gain that'll still give you satisfying volume. Then use the Atom Amp's own volume knob to adjust.