No you can for sure cut the complete silence, you just got to leave everything even if it's -120 dB in, your signal gets folded with the impulse response. If the IR is = 0 there is nothing to fold. You can also test this if you want but i am pretty sure you can cut every sample that says 0 on...