Delay the video.You didn't say this but to be clear, the latency cannot be dealt with using faster processors. You still have to wait for samples to arrive to process them. Therefore, the longer the filter, the longer the latency.
What you can do of course process more channels, or more accurate filtering, etc.