On top of that, I don't think my CPU (i7 7700k at 4.5 ghz) is ever close to being stressed.
Latency is kind of a different issue. You can have all kinds of things that get in the way of real time audio without consuming significant CPU time.
I've used LatencyMon to track down problems before.
Though like you said, if larger buffer sizes make it worse it might be something else.