This is a really good analogy. Buffers are small, temporary storage of data, in this case audio data.
Delay = Buffer size / Sample rate
So, for instance, 96 KHz with a 4096 sample buffer size should give you: 4096 / 96000 = 0.04267 seconds, or 42.67 milliseconds of delay.
Your computer should...