Eliminate latency when resampling is disabled

Since the latency was introduced to avoid egregious errors during
resampling, remove it when we aren't resampling.

Change-Id: Ia8e9a9afb1e16212fec81a8c9c661c598e1b0ebf