[sf] update latch time only for layers that will not be latched

Latching happens after the transactions are committed, so fix an
issue where we passed in the incorrect latch time for buffer layers
that was used to classify jank.

Note: this will probably cause a perf regression but this is fixing
an incorrect jank classification caused by
Ie211aa3bd5821f6052cf84a62a2e245132a19d90

Bug: 270041584
Test: presubmit
Change-Id: I632a8ee6a1ed804a17a001e87c06e617ae361b25