Only call layout during layout pass

In other cases, let's just adjust the bounds, otherwise we'll conflict
with the measuring pass, causing the width to be 0 after screen
rotations.

Test: manual
Fixes: 204497727
Change-Id: I759f084577814ff1a4372f1d90cd05ce73604950