Fix back taking multiple swipes.

Bug: 226417634
Bug: 226723586
Test: Try swiping back on multiple devices. It seems it's easier to
repro on bluechip devices. A good test case for me was to bring up and
dismiss all apps folder repeatedly on a p22 device.

Change-Id: I4dc5f5ee1f67822206a5e0442f26a52114cf63f0