Apply correct restoring logic.

We need to continue on the work left from last time, rather than blindly
believe everything is restored by Android.

Test: Manual test & auto test.
Bug: 38321082
Change-Id: Ic9f80ab16a203a1526c7d23d86512b841c750366