Workarounds to avoid removing all prefs

Removing all prefs causes ugly animations, so avoid it at all cost
and cache all the prefs (while still added) as long as possible.

Bug: 26271353
Change-Id: I33b84d751938b460f4b66c0158057407dd45d974