Fix a cosmetic bug where text would blink on space swap

On space swap or space removal in some occasions there would be
a small blink on the screen. This change fixes that.

Change-Id: I486cbcc5dfcafd531b3fa92c4f9a3255832f22cc