Stop switching to main contacts list unnecessarily.

Test
Manual: follow repro steps from each bug and verify expected behavior occurs

Bug 31838582
Bug 31829161
Bug 32068858
Bug 32379067
Bug 32377019
Bug 32339019

Change-Id: Ie2760ed144854f8c6e57b6b7ba7353864180c2e7