Remove incorrect optimization in visibility cache

During review when introducing the visibility cache, a suggested
optimization when iterating over users turned out to prevent populating
of the cache for certain multi-user cases.

This change ensures that we pass over all users in both directions,
making for a complete cache.

Test: atest AppEnumerationTests AppsFilterTest
Fixes: 159193336
Change-Id: Ic9ae27f1e1402b42a60a333c3ae99373f4829101