Revert "Defer stop for all activities becoming invisible"

This reverts commit b824f87c (for b/80313326)

Reason for revert: Cts issues (b/110032866)

Bug: 80313326
Bug: 110032866

Change-Id: Iaca3a8650d30b475ef4331e98b4b14377072ee4d
(cherry picked from commit f49b106d)