Do not defer removal for empty tasks or activities

It is possible for empty activities to be leaked if their last
window is removed before animations finish. This change keeps tasks
and activities from having removal deferred if they have no windows.

Fixes bug 19047432

Change-Id: If6562f46bbfcac9ba987e2c834a1d55a9a8f3766