Don't remove bubbles after mode switching

We currently remove TaskViews as part of mode switching cleanup, but that
eventually removes existing bubbles which means bubbles are lost after
mode switching.

As a short term solution, this CL avoids removing the bubble if the task
is removed as a result of mode switching. A longer term solution is to
refactor the task views outside of the expanded views so they don't need
to be removed as part of mode switching. See b/316358859.

Fixes: 316358859
Test: Manual
      - Add bubbles
      - Switch modes
      - Observe that bubbles are not removed
Change-Id: I382725e318bc441eb9e8b277e615ca21fdbdc871