Removing notifications always immediately when swiping them away

When swiping away notifications we were previously relying on the
notification manager to cancel the notification for us. If the state
diverges however (due to bugs) this leads to unrecoverable empty
holes which is super bad.

Merged-In: If63c5434b4e438fea560e0013a12c60655103977
Change-Id: If63c5434b4e438fea560e0013a12c60655103977
Test: existing tests pass
Bug: 33457118