Clear all button should not be swipable

Unlike other task items in the recycler view, the clear all button
should not be swipable as you obviously can't remove the button.

Bug: 114136250
Test: Go to recents, attempt to swipe clear all, does not work
Change-Id: I0ab00c03b697f2174431b69bbf758c3ff104db97