Use bitwise operator to remove pointers

After we converted to std::bitset, we can now use a bitwise operator to
remove pointers instead of doing a loop.

Bug: 211379801
Test: m inputflinger_tests && $ANDROID_HOST_OUT/nativetest64/inputflinger_tests/inputflinger_tests
Change-Id: Ib95f7c67a9d4ee493fa913381953b092a692669c