Ensure address removed from filter accept list

A direct connection not present in the internal
connection manager data structure prevents the address
from being removed from the filter accept list.

Should that occur, go ahead and try to remove it anyway.

Bug: 237353773
Test: gd/cert/run
Tag: #refactor
BYPASS_LONG_LINES_REASON: Bluetooth likes 120 lines
Ignore-AOSP-First: cherry-pick

Merged-In: I0253b9d66b12bbee3c9994298ba054b47cd035dd
Change-Id: I0253b9d66b12bbee3c9994298ba054b47cd035dd
(cherry picked from commit 7a4e504e)
Merged-In: I0253b9d66b12bbee3c9994298ba054b47cd035dd