leaudio: Fix RemoveDevice just after Connection Timeout

When Connection Timeout occurs, Android does Direct Connect for 30
seconds.
If during this time user will Forget Device, the device will stay in
the controller allow list.

This patch fixes that.

Bug: 352673441
Test: atest bluetooth_le_audio_test
Flag: Exempt, regression test with unit test new unit tests added
Change-Id: Id383e73038b2ef43d4cdb19b83c3fbbf8a299fd7