LeAudioService: Do not make device Inactive when AudioManager removes it

AudioManager can remove device during its own restart and shall add it
back when restarted. Therefore, Bluetooth shall not take any actions on
this.

Bug: 397416130
Flag: Exempt, dat-to-day fix
Test: mmm packages/modules/Bluetooth
Change-Id: Ib4d4bee64c2c12bc08fbd5bc591348b21039626d