leaudio: Fix handling disconnection of single device

If device got disconnected while CISes where not created,
the very first disconnected device would trigger group removal.
THis patch fixes that.

Bug: 278960455
Test: atest BluetoothInstrumentationTests
Test: atest bluetooth_le_audio_test
Tag: #feature
Change-Id: I25aec2fdc6c24a0c44373607482352c32587c9e4