leaudio: Fix possible error path when switching LeAudio HS

Found on the code review. Probably will not happen just make sure that
when switching group, lets do not make any AudioHAL operations in the
context on inactive group.

Bug: 331775328
Flag: Exempt, day-to-day fix. Regression verified with unit tests
Test: atest bluetooth_le_audio_client_test

Change-Id: Ib6a5b2f859af8862a29c79c9f0b93d73451b842f