le_audio: Fix sink hal client clean-up

Advancing new flag encountered test failure in le_audio_client_test.
This is a follow-up fix with aosp/3377853 changes only apply with flag enabled.

Bug: 349310336
Bug: 381054654
Test: atest bluetooth_le_audio_client_test
Flag: com::android::bluetooth::flags::leaudio_use_audio_recording_listener
Change-Id: Iba5ef554ce322a8c60c11eff6545077ebd6382ed