Revert "LeAudio: Fix missing audio HAL responses on failure"

This reverts commit 8e65af97.

Reason for revert: Fix for crash.

If there is no active group, there is no audio session.
So le_audio_source_hal_client_ and le_audio_sink_hal_client_ are null

Bug: 271790037
Change-Id: I7cffe8262e4361467ad517cc2661f0ee5230c5a2
Test: Manual
Test: atest BluetoothInstrumentationTests