le_audio: Allow broadcast handover fallback device clear

With this change there is possibile clearance of fallback group id in
case if active LE Audio unicast device would change to null.

Tag: #Bug
Bug: 338341279
Test: atest LeAudioServiceTest
Flag: Exempt, simple fix
Change-Id: I07a645493b7f05ae8811650eb205683bb91408e4