le_audio: Handle primary device switch while being in handover

In conversation mode from handover context (unicast->broadcast->unicast)
a switch to primary device e.g. speaker should not trigger broadcast
(unicast sink stream suspension).

Tag: #bug
Bug: 329172203
Bug: 339612732
Flag: com.android.bluetooth.flags.leaudio_use_audio_mode_listener
Test: atest LeAudioBroadcastServiceTest#testBroadcastResumeUnicastGroupChangeRequestDriven
Change-Id: I8b82a0151198de02c1d7a985efe915f36b6dc381