soc: soundwire: Fix wsa mute issue for stereo playback

When stereo playback start and temperature read
initiated at same time, one of speaker PA bits are
not set in particular race scenario. Handle the
broadcast check in soundwire framework to ensure
broadcast is disabled only when respective gr_sid
slave calls swr_remove_from_group.

Change-Id: I6f82a8c3c0f39dc20c1def09c1728ce4c26c2f5d
Signed-off-by: Laxminath Kasam <lkasam@codeaurora.org>