ASoC: codecs: wsa883x: fix swr probe error handling

Remove wsa soundwire device from master device list only during
critical error and not for probe defer.

Change-Id: I5c4f71a4735de4bb459094043d41a6b5c560a6fd
Signed-off-by: Vignesh Kulothungan <vigneshk@codeaurora.org>