drivers: fsa4480: Fix USB re-enumeration failure after PDR

USB re-enumeration failure is coming post PDR due to
switch settings being reset in fsa unreg notifier.
Reset switch settings only for audio adapter to
resolve issue.

Change-Id: I2462a2f1d3679adbd71dd1a7abd30726f7c68e6a
Signed-off-by: Vatsal Bucha <vbucha@codeaurora.org>