Revert "mfd: slimslave: Audio recording doesn't work sometimes"

This reverts commit 62e08ad71751ada1a26a77b0b75e525fbc6aa2fb.

The patch has a side-effect and causes the audio loss during
device switch

Signed-off-by: Jay Wang <jaywang@codeaurora.org>