Revert "mmc: core: extend SDR104 workaround for other paths"

Commit 8cd402a5 ("mmc: core: extend SDR104
workaround for other paths") would cause SD
card remove detect malfunction. This change
reverts it.

Change-Id: I2a5ecebc8c355ff2b12c72596cb96d0f4f4b819e
Signed-off-by: Can Guo <cang@codeaurora.org>