mmc: sdhci: delay the QoS vote removal

By delaying the QoS vote removal, there is some improvement
in performance in single threaded use cases.

Change-Id: I80545486057c55c697b72b56d57e2ea47cff86b9
Signed-off-by: Asutosh Das <asutoshd@codeaurora.org>
[xiaonian@codeaurora.org: fixed trivial merge conflict]
Signed-off-by: Xiaonian Wang <xiaonian@codeaurora.org>