mdss: mdp: fix invalid error check in rotator module

Invalid input/output format error checks in rotator module
is causing rotation request failure or wrong error message
print. This change fixes such invalid checks in rotator
module.

Change-Id: I0f5f19b65c6017dd16deaa09715d29335e551b01
Signed-off-by: Dhaval Patel <pdhaval@codeaurora.org>