drm/amd/display: fix mirror rotation scaling math

Curretly dc will incorrectly calculate viewport when there is
rotation or mirror being applied

Signed-off-by: Dmytro Laktyushkin <Dmytro.Laktyushkin@amd.com>
Reviewed-by: Su Chung <Su.Chung@amd.com>
Acked-by: Leo Li <sunpeng.li@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>