serial: cast before shifting on port io

Without this we will shift data into oblivion and give wrong results on
some configurations

Signed-off-by: Alan Cox <alan@linux.intel.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>