serial-uartlite: Fix the unbind path

Currently the clocks are not enabled at probe but when the port is used.
Remove the unconditional disable at remove.

Fixes the below
[   77.660196] ------------[ cut here ]------------
[   77.664749] WARNING: CPU: 0 PID: 1992 at drivers/clk/clk.c:622
clk_core_disable+0x78/0x80
[   77.672892] Modules linked in:
[   77.675930] CPU: 0 PID: 1992 Comm: sh Not tainted 4.14.0 #23
[   77.681570] Hardware name: xlnx,zynqmp (DT)
[   77.685736] task: ffffffc879e2e580 task.stack: ffffff800be30000
[   77.691641] PC is at clk_core_disable+0x78/0x80

Signed-off-by: Shubhrajyoti Datta <shubhrajyoti.datta@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>