xhci: xhci-mem: remove port_arrays and the code initializing them

As we are now using the new port strtuctes the port_arrays
are no longer needed, remove them completely

Signed-off-by: Mathias Nyman <mathias.nyman@linux.intel.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>