USB: u_bam: Fix NULL pointer dereference on device removal

On device removal, port->gr is accessed to get grmnet and gadget
before calling DMA APIs. port->gr is mainly used for BAM2BAM
transports and it not updated when bam_dmux is used. This results
in NULL pointer dereference if device is removed after bootup.
To fix this use port->port_usb instead of port->gr for this
purpose and later replace all usage of port->gr with port_usb.

CRs-fixed: 673429
Change-Id: Ifc68a9f3137ede19fea6c10d63bfe70b6174da50
Signed-off-by: Manu Gautam <mgautam@codeaurora.org>