usb: gadget: Fix f_rmnet enumeration with destroyed gbam_wq and ports

Commit '5a8d1e95' 'usb: gadget: Fix race condition in enable\disable
Rmnet device' introduce a bug in which we destroy the gbam ports and
gbam workqueue while these are allocated only once on first and only
bind config.

CRs-Fixed: 544296
Change-Id: Id0ca37119fef107ca632bfd9154449793c7bca66
Signed-off-by: Ido Shayevitz <idos@codeaurora.org>