IB/mad: Ignore iWARP devices on device removal

When an iWARP device is unloaded, the ib_mad module logs errors.  It
should be ignoring iWARP devices on device removal just like it does
on device add.

Signed-off-by: Steve Wise <swise@opengridcomputing.com>
Acked-by: Sean Hefty <sean.hefty@intel.com>
Signed-off-by: Roland Dreier <rolandd@cisco.com>