Only remove client after the hardware is teared down, so a new client

is rejected before old client is done.

Also check mUsers in all cases to make sure there are no existing client.