Do no try to unbind when unbound

This causes an illegal argument exception, so instead we will just skip
the operation if we know we're already unbound.

Bug: 144140043
Test: manually disable twice and see no crash
Change-Id: Idbb07c47dca43f3495703f983a7e2710ffa9dbe3