Revert "USB: core: only clean up what we allocated"

commit cf4df407e0d7cde60a45369c2a3414d18e2d4fdd upstream.

This reverts commit 32fd87b3bbf5f7a045546401dfe2894dbbf4d8c3.

Alan wrote a better fix for this...

Cc: Andrey Konovalov <andreyknvl@google.com>
Cc: stable <stable@vger.kernel.org>
Cc: Nathan Chancellor <natechancellor@gmail.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>