viafb: Do not remove gpiochip under spinlock

gpiochip_remove() is not meant to be called with interrupts disabled, and
there is no need for the lock here in any case.

Reported-by: Bruno Prémont <bonbons@linux-vserver.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>