driver: soc: adsp vote: Fix to avoid panic on absence of debugfs

Handle cleanup code properly to avoid panic on absence debugfs.

Change-Id: Idfac814439612b5fcb29c2c0b96edba21b6af224
Signed-off-by: Vinay Sudra <vsudra@codeaurora.org>