qos: Disable irq notifier when qos request is removed

When a qos request is added with IRQ affinity, irq notifier
for that irq is added and with out disabling the irq
notifier if the irq is free it will lead to crashes.
So disable irq notifier when the qos request is removed.

Change-Id: I72818bee97e65ae41f6f07667bf90bd629357caf
Signed-off-by: Srinivas Rao L <lsrao@codeaurora.org>