qos: Disable irq notifier when qos request is removed

When a qos request is added with IRQ affinity, irq notifier
for that irq is added and with out disabling the irq
notifier if the irq is free it is through a warning.
So disable irq notifier when the qos request is removed.

Change-Id: I50faa4ecbe1b632c0f0f203ca52faf18753c33d4
Signed-off-by: Srinivas Rao L <lsrao@codeaurora.org>