pinctrl: fixup problematic flag

This removes the set_irq_flags() call that unfortunately slipped
into the BCM NSP driver.

Reported-by: Stephen Rothwell <sfr@canb.auug.org.au>
Cc: Yendapally Reddy Dhananjaya Reddy <yrdreddy@broadcom.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>