staging/rdma/hfi1: Add irqsaves in the packet processing path

In preparation for threading the receive interrupt, add irqsaves in the packet
processing path.

When the receive interrupt is threaded, the packet processing path is no longer
guaranteed to have IRQs disabled.  Add irqsaves where needed on several locks
in the packet processing path.  Anything that did not have an obvious, "close"
irqsave in its caller is a candidate.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: Dean Luick <dean.luick@intel.com>
Signed-off-by: Ira Weiny <ira.weiny@intel.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>