Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 440ddad5 authored by Chuck Lever's avatar Chuck Lever Committed by Anna Schumaker
Browse files

xprtrdma: Don't post a LOCAL_INV in rpcrdma_register_frmr_external() 



Any FRMR arriving in rpcrdma_register_frmr_external() is now
guaranteed to be either invalid, or to be targeted by a queued
LOCAL_INV that will invalidate it before the adapter processes
the FAST_REG_MR being built here.

The problem with current arrangement of chaining a LOCAL_INV to the
FAST_REG_MR is that if the transport is not connected, the LOCAL_INV
is flushed and the FAST_REG_MR is flushed. This leaves the FRMR
valid with the old rkey. But rpcrdma_register_frmr_external() has
already bumped the in-memory rkey.

Next time through rpcrdma_register_frmr_external(), a LOCAL_INV and
FAST_REG_MR is attempted again because the FRMR is still valid. But
the rkey no longer matches the hardware's rkey, and a memory
management operation error occurs.

Signed-off-by: default avatarChuck Lever <chuck.lever@oracle.com>
Tested-by: default avatarSteve Wise <swise@opengridcomputing.com>
Tested-by: default avatarShirley Ma <shirley.ma@oracle.com>
Tested-by: default avatarDevesh Sharma <devesh.sharma@emulex.com>
Signed-off-by: default avatarAnna Schumaker <Anna.Schumaker@Netapp.com>
parent ddb6bebc

net/sunrpc/xprtrdma/verbs.c

+2 −20
Original line number Diff line number Diff line
@@ -1695,8 +1695,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, 
	struct rpcrdma_mw *mw = seg1->mr_chunk.rl_mw;

	struct rpcrdma_frmr *frmr = &mw->r.frmr;

	struct ib_mr *mr = frmr->fr_mr;

	struct ib_send_wr invalidate_wr, frmr_wr, *bad_wr, *post_wr;



	struct ib_send_wr frmr_wr, *bad_wr;

	u8 key;

	int len, pageoff;

	int i, rc;
@@ -1728,22 +1727,6 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, 
	dprintk("RPC:       %s: Using frmr %p to map %d segments\n",

		__func__, mw, i);



	if (unlikely(frmr->fr_state != FRMR_IS_INVALID)) {

		dprintk("RPC:       %s: frmr %x left valid, posting invalidate.\n",

			__func__, mr->rkey);

		/* Invalidate before using. */

		memset(&invalidate_wr, 0, sizeof invalidate_wr);

		invalidate_wr.wr_id = (unsigned long)(void *)mw;

		invalidate_wr.next = &frmr_wr;

		invalidate_wr.opcode = IB_WR_LOCAL_INV;

		invalidate_wr.send_flags = IB_SEND_SIGNALED;

		invalidate_wr.ex.invalidate_rkey = mr->rkey;

		DECR_CQCOUNT(&r_xprt->rx_ep);

		post_wr = &invalidate_wr;

	} else

		post_wr = &frmr_wr;



	/* Prepare FRMR WR */

	memset(&frmr_wr, 0, sizeof frmr_wr);

	frmr_wr.wr_id = (unsigned long)(void *)mw;

	frmr_wr.opcode = IB_WR_FAST_REG_MR;
@@ -1768,8 +1751,7 @@ rpcrdma_register_frmr_external(struct rpcrdma_mr_seg *seg, 
	frmr_wr.wr.fast_reg.rkey = mr->rkey;

	DECR_CQCOUNT(&r_xprt->rx_ep);



	rc = ib_post_send(ia->ri_id->qp, post_wr, &bad_wr);



	rc = ib_post_send(ia->ri_id->qp, &frmr_wr, &bad_wr);

	if (rc) {

		dprintk("RPC:       %s: failed ib_post_send for register,"

			" status %i\n", __func__, rc);