Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 825a9911 authored by Paul E. McKenney's avatar Paul E. McKenney
Browse files

rcu: Make rcu_start_future_gp()'s grace-period check more precise 



The rcu_start_future_gp() function uses a sloppy check for a grace
period being in progress, which works today because there are a number
of code sequences that resolve the resulting races.  However, some of
these race-resolution code sequences must acquire the root rcu_node
structure's ->lock, and contention on that lock has started manifesting.
This commit therefore makes rcu_start_future_gp() check more precise,
eliminating the sloppy lockless check of the rcu_state structure's ->gpnum
and ->completed fields.  The effect is that rcu_start_future_gp() will
sometimes unnecessarily attempt to start a new grace period, but this
overhead will be reduced later using funnel locking.

Reported-by: default avatarNicholas Piggin <npiggin@gmail.com>
Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: default avatarNicholas Piggin <npiggin@gmail.com>
parent 9036c2ff

kernel/rcu/tree.c

+5 −13
Original line number Diff line number Diff line
@@ -1705,20 +1705,12 @@ rcu_start_future_gp(struct rcu_node *rnp, struct rcu_data *rdp, 
	}



	/*

	 * If either this rcu_node structure or the root rcu_node structure

	 * believe that a grace period is in progress, then we must wait

	 * for the one following, which is in "c".  Because our request

	 * will be noticed at the end of the current grace period, we don't

	 * need to explicitly start one.  We only do the lockless check

	 * of rnp_root's fields if the current rcu_node structure thinks

	 * there is no grace period in flight, and because we hold rnp->lock,

	 * the only possible change is when rnp_root's two fields are

	 * equal, in which case rnp_root->gpnum might be concurrently

	 * incremented.  But that is OK, as it will just result in our

	 * doing some extra useless work.

	 */

	if (rnp->gpnum != rnp->completed ||

	    READ_ONCE(rnp_root->gpnum) != READ_ONCE(rnp_root->completed)) {

	 * If this rcu_node structure believes that a grace period is in

	 * progress, then we must wait for the one following, which is in

	 * "c".  Because our request will be noticed at the end of the

	 * current grace period, we don't need to explicitly start one.

	 */

	if (rnp->gpnum != rnp->completed) {

		rnp->need_future_gp[c & 0x1]++;

		trace_rcu_future_gp(rnp, rdp, c, TPS("Startedleaf"));

		goto out;