Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit bd7af846 authored by Paul E. McKenney's avatar Paul E. McKenney
Browse files

rcu: Switch __rcu_process_callbacks() to rcu_accelerate_cbs() 



The __rcu_process_callbacks() function currently checks to see if
the current CPU needs a grace period and also if there is any other
reason to kick off a new grace period.  This is one of the fail-safe
checks that has been rendered unnecessary by the changes that increase
the accuracy of rcu_gp_cleanup()'s estimate as to whether another grace
period is required.  Because this particular fail-safe involved acquiring
the root rcu_node structure's ->lock, which has seen excessive contention
in real life, this fail-safe needs to go.

However, one check must remain, namely the check for newly arrived
RCU callbacks that have not yet been associated with a grace period.
One might hope that the checks in __note_gp_changes(), which is invoked
indirectly from rcu_check_quiescent_state(), would suffice, but this
function won't be invoked at all if RCU is idle.  It is therefore necessary
to replace the fail-safe checks with a simpler check for newly arrived
callbacks during an RCU idle period, which is exactly what this commit
does.  This change removes the final call to rcu_start_gp(), so this
function is removed as well.

Note that lockless use of cpu_needs_another_gp() is racy, but that
these races are harmless in this case.  If RCU really is idle, the
values will not change, so the return value from cpu_needs_another_gp()
will be correct.  If RCU is not idle, the resulting redundant call to
rcu_accelerate_cbs() will be harmless, and might even have the benefit
of reducing grace-period latency a bit.

This commit also moves interrupt disabling into the "if" statement to
improve real-time response a bit.

Reported-by: default avatarNicholas Piggin <npiggin@gmail.com>
Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: default avatarNicholas Piggin <npiggin@gmail.com>
parent a6058d85

kernel/rcu/tree.c

+15 −38
Original line number Diff line number Diff line
@@ -2334,34 +2334,6 @@ rcu_start_gp_advanced(struct rcu_state *rsp, struct rcu_node *rnp, 
	return true;

}



/*

 * Similar to rcu_start_gp_advanced(), but also advance the calling CPU's

 * callbacks.  Note that rcu_start_gp_advanced() cannot do this because it

 * is invoked indirectly from rcu_advance_cbs(), which would result in

 * endless recursion -- or would do so if it wasn't for the self-deadlock

 * that is encountered beforehand.

 *

 * Returns true if the grace-period kthread needs to be awakened.

 */

static bool rcu_start_gp(struct rcu_state *rsp)

{

	struct rcu_data *rdp = this_cpu_ptr(rsp->rda);

	struct rcu_node *rnp = rcu_get_root(rsp);

	bool ret = false;



	/*

	 * If there is no grace period in progress right now, any

	 * callbacks we have up to this point will be satisfied by the

	 * next grace period.  Also, advancing the callbacks reduces the

	 * probability of false positives from cpu_needs_another_gp()

	 * resulting in pointless grace periods.  So, advance callbacks

	 * then start the grace period!

	 */

	ret = rcu_advance_cbs(rsp, rnp, rdp) || ret;

	ret = rcu_start_gp_advanced(rsp, rnp, rdp) || ret;

	return ret;

}



/*

 * Report a full set of quiescent states to the specified rcu_state data

 * structure.  Invoke rcu_gp_kthread_wake() to awaken the grace-period
@@ -2889,22 +2861,27 @@ __rcu_process_callbacks(struct rcu_state *rsp) 
	unsigned long flags;

	bool needwake;

	struct rcu_data *rdp = raw_cpu_ptr(rsp->rda);

	struct rcu_node *rnp;



	WARN_ON_ONCE(!rdp->beenonline);



	/* Update RCU state based on any recent quiescent states. */

	rcu_check_quiescent_state(rsp, rdp);



	/* Does this CPU require a not-yet-started grace period? */

	/* No grace period and unregistered callbacks? */

	if (!rcu_gp_in_progress(rsp) &&

	    rcu_segcblist_is_enabled(&rdp->cblist)) {

		local_irq_save(flags);

	if (cpu_needs_another_gp(rsp, rdp)) {

		raw_spin_lock_rcu_node(rcu_get_root(rsp)); /* irqs disabled. */

		needwake = rcu_start_gp(rsp);

		raw_spin_unlock_irqrestore_rcu_node(rcu_get_root(rsp), flags);

		if (rcu_segcblist_restempty(&rdp->cblist, RCU_NEXT_READY_TAIL)) {

			local_irq_restore(flags);

		} else {

			rnp = rdp->mynode;

			raw_spin_lock_rcu_node(rnp); /* irqs disabled. */

			needwake = rcu_accelerate_cbs(rsp, rnp, rdp);

			raw_spin_unlock_irqrestore_rcu_node(rnp, flags);

			if (needwake)

				rcu_gp_kthread_wake(rsp);

	} else {

		local_irq_restore(flags);

		}

	}



	/* If there are callbacks ready, invoke them. */