Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 62da1921 authored by Paul E. McKenney's avatar Paul E. McKenney
Browse files

rcu: Accelerate callbacks for CPU initiating a grace period 



Because grace-period initialization is carried out by a separate
kthread, it might happen on a different CPU than the one that
had the callback needing a grace period -- which is where the
callback acceleration needs to happen.

Fortunately, rcu_start_gp() holds the root rcu_node structure's
->lock, which prevents a new grace period from starting.  This
allows this function to safely determine that a grace period has
not yet started, which in turn allows it to fully accelerate any
callbacks that it has pending.  This commit adds this acceleration.

Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
parent 6f0c0580

kernel/rcutree.c

+24 −2
Original line number Diff line number Diff line
@@ -1404,15 +1404,37 @@ rcu_start_gp(struct rcu_state *rsp, unsigned long flags) 
	    !cpu_needs_another_gp(rsp, rdp)) {

		/*

		 * Either we have not yet spawned the grace-period

		 * task or this CPU does not need another grace period.

		 * task, this CPU does not need another grace period,

		 * or a grace period is already in progress.

		 * Either way, don't start a new grace period.

		 */

		raw_spin_unlock_irqrestore(&rnp->lock, flags);

		return;

	}



	/*

	 * Because there is no grace period in progress right now,

	 * any callbacks we have up to this point will be satisfied

	 * by the next grace period.  So promote all callbacks to be

	 * handled after the end of the next grace period.  If the

	 * CPU is not yet aware of the end of the previous grace period,

	 * we need to allow for the callback advancement that will

	 * occur when it does become aware.  Deadlock prevents us from

	 * making it aware at this point: We cannot acquire a leaf

	 * rcu_node ->lock while holding the root rcu_node ->lock.

	 */

	rdp->nxttail[RCU_NEXT_READY_TAIL] = rdp->nxttail[RCU_NEXT_TAIL];

	if (rdp->completed == rsp->completed)

		rdp->nxttail[RCU_WAIT_TAIL] = rdp->nxttail[RCU_NEXT_TAIL];



	rsp->gp_flags = RCU_GP_FLAG_INIT;

	raw_spin_unlock_irqrestore(&rnp->lock, flags);

	raw_spin_unlock(&rnp->lock); /* Interrupts remain disabled. */



	/* Ensure that CPU is aware of completion of last grace period. */

	rcu_process_gp_end(rsp, rdp);

	local_irq_restore(flags);



	/* Wake up rcu_gp_kthread() to start the grace period. */

	wake_up(&rsp->gp_wq);

}