Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 5257514d authored by Paul E. McKenney's avatar Paul E. McKenney
Browse files

rcu: Make expedited grace period use direct call on last leaf 



During expedited grace-period initialization, a work item is scheduled
for each leaf rcu_node structure.  However, that initialization code
is itself (normally) executing from a workqueue, so one of the leaf
rcu_node structures could just as well be handled by that pre-existing
workqueue, and with less overhead.  This commit therefore uses a
shiny new rcu_is_leaf_node() macro to execute the last leaf rcu_node
structure's initialization directly from the pre-existing workqueue.

Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
parent ce397d21

kernel/rcu/rcu.h

+3 −0
Original line number Diff line number Diff line
@@ -276,6 +276,9 @@ static inline void rcu_init_levelspread(int *levelspread, const int *levelcnt) 
/* Is this rcu_node a leaf? */

#define rcu_is_leaf_node(rnp) ((rnp)->level == rcu_num_lvls - 1)



/* Is this rcu_node the last leaf? */

#define rcu_is_last_leaf_node(rsp, rnp) ((rnp) == &(rsp)->node[rcu_num_nodes - 1])



/*

 * Do a full breadth-first scan of the rcu_node structures for the

 * specified rcu_state structure.

kernel/rcu/tree_exp.h

+3 −2
Original line number Diff line number Diff line
@@ -486,8 +486,9 @@ static void sync_rcu_exp_select_cpus(struct rcu_state *rsp, 
		rnp->rew.rew_func = func;

		rnp->rew.rew_rsp = rsp;

		if (!READ_ONCE(rcu_par_gp_wq) ||

		    rcu_scheduler_active != RCU_SCHEDULER_RUNNING) {

			/* No workqueues yet. */

		    rcu_scheduler_active != RCU_SCHEDULER_RUNNING ||

		    rcu_is_last_leaf_node(rsp, rnp)) {

			/* No workqueues yet or last leaf, do direct call. */

			sync_rcu_exp_select_node_cpus(&rnp->rew.rew_work);

			continue;

		}