Commit a1e12248 authored Jan 13, 2016 by Paul E. McKenney

rcu: Make cond_resched_rcu_qs() supply RCU-sched expedited QS

Although cond_resched_rcu_qs() supplies quiescent states to all flavors
of normal RCU grace periods, it does nothing for expedited RCU-sched
grace periods. This commit therefore adds a check for a need for a
quiescent state from the current CPU by an expedited RCU-sched grace
period, and invokes rcu_sched_qs() to supply that quiescent state if so.

Note that the check is racy in that we might be migrated to some other
CPU just after checking the per-CPU variable. This is OK because the
act of migration will do a context switch, which will supply the needed
quiescent state. The only downside is that we might do an unnecessary
call to rcu_sched_qs(), but the probability is low and the overhead
is small.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

parent 251c617c

kernel/rcu/tree.c

+15 −0

Original line number	Diff line number	Diff line
		@@ -370,6 +370,21 @@ void rcu_all_qs(void)
		rcu_momentary_dyntick_idle();
		local_irq_restore(flags);
		}
		if (unlikely(raw_cpu_read(rcu_sched_data.cpu_no_qs.b.exp))) {
		/*
		* Yes, we just checked a per-CPU variable with preemption
		* enabled, so we might be migrated to some other CPU at
		* this point. That is OK because in that case, the
		* migration will supply the needed quiescent state.
		* We might end up needlessly disabling preemption and
		* invoking rcu_sched_qs() on the destination CPU, but
		* the probability and cost are both quite low, so this
		* should not be a problem in practice.
		*/
		preempt_disable();
		rcu_sched_qs();
		preempt_enable();
		}
		this_cpu_inc(rcu_qs_ctr);
		barrier(); /* Avoid RCU read-side critical sections leaking up. */
		}