Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit f69b17d7 authored by Lai Jiangshan's avatar Lai Jiangshan Committed by Ingo Molnar
Browse files

rcu: rcu_barrier VS cpu_hotplug: Ensure callbacks in dead cpu are migrated to online cpu 



cpu hotplug may happen asynchronously, some rcu callbacks are maybe
still on dead cpu, rcu_barrier() also needs to wait for these rcu
callbacks to complete, so we must ensure callbacks in dead cpu are
migrated to online cpu.

Paul E. McKenney's review:

  Good stuff, Lai!!!  Simpler than any of the approaches that I was
  considering, and, better yet, independent of the underlying RCU
  implementation!!!

  I was initially worried that wake_up() might wake only one of two
  possible wait_event()s, namely rcu_barrier() and the CPU_POST_DEAD code,
  but the fact that wait_event() clears WQ_FLAG_EXCLUSIVE avoids that issue.
  I was also worried about the fact that different RCU implementations have
  different mappings of call_rcu(), call_rcu_bh(), and call_rcu_sched(), but
  this is OK as well because we just get an extra (harmless) callback in the
  case that they map together (for example, Classic RCU has call_rcu_sched()
  mapping to call_rcu()).

  Overlap of CPU-hotplug operations is prevented by cpu_add_remove_lock,
  and any stray callbacks that arrive (for example, from irq handlers
  running on the dying CPU) either are ahead of the CPU_DYING callbacks on
  the one hand (and thus accounted for), or happened after the rcu_barrier()
  started on the other (and thus don't need to be accounted for).

Signed-off-by: default avatarLai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
LKML-Reference: <49C36476.1010400@cn.fujitsu.com>
Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
parent dfbbe89e

kernel/rcupdate.c

+44 −0
Original line number Diff line number Diff line
@@ -122,6 +122,8 @@ static void rcu_barrier_func(void *type) 
	}

}



static inline void wait_migrated_callbacks(void);



/*

 * Orchestrate the specified type of RCU barrier, waiting for all

 * RCU callbacks of the specified type to complete.
@@ -147,6 +149,7 @@ static void _rcu_barrier(enum rcu_barrier type) 
		complete(&rcu_barrier_completion);

	wait_for_completion(&rcu_barrier_completion);

	mutex_unlock(&rcu_barrier_mutex);

	wait_migrated_callbacks();

}



/**
@@ -176,9 +179,50 @@ void rcu_barrier_sched(void) 
}

EXPORT_SYMBOL_GPL(rcu_barrier_sched);



static atomic_t rcu_migrate_type_count = ATOMIC_INIT(0);

static struct rcu_head rcu_migrate_head[3];

static DECLARE_WAIT_QUEUE_HEAD(rcu_migrate_wq);



static void rcu_migrate_callback(struct rcu_head *notused)

{

	if (atomic_dec_and_test(&rcu_migrate_type_count))

		wake_up(&rcu_migrate_wq);

}



static inline void wait_migrated_callbacks(void)

{

	wait_event(rcu_migrate_wq, !atomic_read(&rcu_migrate_type_count));

}



static int __cpuinit rcu_barrier_cpu_hotplug(struct notifier_block *self,

		unsigned long action, void *hcpu)

{

	if (action == CPU_DYING) {

		/*

		 * preempt_disable() in on_each_cpu() prevents stop_machine(),

		 * so when "on_each_cpu(rcu_barrier_func, (void *)type, 1);"

		 * returns, all online cpus have queued rcu_barrier_func(),

		 * and the dead cpu(if it exist) queues rcu_migrate_callback()s.

		 *

		 * These callbacks ensure _rcu_barrier() waits for all

		 * RCU callbacks of the specified type to complete.

		 */

		atomic_set(&rcu_migrate_type_count, 3);

		call_rcu_bh(rcu_migrate_head, rcu_migrate_callback);

		call_rcu_sched(rcu_migrate_head + 1, rcu_migrate_callback);

		call_rcu(rcu_migrate_head + 2, rcu_migrate_callback);

	} else if (action == CPU_POST_DEAD) {

		/* rcu_migrate_head is protected by cpu_add_remove_lock */

		wait_migrated_callbacks();

	}



	return NOTIFY_OK;

}



void __init rcu_init(void)

{

	__rcu_init();

	hotcpu_notifier(rcu_barrier_cpu_hotplug, 0);

}



void rcu_scheduler_starting(void)