Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit be0e1e21 authored by Paul E. McKenney's avatar Paul E. McKenney
Browse files

rcu: Streamline code produced by __rcu_read_unlock() 



Given some common flag combinations, particularly -Os, gcc will inline
rcu_read_unlock_special() despite its being in an unlikely() clause.
Use noinline to prohibit this misoptimization.

In addition, move the second barrier() in __rcu_read_unlock() so that
it is not on the common-case code path.  This will allow the compiler to
generate better code for the common-case path through __rcu_read_unlock().

Suggested-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: default avatarMathieu Desnoyers <mathieu.desnoyers@efficios.com>
parent 7765be2f

kernel/rcutree_plugin.h

+6 −6
Original line number Diff line number Diff line
@@ -284,7 +284,7 @@ static struct list_head *rcu_next_node_entry(struct task_struct *t, 
 * notify RCU core processing or task having blocked during the RCU

 * read-side critical section.

 */

static void rcu_read_unlock_special(struct task_struct *t)

static noinline void rcu_read_unlock_special(struct task_struct *t)

{

	int empty;

	int empty_exp;
@@ -391,11 +391,11 @@ void __rcu_read_unlock(void) 
	struct task_struct *t = current;



	barrier();  /* needed if we ever invoke rcu_read_unlock in rcutree.c */

	--t->rcu_read_lock_nesting;

	barrier();  /* decrement before load of ->rcu_read_unlock_special */

	if (t->rcu_read_lock_nesting == 0 &&

	    unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))

	if (--t->rcu_read_lock_nesting == 0) {

		barrier();  /* decr before ->rcu_read_unlock_special load */

		if (unlikely(ACCESS_ONCE(t->rcu_read_unlock_special)))

			rcu_read_unlock_special(t);

	}

#ifdef CONFIG_PROVE_LOCKING

	WARN_ON_ONCE(ACCESS_ONCE(t->rcu_read_lock_nesting) < 0);

#endif /* #ifdef CONFIG_PROVE_LOCKING */