[PATCH] Yet another RCU documentation update

This document describes RCU-related publications, and is followed by

the corresponding bibtex entries.

the corresponding bibtex entries.  A number of the publications may

be found at http://www.rdrop.com/users/paulmck/RCU/.

The first thing resembling RCU was published in 1980, when Kung and Lehman

[Kung80] recommended use of a garbage collector to defer destruction

and a paper describing SELinux performance with RCU [JamesMorris04b].

2005 has seen further adaptation of RCU to realtime use, permitting

preemption of RCU realtime critical sections [PaulMcKenney05a,

PaulMcKenney05b].

Bibtex Entries

@article{Kung80

\url{http://www.livejournal.com/users/james_morris/2153.html}

[Viewed December 10, 2004]"

}

@unpublished{PaulMcKenney05a

,Author="Paul E. McKenney"

,Title="{[RFC]} {RCU} and {CONFIG\_PREEMPT\_RT} progress"

,month="May"

,year="2005"

,note="Available:

\url{http://lkml.org/lkml/2005/5/9/185}

[Viewed May 13, 2005]"

,annotation="

	First publication of working lock-based deferred free patches

	for the CONFIG_PREEMPT_RT environment.

"

}

@conference{PaulMcKenney05b

,Author="Paul E. McKenney and Dipankar Sarma"

,Title="Towards Hard Realtime Response from the Linux Kernel on SMP Hardware"

,Booktitle="linux.conf.au 2005"

,month="April"

,year="2005"

,address="Canberra, Australia"

,note="Available:

\url{http://www.rdrop.com/users/paulmck/RCU/realtimeRCU.2005.04.23a.pdf}

[Viewed May 13, 2005]"

,annotation="

	Realtime turns into making RCU yet more realtime friendly.

"

}

wait for anything else to get done, since there are no other CPUs for

anything else to be happening on.  Although this approach will -sort- -of-

work a surprising amount of the time, it is a very bad idea in general.

This document presents two examples that demonstrate exactly how bad an

This document presents three examples that demonstrate exactly how bad an

idea this is.

element B.  This situation can greatly decrease the life expectancy of

your kernel.

This same problem can occur if call_rcu() is invoked from a hardware

interrupt handler.

Example 2: Function-Call Fatality

underlying RCU, namely that call_rcu() defers invoking its arguments until

all RCU read-side critical sections currently executing have completed.

Quick Quiz: why is it -not- legal to invoke synchronize_rcu() in

Quick Quiz #1: why is it -not- legal to invoke synchronize_rcu() in

	this case?

Example 3: Death by Deadlock

Suppose that call_rcu() is invoked while holding a lock, and that the

callback function must acquire this same lock.  In this case, if

call_rcu() were to directly invoke the callback, the result would

be self-deadlock.

In some cases, it would possible to restructure to code so that

the call_rcu() is delayed until after the lock is released.  However,

there are cases where this can be quite ugly:

1.	If a number of items need to be passed to call_rcu() within

	the same critical section, then the code would need to create

	a list of them, then traverse the list once the lock was

	released.

2.	In some cases, the lock will be held across some kernel API,

	so that delaying the call_rcu() until the lock is released

	requires that the data item be passed up via a common API.

	It is far better to guarantee that callbacks are invoked

	with no locks held than to have to modify such APIs to allow

	arbitrary data items to be passed back up through them.

If call_rcu() directly invokes the callback, painful locking restrictions

or API changes would be required.

Quick Quiz #2: What locking restriction must RCU callbacks respect?

Summary

Permitting call_rcu() to immediately invoke its arguments or permitting

synchronize_rcu() to immediately return breaks RCU, even on a UP system.

So do not do it!  Even on a UP system, the RCU infrastructure -must-

respect grace periods.

Answer to Quick Quiz

The calling function is scanning an RCU-protected linked list, and

is therefore within an RCU read-side critical section.  Therefore,

the called function has been invoked within an RCU read-side critical

section, and is not permitted to block.

respect grace periods, and -must- invoke callbacks from a known environment

in which no locks are held.

Answer to Quick Quiz #1:

	Why is it -not- legal to invoke synchronize_rcu() in this case?

	Because the calling function is scanning an RCU-protected linked

	list, and is therefore within an RCU read-side critical section.

	Therefore, the called function has been invoked within an RCU

	read-side critical section, and is not permitted to block.

Answer to Quick Quiz #2:

	What locking restriction must RCU callbacks respect?

	Any lock that is acquired within an RCU callback must be

	acquired elsewhere using an _irq variant of the spinlock

	primitive.  For example, if "mylock" is acquired by an

	RCU callback, then a process-context acquisition of this

	lock must use something like spin_lock_irqsave() to

	acquire the lock.

	If the process-context code were to simply use spin_lock(),

	then, since RCU callbacks can be invoked from softirq context,

	the callback might be called from a softirq that interrupted

	the process-context critical section.  This would result in

	self-deadlock.

	This restriction might seem gratuitous, since very few RCU

	callbacks acquire locks directly.  However, a great many RCU

	callbacks do acquire locks -indirectly-, for example, via

	the kfree() primitive.

	rcu_read_lock_bh()) in the read-side critical sections,

	and are also an excellent aid to readability.

	As a rough rule of thumb, any dereference of an RCU-protected

	pointer must be covered by rcu_read_lock() or rcu_read_lock_bh()

	or by the appropriate update-side lock.

3.	Does the update code tolerate concurrent accesses?

	The whole point of RCU is to permit readers to run without

		The rcu_dereference() primitive is used by the various

		"_rcu()" list-traversal primitives, such as the

		list_for_each_entry_rcu().

		list_for_each_entry_rcu().  Note that it is perfectly

		legal (if redundant) for update-side code to use

		rcu_dereference() and the "_rcu()" list-traversal

		primitives.  This is particularly useful in code

		that is common to readers and updaters.

	b.	If the list macros are being used, the list_add_tail_rcu()

		and list_add_rcu() primitives must be used in order

	Use of the _rcu() list-traversal primitives outside of an

	RCU read-side critical section causes no harm other than

	a slight performance degradation on Alpha CPUs and some

	confusion on the part of people trying to read the code.

	Another way of thinking of this is "If you are holding the

	lock that prevents the data structure from changing, why do

	you also need RCU-based protection?"  That said, there may

	well be situations where use of the _rcu() list-traversal

	primitives while the update-side lock is held results in

	simpler and more maintainable code.  The jury is still out

	on this question.

	a slight performance degradation on Alpha CPUs.  It can

	also be quite helpful in reducing code bloat when common

	code is shared between readers and updaters.

10.	Conversely, if you are in an RCU read-side critical section,

	you -must- use the "_rcu()" variants of the list macros.

	Of these, one was allowed to lapse by the assignee, and the

	others have been contributed to the Linux kernel under GPL.

o	I hear that RCU needs work in order to support realtime kernels?

	Yes, work in progress.

o	Where can I find more information on RCU?

	See the RTFP.txt file in this directory.

	Or point your browser at http://www.rdrop.com/users/paulmck/RCU/.

o	What are all these files in this directory?

	NMI-RCU.txt

		Describes how to use RCU to implement dynamic

		NMI handlers, which can be revectored on the fly,

		without rebooting.

	RTFP.txt

		List of RCU-related publications and web sites.

	UP.txt

		Discussion of RCU usage in UP kernels.

	arrayRCU.txt

		Describes how to use RCU to protect arrays, with

		resizeable arrays whose elements reference other

		data structures being of the most interest.

	checklist.txt

		Lists things to check for when inspecting code that

		uses RCU.

	listRCU.txt

		Describes how to use RCU to protect linked lists.

		This is the simplest and most common use of RCU

		in the Linux kernel.

	rcu.txt

		You are reading it!

	whatisRCU.txt

		Overview of how the RCU implementation works.  Along

		the way, presents a conceptual view of RCU.