docs: sphinxify kmemcheck.txt and move to dev-tools

GETTING STARTED WITH KMEMCHECK

Getting started with kmemcheck

==============================

Vegard Nossum <vegardno@ifi.uio.no>

Contents

========

0. Introduction

1. Downloading

2. Configuring and compiling

3. How to use

3.1. Booting

3.2. Run-time enable/disable

3.3. Debugging

3.4. Annotating false positives

4. Reporting errors

5. Technical description

0. Introduction

===============

Introduction

------------

kmemcheck is a debugging feature for the Linux Kernel. More specifically, it

is a dynamic checker that detects and warns about some uses of uninitialized

feature.

1. Downloading

==============

Downloading

-----------

As of version 2.6.31-rc1, kmemcheck is included in the mainline kernel.

2. Configuring and compiling

============================

Configuring and compiling

-------------------------

kmemcheck only works for the x86 (both 32- and 64-bit) platform. A number of

configuration variables must have specific settings in order for the kmemcheck

menu to even appear in "menuconfig". These are:

  o CONFIG_CC_OPTIMIZE_FOR_SIZE=n

- ``CONFIG_CC_OPTIMIZE_FOR_SIZE=n``

	This option is located under "General setup" / "Optimize for size".

	Without this, gcc will use certain optimizations that usually lead to

	16 bits. kmemcheck sees only the 32-bit load, and may trigger a

	warning for the upper 16 bits (if they're uninitialized).

  o CONFIG_SLAB=y or CONFIG_SLUB=y

- ``CONFIG_SLAB=y`` or ``CONFIG_SLUB=y``

	This option is located under "General setup" / "Choose SLAB

	allocator".

  o CONFIG_FUNCTION_TRACER=n

- ``CONFIG_FUNCTION_TRACER=n``

	This option is located under "Kernel hacking" / "Tracers" / "Kernel

	Function Tracer"

	modifies memory that was tracked by kmemcheck, the result is an

	endless recursive page fault.

  o CONFIG_DEBUG_PAGEALLOC=n

- ``CONFIG_DEBUG_PAGEALLOC=n``

	This option is located under "Kernel hacking" / "Memory Debugging"

	/ "Debug page memory allocations".

In addition, I highly recommend turning on CONFIG_DEBUG_INFO=y. This is also

In addition, I highly recommend turning on ``CONFIG_DEBUG_INFO=y``. This is also

located under "Kernel hacking". With this, you will be able to get line number

information from the kmemcheck warnings, which is extremely valuable in

debugging a problem. This option is not mandatory, however, because it slows

Debugging" / "kmemcheck: trap use of uninitialized memory"). Here follows

a description of the kmemcheck configuration variables:

  o CONFIG_KMEMCHECK

- ``CONFIG_KMEMCHECK``

	This must be enabled in order to use kmemcheck at all...

  o CONFIG_KMEMCHECK_[DISABLED | ENABLED | ONESHOT]_BY_DEFAULT

- ``CONFIG_KMEMCHECK_``[``DISABLED`` | ``ENABLED`` | ``ONESHOT``]``_BY_DEFAULT``

	This option controls the status of kmemcheck at boot-time. "Enabled"

	will enable kmemcheck right from the start, "disabled" will boot the

	kernel as normal (but with the kmemcheck code compiled in, so it can

	time overhead is not incurred, and the kernel will be almost as fast

	as normal.

  o CONFIG_KMEMCHECK_QUEUE_SIZE

- ``CONFIG_KMEMCHECK_QUEUE_SIZE``

	Select the maximum number of error reports to store in an internal

	(fixed-size) buffer. Since errors can occur virtually anywhere and in

	any context, we need a temporary storage area which is guaranteed not

	will get lost in that way instead. Try setting this to 10 or so on

	such a setup.

  o CONFIG_KMEMCHECK_SHADOW_COPY_SHIFT

- ``CONFIG_KMEMCHECK_SHADOW_COPY_SHIFT``

	Select the number of shadow bytes to save along with each entry of the

	error-report queue. These bytes indicate what parts of an allocation

	are initialized, uninitialized, etc. and will be displayed when an

	The default value should be fine for debugging most problems. It also

	fits nicely within 80 columns.

  o CONFIG_KMEMCHECK_PARTIAL_OK

- ``CONFIG_KMEMCHECK_PARTIAL_OK``

	This option (when enabled) works around certain GCC optimizations that

	produce 32-bit reads from 16-bit variables where the upper 16 bits are

	thrown away afterwards.

	some real errors, but disabling it would probably produce a lot of

	false positives.

  o CONFIG_KMEMCHECK_BITOPS_OK

- ``CONFIG_KMEMCHECK_BITOPS_OK``

	This option silences warnings that would be generated for bit-field

	accesses where not all the bits are initialized at the same time. This

	may also hide some real bugs.

Now compile the kernel as usual.

3. How to use

=============

How to use

----------

3.1. Booting

============

Booting

~~~~~~~

First some information about the command-line options. There is only one

option specific to kmemcheck, and this is called "kmemcheck". It can be used

to override the default mode as chosen by the CONFIG_KMEMCHECK_*_BY_DEFAULT

to override the default mode as chosen by the ``CONFIG_KMEMCHECK_*_BY_DEFAULT``

option. Its possible settings are:

  o kmemcheck=0 (disabled)

  o kmemcheck=1 (enabled)

  o kmemcheck=2 (one-shot mode)

- ``kmemcheck=0`` (disabled)

- ``kmemcheck=1`` (enabled)

- ``kmemcheck=2`` (one-shot mode)

If SLUB debugging has been enabled in the kernel, it may take precedence over

kmemcheck in such a way that the slab caches which are under SLUB debugging

will not be tracked by kmemcheck. In order to ensure that this doesn't happen

(even though it shouldn't by default), use SLUB's boot option "slub_debug",

like this: slub_debug=-

(even though it shouldn't by default), use SLUB's boot option ``slub_debug``,

like this: ``slub_debug=-``

In fact, this option may also be used for fine-grained control over SLUB vs.

kmemcheck. For example, if the command line includes "kmemcheck=1

slub_debug=,dentry", then SLUB debugging will be used only for the "dentry"

slab cache, and with kmemcheck tracking all the other caches. This is advanced

usage, however, and is not generally recommended.

kmemcheck. For example, if the command line includes

``kmemcheck=1 slub_debug=,dentry``, then SLUB debugging will be used only

for the "dentry" slab cache, and with kmemcheck tracking all the other

caches. This is advanced usage, however, and is not generally recommended.

3.2. Run-time enable/disable

============================

Run-time enable/disable

~~~~~~~~~~~~~~~~~~~~~~~

When the kernel has booted, it is possible to enable or disable kmemcheck at

run-time. WARNING: This feature is still experimental and may cause false

it doesn't work properly (e.g. you see an unreasonable amount of warnings), I

will be happy to take bug reports.

Use the file /proc/sys/kernel/kmemcheck for this purpose, e.g.:

Use the file ``/proc/sys/kernel/kmemcheck`` for this purpose, e.g.::

	$ echo 0 > /proc/sys/kernel/kmemcheck # disables kmemcheck

The numbers are the same as for the kmemcheck= command-line option.

The numbers are the same as for the ``kmemcheck=`` command-line option.

3.3. Debugging

==============

Debugging

~~~~~~~~~

A typical report will look something like this:

A typical report will look something like this::

    WARNING: kmemcheck: Caught 32-bit read from uninitialized memory (ffff88003e4a2024)

    80000000000000000000000000000000000000000088ffff0000000000000000

bit) value. This will help us pinpoint exactly which instruction that caused

the warning.

If your kernel was compiled with CONFIG_DEBUG_INFO=y, then all we have to do

is give this address to the addr2line program, like this:

If your kernel was compiled with ``CONFIG_DEBUG_INFO=y``, then all we have to do

is give this address to the addr2line program, like this::

	$ addr2line -e vmlinux -i ffffffff8104ede8

	arch/x86/include/asm/string_64.h:12

	kernel/signal.c:380

	kernel/signal.c:410

The "-e vmlinux" tells addr2line which file to look in. IMPORTANT: This must

be the vmlinux of the kernel that produced the warning in the first place! If

not, the line number information will almost certainly be wrong.

The "``-e vmlinux``" tells addr2line which file to look in. **IMPORTANT:**

This must be the vmlinux of the kernel that produced the warning in the

first place! If not, the line number information will almost certainly be

wrong.

The "-i" tells addr2line to also print the line numbers of inlined functions.

In this case, the flag was very important, because otherwise, it would only

have printed the first line, which is just a call to memcpy(), which could be

called from a thousand places in the kernel, and is therefore not very useful.

These inlined functions would not show up in the stack trace above, simply

because the kernel doesn't load the extra debugging information. This

technique can of course be used with ordinary kernel oopses as well.

The "``-i``" tells addr2line to also print the line numbers of inlined

functions.  In this case, the flag was very important, because otherwise,

it would only have printed the first line, which is just a call to

``memcpy()``, which could be called from a thousand places in the kernel, and

is therefore not very useful.  These inlined functions would not show up in

the stack trace above, simply because the kernel doesn't load the extra

debugging information. This technique can of course be used with ordinary

kernel oopses as well.

In this case, it's the caller of memcpy() that is interesting, and it can be

found in include/asm-generic/siginfo.h, line 287:

In this case, it's the caller of ``memcpy()`` that is interesting, and it can be

found in ``include/asm-generic/siginfo.h``, line 287::

    281 static inline void copy_siginfo(struct siginfo *to, struct siginfo *from)

    282 {

warn about writes to unallocated or freed memory as well), it was probably the

"from" argument which contained some uninitialized bytes. Following the chain

of calls, we move upwards to see where "from" was allocated or initialized,

kernel/signal.c, line 380:

``kernel/signal.c``, line 380::

    359 static void collect_signal(int sig, struct sigpending *list, siginfo_t *info)

    360 {

    392         }

    393 }

Here, it is &first->info that is being passed on to copy_siginfo(). The

variable "first" was found on a list -- passed in as the second argument to

collect_signal(). We  continue our journey through the stack, to figure out

where the item on "list" was allocated or initialized. We move to line 410:

Here, it is ``&first->info`` that is being passed on to ``copy_siginfo()``. The

variable ``first`` was found on a list -- passed in as the second argument to

``collect_signal()``. We  continue our journey through the stack, to figure out

where the item on "list" was allocated or initialized. We move to line 410::

    395 static int __dequeue_signal(struct sigpending *pending, sigset_t *mask,

    396                         siginfo_t *info)

    ...

    414 }

Now we need to follow the "pending" pointer, since that is being passed on to

collect_signal() as "list". At this point, we've run out of lines from the

Now we need to follow the ``pending`` pointer, since that is being passed on to

``collect_signal()`` as ``list``. At this point, we've run out of lines from the

"addr2line" output. Not to worry, we just paste the next addresses from the

kmemcheck stack dump, i.e.:

kmemcheck stack dump, i.e.::

     [<ffffffff8104f04e>] dequeue_signal+0x8e/0x170

     [<ffffffff81050bd8>] get_signal_to_deliver+0x98/0x390

Remember that since these addresses were found on the stack and not as the

RIP value, they actually point to the _next_ instruction (they are return

addresses). This becomes obvious when we look at the code for line 446:

addresses). This becomes obvious when we look at the code for line 446::

    422 int dequeue_signal(struct task_struct *tsk, sigset_t *mask, siginfo_t *info)

    423 {

So instead of looking at 446, we should be looking at 431, which is the line

that executes just before 446. Here we see that what we are looking for is

&tsk->signal->shared_pending.

``&tsk->signal->shared_pending``.

Our next task is now to figure out which function that puts items on this

"shared_pending" list. A crude, but efficient tool, is git grep:

``shared_pending`` list. A crude, but efficient tool, is ``git grep``::

	$ git grep -n 'shared_pending' kernel/

	...

There were more results, but none of them were related to list operations,

and these were the only assignments. We inspect the line numbers more closely

and find that this is indeed where items are being added to the list:

and find that this is indeed where items are being added to the list::

    816 static int send_signal(int sig, struct siginfo *info, struct task_struct *t,

    817				int group)

    ...

    890 }

and:

and::

    1309 int send_sigqueue(struct sigqueue *q, struct task_struct *t, int group)

    1310 {

    ....

    1347 }

In the first case, the list element we are looking for, "q", is being returned

from the function __sigqueue_alloc(), which looks like an allocation function.

Let's take a look at it:

In the first case, the list element we are looking for, ``q``, is being

returned from the function ``__sigqueue_alloc()``, which looks like an

allocation function.  Let's take a look at it::

    187 static struct sigqueue *__sigqueue_alloc(struct task_struct *t, gfp_t flags,

    188						 int override_rlimit)

    214		return q;

    215 }

We see that this function initializes q->list, q->flags, and q->user. It seems

that now is the time to look at the definition of "struct sigqueue", e.g.:

We see that this function initializes ``q->list``, ``q->flags``, and

``q->user``. It seems that now is the time to look at the definition of

``struct sigqueue``, e.g.::

    14 struct sigqueue {

    15	       struct list_head list;

    18	       struct user_struct *user;

    19 };

And, you might remember, it was a memcpy() on &first->info that caused the

warning, so this makes perfect sense. It also seems reasonable to assume that

it is the caller of __sigqueue_alloc() that has the responsibility of filling

out (initializing) this member.

And, you might remember, it was a ``memcpy()`` on ``&first->info`` that

caused the warning, so this makes perfect sense. It also seems reasonable

to assume that it is the caller of ``__sigqueue_alloc()`` that has the

responsibility of filling out (initializing) this member.

But just which fields of the struct were uninitialized? Let's look at

kmemcheck's report again:

kmemcheck's report again::

    WARNING: kmemcheck: Caught 32-bit read from uninitialized memory (ffff88003e4a2024)

    80000000000000000000000000000000000000000088ffff0000000000000000

     i i i i u u u u i i i i i i i i u u u u u u u u u u u u u u u u

	     ^

These first two lines are the memory dump of the memory object itself, and the

shadow bytemap, respectively. The memory object itself is in this case

&first->info. Just beware that the start of this dump is NOT the start of the

object itself! The position of the caret (^) corresponds with the address of

the read (ffff88003e4a2024).

These first two lines are the memory dump of the memory object itself, and

the shadow bytemap, respectively. The memory object itself is in this case

``&first->info``. Just beware that the start of this dump is NOT the start

of the object itself! The position of the caret (^) corresponds with the

address of the read (ffff88003e4a2024).

The shadow bytemap dump legend is as follows:

  i - initialized

  u - uninitialized

  a - unallocated (memory has been allocated by the slab layer, but has not

- i: initialized

- u: uninitialized

- a: unallocated (memory has been allocated by the slab layer, but has not

  yet been handed off to anybody)

  f - freed (memory has been allocated by the slab layer, but has been freed

- f: freed (memory has been allocated by the slab layer, but has been freed

  by the previous owner)

In order to figure out where (relative to the start of the object) the

uninitialized memory was located, we have to look at the disassembly. For

that, we'll need the RIP address again:

that, we'll need the RIP address again::

    RIP: 0010:[<ffffffff8104ede8>]  [<ffffffff8104ede8>] __dequeue_signal+0xc8/0x190

	ffffffff8104edf5:	mov    %r8,%rdi

	ffffffff8104edf8:	callq  ffffffff8104de60 <__sigqueue_free>

As expected, it's the "rep movsl" instruction from the memcpy() that causes

the warning. We know about REP MOVSL that it uses the register RCX to count

the number of remaining iterations. By taking a look at the register dump

again (from the kmemcheck report), we can figure out how many bytes were left

to copy:

As expected, it's the "``rep movsl``" instruction from the ``memcpy()``

that causes the warning. We know about ``REP MOVSL`` that it uses the register

``RCX`` to count the number of remaining iterations. By taking a look at the

register dump again (from the kmemcheck report), we can figure out how many

bytes were left to copy::

    RAX: 0000000000000030 RBX: ffff88003d4ea968 RCX: 0000000000000009

By looking at the disassembly, we also see that %ecx is being loaded with the

value $0xc just before (ffffffff8104edd8), so we are very lucky. Keep in mind

that this is the number of iterations, not bytes. And since this is a "long"

operation, we need to multiply by 4 to get the number of bytes. So this means

that the uninitialized value was encountered at 4 * (0xc - 0x9) = 12 bytes

from the start of the object.

By looking at the disassembly, we also see that ``%ecx`` is being loaded

with the value ``$0xc`` just before (ffffffff8104edd8), so we are very

lucky. Keep in mind that this is the number of iterations, not bytes. And

since this is a "long" operation, we need to multiply by 4 to get the

number of bytes. So this means that the uninitialized value was encountered

at 4 * (0xc - 0x9) = 12 bytes from the start of the object.

We can now try to figure out which field of the "struct siginfo" that was not

initialized. This is the beginning of the struct:

We can now try to figure out which field of the "``struct siginfo``" that

was not initialized. This is the beginning of the struct::

    40 typedef struct siginfo {

    41	       int si_signo;

    93 } siginfo_t;

On 64-bit, the int is 4 bytes long, so it must the union member that has

not been initialized. We can verify this using gdb:

not been initialized. We can verify this using gdb::

	$ gdb vmlinux

	...

	$1 = (union {...} *) 0x10

Actually, it seems that the union member is located at offset 0x10 -- which

means that gcc has inserted 4 bytes of padding between the members si_code

and _sifields. We can now get a fuller picture of the memory dump:

means that gcc has inserted 4 bytes of padding between the members ``si_code``

and ``_sifields``. We can now get a fuller picture of the memory dump::

		 _----------------------------=> si_code

		/	 _--------------------=> (padding)

	80000000000000000000000000000000000000000088ffff0000000000000000

	 i i i i u u u u i i i i i i i i u u u u u u u u u u u u u u u u

This allows us to realize another important fact: si_code contains the value

0x80. Remember that x86 is little endian, so the first 4 bytes "80000000" are

really the number 0x00000080. With a bit of research, we find that this is

actually the constant SI_KERNEL defined in include/asm-generic/siginfo.h:

This allows us to realize another important fact: ``si_code`` contains the

value 0x80. Remember that x86 is little endian, so the first 4 bytes

"80000000" are really the number 0x00000080. With a bit of research, we

find that this is actually the constant ``SI_KERNEL`` defined in

``include/asm-generic/siginfo.h``::

    144 #define SI_KERNEL	0x80		/* sent by the kernel from somewhere	 */

This macro is used in exactly one place in the x86 kernel: In send_signal()

in kernel/signal.c:

This macro is used in exactly one place in the x86 kernel: In ``send_signal()``

in ``kernel/signal.c``::

    816 static int send_signal(int sig, struct siginfo *info, struct task_struct *t,

    817				int group)

    ...

    890 }

Not only does this match with the .si_code member, it also matches the place

Not only does this match with the ``.si_code`` member, it also matches the place

we found earlier when looking for where siginfo_t objects are enqueued on the

"shared_pending" list.

``shared_pending`` list.

So to sum up: It seems that it is the padding introduced by the compiler

between two struct fields that is uninitialized, and this gets reported when

we do a memcpy() on the struct. This means that we have identified a false

we do a ``memcpy()`` on the struct. This means that we have identified a false

positive warning.

Normally, kmemcheck will not report uninitialized accesses in memcpy() calls

Normally, kmemcheck will not report uninitialized accesses in ``memcpy()`` calls

when both the source and destination addresses are tracked. (Instead, we copy

the shadow bytemap as well). In this case, the destination address clearly

was not tracked. We can dig a little deeper into the stack trace from above:

was not tracked. We can dig a little deeper into the stack trace from above::

	arch/x86/kernel/signal.c:805

	arch/x86/kernel/signal.c:871

	arch/x86/kernel/entry_64.S:694

And we clearly see that the destination siginfo object is located on the

stack:

stack::

    782 static void do_signal(struct pt_regs *regs)

    783 {

    ...

    854 }

And this &info is what eventually gets passed to copy_siginfo() as the

And this ``&info`` is what eventually gets passed to ``copy_siginfo()`` as the

destination argument.

Now, even though we didn't find an actual error here, the example is still a

was all about.

3.4. Annotating false positives

===============================

Annotating false positives

~~~~~~~~~~~~~~~~~~~~~~~~~~

There are a few different ways to make annotations in the source code that

will keep kmemcheck from checking and reporting certain allocations. Here

they are:

  o __GFP_NOTRACK_FALSE_POSITIVE

	This flag can be passed to kmalloc() or kmem_cache_alloc() (therefore

	also to other functions that end up calling one of these) to indicate

	that the allocation should not be tracked because it would lead to

	a false positive report. This is a "big hammer" way of silencing

	kmemcheck; after all, even if the false positive pertains to 

	particular field in a struct, for example, we will now lose the

	ability to find (real) errors in other parts of the same struct.

- ``__GFP_NOTRACK_FALSE_POSITIVE``

	This flag can be passed to ``kmalloc()`` or ``kmem_cache_alloc()``

	(therefore also to other functions that end up calling one of

	these) to indicate that the allocation should not be tracked

	because it would lead to a false positive report. This is a "big

	hammer" way of silencing kmemcheck; after all, even if the false

	positive pertains to particular field in a struct, for example, we

	will now lose the ability to find (real) errors in other parts of

	the same struct.

	Example:

	Example::

	    /* No warnings will ever trigger on accessing any part of x */

	    x = kmalloc(sizeof *x, GFP_KERNEL | __GFP_NOTRACK_FALSE_POSITIVE);

  o kmemcheck_bitfield_begin(name)/kmemcheck_bitfield_end(name) and

	kmemcheck_annotate_bitfield(ptr, name)

- ``kmemcheck_bitfield_begin(name)``/``kmemcheck_bitfield_end(name)`` and

	``kmemcheck_annotate_bitfield(ptr, name)``

	The first two of these three macros can be used inside struct

	definitions to signal, respectively, the beginning and end of a

	bitfield. Additionally, this will assign the bitfield a name, which

	kmemcheck_annotate_bitfield() at the point of allocation, to indicate

	which parts of the allocation is part of a bitfield.

	Example:

	Example::

	    struct foo {

		int x;

	    /* No warnings will trigger on accessing the bitfield of x */

	    kmemcheck_annotate_bitfield(x, flags);

	Note that kmemcheck_annotate_bitfield() can be used even before the

	return value of kmalloc() is checked -- in other words, passing NULL

	Note that ``kmemcheck_annotate_bitfield()`` can be used even before the

	return value of ``kmalloc()`` is checked -- in other words, passing NULL

	as the first argument is legal (and will do nothing).

4. Reporting errors

===================

Reporting errors

----------------

As we have seen, kmemcheck will produce false positive reports. Therefore, it

is not very wise to blindly post kmemcheck warnings to mailing lists and

Happy hacking!

5. Technical description

========================

Technical description

---------------------

kmemcheck works by marking memory pages non-present. This means that whenever

somebody attempts to access the page, a page fault is generated. The page

   kasan

   ubsan

   kmemleak

   kmemcheck

M:	Vegard Nossum <vegardno@ifi.uio.no>

M:	Pekka Enberg <penberg@kernel.org>

S:	Maintained

F:	Documentation/kmemcheck.txt

F:	Documentation/dev-tools/kmemcheck.rst

F:	arch/x86/include/asm/kmemcheck.h

F:	arch/x86/mm/kmemcheck/

F:	include/linux/kmemcheck.h