Merge branch 'topic/kmemtrace' of...

What:		/sys/kernel/debug/kmemtrace/

Date:		July 2008

Contact:	Eduard - Gabriel Munteanu <eduard.munteanu@linux360.ro>

Description:

In kmemtrace-enabled kernels, the following files are created:

/sys/kernel/debug/kmemtrace/

	cpu<n>		(0400)	Per-CPU tracing data, see below. (binary)

	total_overruns	(0400)	Total number of bytes which were dropped from

				cpu<n> files because of full buffer condition,

				non-binary. (text)

	abi_version	(0400)	Kernel's kmemtrace ABI version. (text)

Each per-CPU file should be read according to the relay interface. That is,

the reader should set affinity to that specific CPU and, as currently done by

the userspace application (though there are other methods), use poll() with

an infinite timeout before every read(). Otherwise, erroneous data may be

read. The binary data has the following _core_ format:

	Event ID	(1 byte)	Unsigned integer, one of:

		0 - represents an allocation (KMEMTRACE_EVENT_ALLOC)

		1 - represents a freeing of previously allocated memory

		    (KMEMTRACE_EVENT_FREE)

	Type ID		(1 byte)	Unsigned integer, one of:

		0 - this is a kmalloc() / kfree()

		1 - this is a kmem_cache_alloc() / kmem_cache_free()

		2 - this is a __get_free_pages() et al.

	Event size	(2 bytes)	Unsigned integer representing the

					size of this event. Used to extend

					kmemtrace. Discard the bytes you

					don't know about.

	Sequence number	(4 bytes)	Signed integer used to reorder data

					logged on SMP machines. Wraparound

					must be taken into account, although

					it is unlikely.

	Caller address	(8 bytes)	Return address to the caller.

	Pointer to mem	(8 bytes)	Pointer to target memory area. Can be

					NULL, but not all such calls might be

					recorded.

In case of KMEMTRACE_EVENT_ALLOC events, the next fields follow:

	Requested bytes	(8 bytes)	Total number of requested bytes,

					unsigned, must not be zero.

	Allocated bytes (8 bytes)	Total number of actually allocated

					bytes, unsigned, must not be lower

					than requested bytes.

	Requested flags	(4 bytes)	GFP flags supplied by the caller.

	Target CPU	(4 bytes)	Signed integer, valid for event id 1.

					If equal to -1, target CPU is the same

					as origin CPU, but the reverse might

					not be true.

The data is made available in the same endianness the machine has.

Other event ids and type ids may be defined and added. Other fields may be

added by increasing event size, but see below for details.

Every modification to the ABI, including new id definitions, are followed

by bumping the ABI version by one.

Adding new data to the packet (features) is done at the end of the mandatory

data:

	Feature size	(2 byte)

	Feature ID	(1 byte)

	Feature data	(Feature size - 3 bytes)

Users:

	kmemtrace-user - git://repo.or.cz/kmemtrace-user.git

	ISAPNP	ISA PnP code is enabled.

	ISAPNP	ISA PnP code is enabled.

	ISDN	Appropriate ISDN support is enabled.

	ISDN	Appropriate ISDN support is enabled.

	JOY	Appropriate joystick support is enabled.

	JOY	Appropriate joystick support is enabled.

	KMEMTRACE kmemtrace is enabled.

	LIBATA  Libata driver is enabled

	LIBATA  Libata driver is enabled

	LP	Printer support is enabled.

	LP	Printer support is enabled.

	LOOP	Loopback device support is enabled.

	LOOP	Loopback device support is enabled.

			use the HighMem zone if it exists, and the Normal

			use the HighMem zone if it exists, and the Normal

			zone if it does not.

			zone if it does not.

	kmemtrace.enable=	[KNL,KMEMTRACE] Format: { yes | no }

				Controls whether kmemtrace is enabled

				at boot-time.

	kmemtrace.subbufs=n	[KNL,KMEMTRACE] Overrides the number of

			subbufs kmemtrace's relay channel has. Set this

			higher than default (KMEMTRACE_N_SUBBUFS in code) if

			you experience buffer overruns.

	movablecore=nn[KMG]	[KNL,X86-32,IA-64,PPC,X86-64] This parameter

	movablecore=nn[KMG]	[KNL,X86-32,IA-64,PPC,X86-64] This parameter

			is similar to kernelcore except it specifies the

			is similar to kernelcore except it specifies the

			amount of memory used for migratable allocations.

			amount of memory used for migratable allocations.

			kmemtrace - Kernel Memory Tracer

			  by Eduard - Gabriel Munteanu

			     <eduard.munteanu@linux360.ro>

I. Introduction

===============

kmemtrace helps kernel developers figure out two things:

1) how different allocators (SLAB, SLUB etc.) perform

2) how kernel code allocates memory and how much

To do this, we trace every allocation and export information to the userspace

through the relay interface. We export things such as the number of requested

bytes, the number of bytes actually allocated (i.e. including internal

fragmentation), whether this is a slab allocation or a plain kmalloc() and so

on.

The actual analysis is performed by a userspace tool (see section III for

details on where to get it from). It logs the data exported by the kernel,

processes it and (as of writing this) can provide the following information:

- the total amount of memory allocated and fragmentation per call-site

- the amount of memory allocated and fragmentation per allocation

- total memory allocated and fragmentation in the collected dataset

- number of cross-CPU allocation and frees (makes sense in NUMA environments)

Moreover, it can potentially find inconsistent and erroneous behavior in

kernel code, such as using slab free functions on kmalloc'ed memory or

allocating less memory than requested (but not truly failed allocations).

kmemtrace also makes provisions for tracing on some arch and analysing the

data on another.

II. Design and goals

====================

kmemtrace was designed to handle rather large amounts of data. Thus, it uses

the relay interface to export whatever is logged to userspace, which then

stores it. Analysis and reporting is done asynchronously, that is, after the

data is collected and stored. By design, it allows one to log and analyse

on different machines and different arches.

As of writing this, the ABI is not considered stable, though it might not

change much. However, no guarantees are made about compatibility yet. When

deemed stable, the ABI should still allow easy extension while maintaining

backward compatibility. This is described further in Documentation/ABI.

Summary of design goals:

	- allow logging and analysis to be done across different machines

	- be fast and anticipate usage in high-load environments (*)

	- be reasonably extensible

	- make it possible for GNU/Linux distributions to have kmemtrace

	included in their repositories

(*) - one of the reasons Pekka Enberg's original userspace data analysis

    tool's code was rewritten from Perl to C (although this is more than a

    simple conversion)

III. Quick usage guide

======================

1) Get a kernel that supports kmemtrace and build it accordingly (i.e. enable

CONFIG_KMEMTRACE).

2) Get the userspace tool and build it:

$ git-clone git://repo.or.cz/kmemtrace-user.git		# current repository

$ cd kmemtrace-user/

$ ./autogen.sh

$ ./configure

$ make

3) Boot the kmemtrace-enabled kernel if you haven't, preferably in the

'single' runlevel (so that relay buffers don't fill up easily), and run

kmemtrace:

# '$' does not mean user, but root here.

$ mount -t debugfs none /sys/kernel/debug

$ mount -t proc none /proc

$ cd path/to/kmemtrace-user/

$ ./kmemtraced

Wait a bit, then stop it with CTRL+C.

$ cat /sys/kernel/debug/kmemtrace/total_overruns	# Check if we didn't

							# overrun, should

							# be zero.

$ (Optionally) [Run kmemtrace_check separately on each cpu[0-9]*.out file to

		check its correctness]

$ ./kmemtrace-report

Now you should have a nice and short summary of how the allocator performs.

IV. FAQ and known issues

========================

Q: 'cat /sys/kernel/debug/kmemtrace/total_overruns' is non-zero, how do I fix

this? Should I worry?

A: If it's non-zero, this affects kmemtrace's accuracy, depending on how

large the number is. You can fix it by supplying a higher

'kmemtrace.subbufs=N' kernel parameter.

---

Q: kmemtrace_check reports errors, how do I fix this? Should I worry?

A: This is a bug and should be reported. It can occur for a variety of

reasons:

	- possible bugs in relay code

	- possible misuse of relay by kmemtrace

	- timestamps being collected unorderly

Or you may fix it yourself and send us a patch.

---

Q: kmemtrace_report shows many errors, how do I fix this? Should I worry?

A: This is a known issue and I'm working on it. These might be true errors

in kernel code, which may have inconsistent behavior (e.g. allocating memory

with kmem_cache_alloc() and freeing it with kfree()). Pekka Enberg pointed

out this behavior may work with SLAB, but may fail with other allocators.

It may also be due to lack of tracing in some unusual allocator functions.

We don't want bug reports regarding this issue yet.

---

V. See also

===========

Documentation/kernel-parameters.txt

Documentation/ABI/testing/debugfs-kmemtrace

L:	kgdb-bugreport@lists.sourceforge.net

L:	kgdb-bugreport@lists.sourceforge.net

S:	Maintained

S:	Maintained

KMEMTRACE

P:	Eduard - Gabriel Munteanu

M:	eduard.munteanu@linux360.ro

L:	linux-kernel@vger.kernel.org

S:	Maintained

KPROBES

KPROBES

P:	Ananth N Mavinakayanahalli

P:	Ananth N Mavinakayanahalli

M:	ananth@in.ibm.com

M:	ananth@in.ibm.com

/*

 * Copyright (C) 2008 Eduard - Gabriel Munteanu

 *

 * This file is released under GPL version 2.

 */

#ifndef _LINUX_KMEMTRACE_H

#define _LINUX_KMEMTRACE_H

#ifdef __KERNEL__

#include <linux/types.h>

#include <linux/marker.h>

enum kmemtrace_type_id {

	KMEMTRACE_TYPE_KMALLOC = 0,	/* kmalloc() or kfree(). */

	KMEMTRACE_TYPE_CACHE,		/* kmem_cache_*(). */

	KMEMTRACE_TYPE_PAGES,		/* __get_free_pages() and friends. */

};

#ifdef CONFIG_KMEMTRACE

extern void kmemtrace_init(void);

static inline void kmemtrace_mark_alloc_node(enum kmemtrace_type_id type_id,

					     unsigned long call_site,

					     const void *ptr,

					     size_t bytes_req,

					     size_t bytes_alloc,

					     gfp_t gfp_flags,

					     int node)

{

	trace_mark(kmemtrace_alloc, "type_id %d call_site %lu ptr %lu "

		   "bytes_req %lu bytes_alloc %lu gfp_flags %lu node %d",

		   type_id, call_site, (unsigned long) ptr,

		   (unsigned long) bytes_req, (unsigned long) bytes_alloc,

		   (unsigned long) gfp_flags, node);

}

static inline void kmemtrace_mark_free(enum kmemtrace_type_id type_id,

				       unsigned long call_site,

				       const void *ptr)

{

	trace_mark(kmemtrace_free, "type_id %d call_site %lu ptr %lu",

		   type_id, call_site, (unsigned long) ptr);

}

#else /* CONFIG_KMEMTRACE */

static inline void kmemtrace_init(void)

{

}

static inline void kmemtrace_mark_alloc_node(enum kmemtrace_type_id type_id,

					     unsigned long call_site,

					     const void *ptr,

					     size_t bytes_req,

					     size_t bytes_alloc,

					     gfp_t gfp_flags,

					     int node)

{

}

static inline void kmemtrace_mark_free(enum kmemtrace_type_id type_id,

				       unsigned long call_site,

				       const void *ptr)

{

}

#endif /* CONFIG_KMEMTRACE */

static inline void kmemtrace_mark_alloc(enum kmemtrace_type_id type_id,

					unsigned long call_site,

					const void *ptr,

					size_t bytes_req,

					size_t bytes_alloc,

					gfp_t gfp_flags)

{

	kmemtrace_mark_alloc_node(type_id, call_site, ptr,

				  bytes_req, bytes_alloc, gfp_flags, -1);

}

#endif /* __KERNEL__ */

#endif /* _LINUX_KMEMTRACE_H */