Merge branch 'dev-tools' into doc/4.9

Copyright 2010 Nicolas Palix <npalix@diku.dk>

Copyright 2010 Julia Lawall <julia@diku.dk>

Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr>

.. Copyright 2010 Nicolas Palix <npalix@diku.dk>

.. Copyright 2010 Julia Lawall <julia@diku.dk>

.. Copyright 2010 Gilles Muller <Gilles.Muller@lip6.fr>

.. highlight:: none

Coccinelle

==========

Coccinelle is a tool for pattern matching and text transformation that has

many uses in kernel development, including the application of complex,

tree-wide patches and detection of problematic programming patterns.

Getting Coccinelle

~~~~~~~~~~~~~~~~~~~~

-------------------

The semantic patches included in the kernel use features and options

which are provided by Coccinelle version 1.0.0-rc11 and above.

 - NetBSD

 - FreeBSD

You can get the latest version released from the Coccinelle homepage at

http://coccinelle.lip6.fr/

Information and tips about Coccinelle are also provided on the wiki

pages at http://cocci.ekstranet.diku.dk/wiki/doku.php

Once you have it, run the following command:

Once you have it, run the following command::

     	./configure

        make

as a regular user, and install it with

as a regular user, and install it with::

        sudo make install

Supplemental documentation

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---------------------------

For supplemental documentation refer to the wiki:

The wiki documentation always refers to the linux-next version of the script.

Using Coccinelle on the Linux kernel

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

------------------------------------

A Coccinelle-specific target is defined in the top level

Makefile. This target is named 'coccicheck' and calls the 'coccicheck'

front-end in the 'scripts' directory.

Makefile. This target is named ``coccicheck`` and calls the ``coccicheck``

front-end in the ``scripts`` directory.

Four basic modes are defined: patch, report, context, and org. The mode to

use is specified by setting the MODE variable with 'MODE=<mode>'.

Four basic modes are defined: ``patch``, ``report``, ``context``, and

``org``. The mode to use is specified by setting the MODE variable with

``MODE=<mode>``.

'patch' proposes a fix, when possible.

- ``patch`` proposes a fix, when possible.

'report' generates a list in the following format:

- ``report`` generates a list in the following format:

  file:line:column-column: message

'context' highlights lines of interest and their context in a

diff-like style.Lines of interest are indicated with '-'.

- ``context`` highlights lines of interest and their context in a

  diff-like style.Lines of interest are indicated with ``-``.

'org' generates a report in the Org mode format of Emacs.

- ``org`` generates a report in the Org mode format of Emacs.

Note that not all semantic patches implement all modes. For easy use

of Coccinelle, the default mode is "report".

Two other modes provide some common combinations of these modes.

'chain' tries the previous modes in the order above until one succeeds.

- ``chain`` tries the previous modes in the order above until one succeeds.

'rep+ctxt' runs successively the report mode and the context mode.

- ``rep+ctxt`` runs successively the report mode and the context mode.

  It should be used with the C option (described later)

  which checks the code on a file basis.

Examples:

	To make a report for every semantic patch, run the following command:

Examples

~~~~~~~~

To make a report for every semantic patch, run the following command::

		make coccicheck MODE=report

	To produce patches, run:

To produce patches, run::

		make coccicheck MODE=patch

The coccicheck target applies every semantic patch available in the

sub-directories of 'scripts/coccinelle' to the entire Linux kernel.

sub-directories of ``scripts/coccinelle`` to the entire Linux kernel.

For each semantic patch, a commit message is proposed.  It gives a

description of the problem being checked by the semantic patch, and

positives. Thus, reports must be carefully checked, and patches

reviewed.

To enable verbose messages set the V= variable, for example:

To enable verbose messages set the V= variable, for example::

   make coccicheck MODE=report V=1

Coccinelle parallelization

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---------------------------

By default, coccicheck tries to run as parallel as possible. To change

the parallelism, set the J= variable. For example, to run across 4 CPUs:

the parallelism, set the J= variable. For example, to run across 4 CPUs::

   make coccicheck MODE=report J=4

if support for this is detected you will benefit from parmap parallelization.

When parmap is enabled coccicheck will enable dynamic load balancing by using

'--chunksize 1' argument, this ensures we keep feeding threads with work

``--chunksize 1`` argument, this ensures we keep feeding threads with work

one by one, so that we avoid the situation where most work gets done by only

a few threads. With dynamic load balancing, if a thread finishes early we keep

feeding it more work.

When parmap is enabled, if an error occurs in Coccinelle, this error

value is propagated back, the return value of the 'make coccicheck'

value is propagated back, the return value of the ``make coccicheck``

captures this return value.

Using Coccinelle with a single semantic patch

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---------------------------------------------

The optional make variable COCCI can be used to check a single

semantic patch. In that case, the variable must be initialized with

the name of the semantic patch to apply.

For instance:

For instance::

	make coccicheck COCCI=<my_SP.cocci> MODE=patch

or

or::

	make coccicheck COCCI=<my_SP.cocci> MODE=report

Controlling Which Files are Processed by Coccinelle

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---------------------------------------------------

By default the entire kernel source tree is checked.

To apply Coccinelle to a specific directory, M= can be used.

For example, to check drivers/net/wireless/ one may write:

To apply Coccinelle to a specific directory, ``M=`` can be used.

For example, to check drivers/net/wireless/ one may write::

    make coccicheck M=drivers/net/wireless/

To apply Coccinelle on a file basis, instead of a directory basis, the

following command may be used:

following command may be used::

    make C=1 CHECK="scripts/coccicheck"

To check only newly edited code, use the value 2 for the C flag, i.e.

To check only newly edited code, use the value 2 for the C flag, i.e.::

    make C=2 CHECK="scripts/coccicheck"

MODE variable explained above.

Debugging Coccinelle SmPL patches

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---------------------------------

Using coccicheck is best as it provides in the spatch command line

include options matching the options used when we compile the kernel.

Alternatively you can debug running Coccinelle against SmPL patches

by asking for stderr to be redirected to stderr, by default stderr

is redirected to /dev/null, if you'd like to capture stderr you

can specify the DEBUG_FILE="file.txt" option to coccicheck. For

instance:

can specify the ``DEBUG_FILE="file.txt"`` option to coccicheck. For

instance::

    rm -f cocci.err

    make coccicheck COCCI=scripts/coccinelle/free/kfree.cocci MODE=report DEBUG_FILE=cocci.err

You can use SPFLAGS to add debugging flags, for instance you may want to

add both --profile --show-trying to SPFLAGS when debugging. For instance

you may want to use:

you may want to use::

    rm -f err.log

    export COCCI=scripts/coccinelle/misc/irqf_oneshot.cocci

DEBUG_FILE support is only supported when using coccinelle >= 1.2.

.cocciconfig support

~~~~~~~~~~~~~~~~~~~~~~

--------------------

Coccinelle supports reading .cocciconfig for default Coccinelle options that

should be used every time spatch is spawned, the order of precedence for

variables for .cocciconfig is as follows:

  o Your current user's home directory is processed first

  o Your directory from which spatch is called is processed next

  o The directory provided with the --dir option is processed last, if used

- Your current user's home directory is processed first

- Your directory from which spatch is called is processed next

- The directory provided with the --dir option is processed last, if used

Since coccicheck runs through make, it naturally runs from the kernel

proper dir, as such the second rule above would be implied for picking up a

.cocciconfig when using 'make coccicheck'.

.cocciconfig when using ``make coccicheck``.

'make coccicheck' also supports using M= targets.If you do not supply

``make coccicheck`` also supports using M= targets.If you do not supply

any M= target, it is assumed you want to target the entire kernel.

The kernel coccicheck script has:

The kernel coccicheck script has::

    if [ "$KBUILD_EXTMOD" = "" ] ; then

        OPTIONS="--dir $srctree $COCCIINCLUDE"

We help Coccinelle when used against Linux with a set of sensible defaults

options for Linux with our own Linux .cocciconfig. This hints to coccinelle

git can be used for 'git grep' queries over coccigrep. A timeout of 200

git can be used for ``git grep`` queries over coccigrep. A timeout of 200

seconds should suffice for now.

The options picked up by coccinelle when reading a .cocciconfig do not appear

as arguments to spatch processes running on your system, to confirm what

options will be used by Coccinelle run:

options will be used by Coccinelle run::

      spatch --print-options-only

idutils.

Additional flags

~~~~~~~~~~~~~~~~~~

----------------

Additional flags can be passed to spatch through the SPFLAGS

variable. This works as Coccinelle respects the last flags

given to it when options are in conflict.

given to it when options are in conflict. ::

    make SPFLAGS=--use-glimpse coccicheck

Coccinelle supports idutils as well but requires coccinelle >= 1.0.6.

When no ID file is specified coccinelle assumes your ID database file

is in the file .id-utils.index on the top level of the kernel, coccinelle

carries a script scripts/idutils_index.sh which creates the database with

carries a script scripts/idutils_index.sh which creates the database with::

    mkid -i C --output .id-utils.index

If you have another database filename you can also just symlink with this

name.

name. ::

    make SPFLAGS=--use-idutils coccicheck

Alternatively you can specify the database filename explicitly, for

instance:

instance::

    make SPFLAGS="--use-idutils /full-path/to/ID" coccicheck

See spatch --help to learn more about spatch options.

See ``spatch --help`` to learn more about spatch options.

Note that the '--use-glimpse' and '--use-idutils' options

Note that the ``--use-glimpse`` and ``--use-idutils`` options

require external tools for indexing the code. None of them is

thus active by default. However, by indexing the code with

one of these tools, and according to the cocci file used,

spatch could proceed the entire code base more quickly.

SmPL patch specific options

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

---------------------------

SmPL patches can have their own requirements for options passed

to Coccinelle. SmPL patch specific options can be provided by

providing them at the top of the SmPL patch, for instance:

providing them at the top of the SmPL patch, for instance::

	// Options: --no-includes --include-headers

SmPL patch Coccinelle requirements

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

----------------------------------

As Coccinelle features get added some more advanced SmPL patches

may require newer versions of Coccinelle. If an SmPL patch requires

at least a version of Coccinelle, this can be specified as follows,

as an example if requiring at least Coccinelle >= 1.0.5:

as an example if requiring at least Coccinelle >= 1.0.5::

	// Requires: 1.0.5

Proposing new semantic patches

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-------------------------------

New semantic patches can be proposed and submitted by kernel

developers. For sake of clarity, they should be organized in the

sub-directories of 'scripts/coccinelle/'.

sub-directories of ``scripts/coccinelle/``.

Detailed description of the ``report`` mode

-------------------------------------------

 Detailed description of the 'report' mode

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``report`` generates a list in the following format::

'report' generates a list in the following format:

  file:line:column-column: message

Example:

Example

~~~~~~~

Running

Running::

	make coccicheck MODE=report COCCI=scripts/coccinelle/api/err_cast.cocci

will execute the following part of the SmPL script.

will execute the following part of the SmPL script::

   <smpl>

   @r depends on !context && !patch && (org || report)@

   </smpl>

This SmPL excerpt generates entries on the standard output, as

illustrated below:

illustrated below::

    /home/user/linux/crypto/ctr.c:188:9-16: ERR_CAST can be used with alg

    /home/user/linux/crypto/authenc.c:619:9-16: ERR_CAST can be used with auth

    /home/user/linux/crypto/xts.c:227:9-16: ERR_CAST can be used with alg

 Detailed description of the 'patch' mode

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Detailed description of the ``patch`` mode

------------------------------------------

When the 'patch' mode is available, it proposes a fix for each problem

When the ``patch`` mode is available, it proposes a fix for each problem

identified.

Example:

Example

~~~~~~~

Running::

Running

	make coccicheck MODE=patch COCCI=scripts/coccinelle/api/err_cast.cocci

will execute the following part of the SmPL script.

will execute the following part of the SmPL script::

    <smpl>

    @ depends on !context && patch && !org && !report @

    </smpl>

This SmPL excerpt generates patch hunks on the standard output, as

illustrated below:

illustrated below::

    diff -u -p a/crypto/ctr.c b/crypto/ctr.c

    --- a/crypto/ctr.c 2010-05-26 10:49:38.000000000 +0200

 	/* Block size must be >= 4 bytes. */

 	err = -EINVAL;

 Detailed description of the 'context' mode

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Detailed description of the ``context`` mode

--------------------------------------------

'context' highlights lines of interest and their context

``context`` highlights lines of interest and their context

in a diff-like style.

NOTE: The diff-like output generated is NOT an applicable patch. The

      intent of the 'context' mode is to highlight the important lines

      (annotated with minus, '-') and gives some surrounding context

      **NOTE**: The diff-like output generated is NOT an applicable patch. The

      intent of the ``context`` mode is to highlight the important lines

      (annotated with minus, ``-``) and gives some surrounding context

      lines around. This output can be used with the diff mode of

      Emacs to review the code.

Example:

Example

~~~~~~~

Running::

Running

	make coccicheck MODE=context COCCI=scripts/coccinelle/api/err_cast.cocci

will execute the following part of the SmPL script.

will execute the following part of the SmPL script::

    <smpl>

    @ depends on context && !patch && !org && !report@

    </smpl>

This SmPL excerpt generates diff hunks on the standard output, as

illustrated below:

illustrated below::

    diff -u -p /home/user/linux/crypto/ctr.c /tmp/nothing

    --- /home/user/linux/crypto/ctr.c	2010-05-26 10:49:38.000000000 +0200

 	/* Block size must be >= 4 bytes. */

 	err = -EINVAL;

 Detailed description of the 'org' mode

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Detailed description of the ``org`` mode

----------------------------------------

``org`` generates a report in the Org mode format of Emacs.

'org' generates a report in the Org mode format of Emacs.

Example

~~~~~~~

Example:

Running::

Running

	make coccicheck MODE=org COCCI=scripts/coccinelle/api/err_cast.cocci

will execute the following part of the SmPL script.

will execute the following part of the SmPL script::

    <smpl>

    @r depends on !context && !patch && (org || report)@

    </smpl>

This SmPL excerpt generates Org entries on the standard output, as

illustrated below:

illustrated below::

    * TODO [[view:/home/user/linux/crypto/ctr.c::face=ovl-face1::linb=188::colb=9::cole=16][ERR_CAST can be used with alg]]

    * TODO [[view:/home/user/linux/crypto/authenc.c::face=ovl-face1::linb=619::colb=9::cole=16][ERR_CAST can be used with auth]]

Using gcov with the Linux kernel

================================

1. Introduction

2. Preparation

3. Customization

4. Files

5. Modules

6. Separated build and test machines

7. Troubleshooting

Appendix A: sample script: gather_on_build.sh

Appendix B: sample script: gather_on_test.sh

1. Introduction

===============

gcov profiling kernel support enables the use of GCC's coverage testing

tool gcov [1] with the Linux kernel. Coverage data of a running kernel

tool gcov_ with the Linux kernel. Coverage data of a running kernel

is exported in gcov-compatible format via the "gcov" debugfs directory.

To get coverage data for a specific file, change to the kernel build

directory and use gcov with the -o option as follows (requires root):

directory and use gcov with the ``-o`` option as follows (requires root)::

    # cd /tmp/linux-out

    # gcov -o /sys/kernel/debug/gcov/tmp/linux-out/kernel spinlock.c

This will create source code files annotated with execution counts

in the current directory. In addition, graphical gcov front-ends such

as lcov [2] can be used to automate the process of collecting data

as lcov_ can be used to automate the process of collecting data

for the entire kernel and provide coverage overviews in HTML format.

Possible uses:

* minimizing kernel configurations (do I need this option if the

  associated code is never run?)

--

[1] http://gcc.gnu.org/onlinedocs/gcc/Gcov.html

[2] http://ltp.sourceforge.net/coverage/lcov.php

.. _gcov: http://gcc.gnu.org/onlinedocs/gcc/Gcov.html

.. _lcov: http://ltp.sourceforge.net/coverage/lcov.php

2. Preparation

==============

Preparation

-----------

Configure the kernel with:

Configure the kernel with::

        CONFIG_DEBUG_FS=y

        CONFIG_GCOV_KERNEL=y

select the gcc's gcov format, default is autodetect based on gcc version:

select the gcc's gcov format, default is autodetect based on gcc version::

        CONFIG_GCOV_FORMAT_AUTODETECT=y

and to get coverage data for the entire kernel:

and to get coverage data for the entire kernel::

        CONFIG_GCOV_PROFILE_ALL=y

on all architectures.

Profiling data will only become accessible once debugfs has been

mounted:

mounted::

        mount -t debugfs none /sys/kernel/debug

3. Customization

================

Customization

-------------

To enable profiling for specific files or directories, add a line

similar to the following to the respective kernel Makefile:

        For a single file (e.g. main.o):

- For a single file (e.g. main.o)::

	GCOV_PROFILE_main.o := y

        For all files in one directory:

- For all files in one directory::

	GCOV_PROFILE := y

To exclude files from being profiled even when CONFIG_GCOV_PROFILE_ALL

is specified, use:

is specified, use::

	GCOV_PROFILE_main.o := n

        and:

and::

	GCOV_PROFILE := n

Only files which are linked to the main kernel image or are compiled as

kernel modules are supported by this mechanism.

4. Files

========

Files

-----

The gcov kernel support creates the following files in debugfs:

        /sys/kernel/debug/gcov

``/sys/kernel/debug/gcov``

	Parent directory for all gcov-related files.

        /sys/kernel/debug/gcov/reset

``/sys/kernel/debug/gcov/reset``

	Global reset file: resets all coverage data to zero when

        written to.

        /sys/kernel/debug/gcov/path/to/compile/dir/file.gcda

``/sys/kernel/debug/gcov/path/to/compile/dir/file.gcda``

	The actual gcov data file as understood by the gcov

        tool. Resets file coverage data to zero when written to.

        /sys/kernel/debug/gcov/path/to/compile/dir/file.gcno

``/sys/kernel/debug/gcov/path/to/compile/dir/file.gcno``

	Symbolic link to a static data file required by the gcov

        tool. This file is generated by gcc when compiling with

                option -ftest-coverage.

        option ``-ftest-coverage``.

5. Modules

==========

Modules

-------

Kernel modules may contain cleanup code which is only run during

module unload time. The gcov mechanism provides a means to collect

initialized with the data from its previous instantiation.

This behavior can be deactivated by specifying the gcov_persist kernel

parameter:

parameter::

        gcov_persist=0

module by writing to its data file or the global reset file.

6. Separated build and test machines

====================================

Separated build and test machines

---------------------------------

The gcov kernel profiling infrastructure is designed to work out-of-the

box for setups where kernels are built and run on the same machine. In

    These files can be copied to any location on the build machine. gcov

    must then be called with the -o option pointing to that directory.

Example directory setup on the build machine:

    Example directory setup on the build machine::

      /tmp/linux:    kernel source tree

      /tmp/out:      kernel build directory as specified by make O=

      [user@build] gcov -o /tmp/coverage/tmp/out/init main.c

7. Troubleshooting

==================

Troubleshooting

---------------

Problem

    Compilation aborts during linker step.

Problem:  Compilation aborts during linker step.

Cause:    Profiling flags are specified for source files which are not

Cause

    Profiling flags are specified for source files which are not

    linked to the main kernel or which are linked by a custom

    linker procedure.

Solution: Exclude affected source files from profiling by specifying

          GCOV_PROFILE := n or GCOV_PROFILE_basename.o := n in the

Solution

    Exclude affected source files from profiling by specifying

    ``GCOV_PROFILE := n`` or ``GCOV_PROFILE_basename.o := n`` in the

    corresponding Makefile.

Problem:  Files copied from sysfs appear empty or incomplete.

Cause:    Due to the way seq_file works, some tools such as cp or tar

Problem

    Files copied from sysfs appear empty or incomplete.

Cause

    Due to the way seq_file works, some tools such as cp or tar

    may not correctly copy files from sysfs.

Solution: Use 'cat' to read .gcda files and 'cp -d' to copy links.

Solution

    Use ``cat``' to read ``.gcda`` files and ``cp -d`` to copy links.

    Alternatively use the mechanism shown in Appendix B.

Appendix A: gather_on_build.sh

==============================

------------------------------

Sample script to gather coverage meta files on the build machine

(see 6a):

(see 6a)::

    #!/bin/bash

    KSRC=$1

Appendix B: gather_on_test.sh

=============================

-----------------------------

Sample script to gather coverage data files on the test machine

(see 6b):

(see 6b)::

    #!/bin/bash -e