Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 10ffebbe authored by Mauro Carvalho Chehab's avatar Mauro Carvalho Chehab Committed by Jonathan Corbet
Browse files

docs: fault-injection: convert docs to ReST and rename to *.rst 



The conversion is actually:
  - add blank lines and identation in order to identify paragraphs;
  - fix tables markups;
  - add some lists markups;
  - mark literal blocks;
  - adjust title markups.

At its new index.rst, let's add a :orphan: while this is not linked to
the main index.rst file, in order to avoid build warnings.

Signed-off-by: default avatarMauro Carvalho Chehab <mchehab+samsung@kernel.org>
Acked-by: default avatarFederico Vaga <federico.vaga@vaga.pv.it>
Signed-off-by: default avatarJonathan Corbet <corbet@lwn.net>
parent f0ba4377

Documentation/fault-injection/fault-injection.txt→Documentation/fault-injection/fault-injection.rst

+146 −135
Original line number Diff line number Diff line 
===========================================

Fault injection capabilities infrastructure

===========================================
@@ -7,36 +8,36 @@ See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug. 
Available fault injection capabilities

--------------------------------------



o failslab

- failslab



  injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)



o fail_page_alloc

- fail_page_alloc



  injects page allocation failures. (alloc_pages(), get_free_pages(), ...)



o fail_futex

- fail_futex



  injects futex deadlock and uaddr fault errors.



o fail_make_request

- fail_make_request



  injects disk IO errors on devices permitted by setting

  /sys/block/<device>/make-it-fail or

  /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())



o fail_mmc_request

- fail_mmc_request



  injects MMC data errors on devices permitted by setting

  debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request



o fail_function

- fail_function



  injects error return on specific functions, which are marked by

  ALLOW_ERROR_INJECTION() macro, by setting debugfs entries

  under /sys/kernel/debug/fail_function. No boot option supported.



o NVMe fault injection

- NVMe fault injection



  inject NVMe status code and retry flag on devices permitted by setting

  debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default
@@ -47,7 +48,8 @@ o NVMe fault injection 
Configure fault-injection capabilities behavior

-----------------------------------------------



o debugfs entries

debugfs entries

^^^^^^^^^^^^^^^



fault-inject-debugfs kernel module provides some debugfs entries for runtime

configuration of fault-injection capabilities.
@@ -55,6 +57,7 @@ configuration of fault-injection capabilities. 
- /sys/kernel/debug/fail*/probability:



	likelihood of failure injection, in percent.



	Format: <percent>



	Note that one-failure-per-hundred is a very high error rate
@@ -83,6 +86,7 @@ configuration of fault-injection capabilities. 
- /sys/kernel/debug/fail*/verbose



	Format: { 0 | 1 | 2 }



	specifies the verbosity of the messages when failure is

	injected.  '0' means no messages; '1' will print only a single

	log line per failure; '2' will print a call trace too -- useful
@@ -91,14 +95,15 @@ configuration of fault-injection capabilities. 
- /sys/kernel/debug/fail*/task-filter:



	Format: { 'Y' | 'N' }



	A value of 'N' disables filtering by process (default).

	Any positive value limits failures to only processes indicated by

	/proc/<pid>/make-it-fail==1.



- /sys/kernel/debug/fail*/require-start:

- /sys/kernel/debug/fail*/require-end:

- /sys/kernel/debug/fail*/reject-start:

- /sys/kernel/debug/fail*/reject-end:

- /sys/kernel/debug/fail*/require-start,

  /sys/kernel/debug/fail*/require-end,

  /sys/kernel/debug/fail*/reject-start,

  /sys/kernel/debug/fail*/reject-end:



	specifies the range of virtual addresses tested during

	stacktrace walking.  Failure is injected only if some caller
@@ -116,6 +121,7 @@ configuration of fault-injection capabilities. 
- /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:



	Format: { 'Y' | 'N' }



	default is 'N', setting it to 'Y' won't inject failures into

	highmem/user allocations.
@@ -123,6 +129,7 @@ configuration of fault-injection capabilities. 
- /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:



	Format: { 'Y' | 'N' }



	default is 'N', setting it to 'Y' will inject failures

	only into non-sleep allocations (GFP_ATOMIC allocations).
@@ -134,12 +141,14 @@ configuration of fault-injection capabilities. 
- /sys/kernel/debug/fail_futex/ignore-private:



	Format: { 'Y' | 'N' }



	default is 'N', setting it to 'Y' will disable failure injections

	when dealing with private (address space) futexes.



- /sys/kernel/debug/fail_function/inject:



	Format: { 'function-name' | '!function-name' | '' }



	specifies the target function of error injection by name.

	If the function name leads '!' prefix, given function is

	removed from injection list. If nothing specified ('')
@@ -160,10 +169,11 @@ configuration of fault-injection capabilities. 
	function for given function. This will be created when

	user specifies new injection entry.



o Boot option

Boot option

^^^^^^^^^^^



In order to inject faults while debugfs is not available (early boot time),

use the boot option:

use the boot option::



	failslab=

	fail_page_alloc=
@@ -171,10 +181,11 @@ use the boot option: 
	fail_futex=

	mmc_core.fail_request=<interval>,<probability>,<space>,<times>



o proc entries

proc entries

^^^^^^^^^^^^



- /proc/<pid>/fail-nth:

- /proc/self/task/<tid>/fail-nth:

- /proc/<pid>/fail-nth,

  /proc/self/task/<tid>/fail-nth:



	Write to this file of integer N makes N-th call in the task fail.

	Read from this file returns a integer value. A value of '0' indicates
@@ -191,16 +202,16 @@ o proc entries 
How to add new fault injection capability

-----------------------------------------



o #include <linux/fault-inject.h>

- #include <linux/fault-inject.h>



o define the fault attributes

- define the fault attributes



  DECLARE_FAULT_ATTR(name);



  Please see the definition of struct fault_attr in fault-inject.h

  for details.



o provide a way to configure fault attributes

- provide a way to configure fault attributes



- boot option
@@ -222,16 +233,16 @@ o provide a way to configure fault attributes 
  single kernel module, it is better to provide module parameters to

  configure the fault attributes.



o add a hook to insert failures

- add a hook to insert failures



  Upon should_fail() returning true, client code should inject a failure.

  Upon should_fail() returning true, client code should inject a failure:



	should_fail(attr, size);



Application Examples

--------------------



o Inject slab allocation failures into module init/exit code

- Inject slab allocation failures into module init/exit code::



    #!/bin/bash
@@ -266,7 +277,7 @@ done 


------------------------------------------------------------------------------



o Inject page allocation failures only for a specific module

- Inject page allocation failures only for a specific module::



    #!/bin/bash
@@ -307,7 +318,7 @@ sleep 1000000 


------------------------------------------------------------------------------



o Inject open_ctree error while btrfs mount

- Inject open_ctree error while btrfs mount::



    #!/bin/bash
@@ -354,19 +365,19 @@ see the following examples. 
Examples:



Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab

allocation failure.

allocation failure::



	# ./tools/testing/fault-injection/failcmd.sh \

		-- make -C tools/testing/selftests/ run_tests



Same as above except to specify 100 times failures at most instead of one time

at most by default.

at most by default::



	# ./tools/testing/fault-injection/failcmd.sh --times=100 \

		-- make -C tools/testing/selftests/ run_tests



Same as above except to inject page allocation failure instead of slab

allocation failure.

allocation failure::



	# env FAILCMD_TYPE=fail_page_alloc \

		./tools/testing/fault-injection/failcmd.sh --times=100 \
@@ -376,7 +387,7 @@ Systematic faults using fail-nth 
---------------------------------



The following code systematically faults 0-th, 1-st, 2-nd and so on

capabilities in the socketpair() system call.

capabilities in the socketpair() system call::



  #include <sys/types.h>

  #include <sys/stat.h>
@@ -415,7 +426,7 @@ int main() 
	return 0;

  }



An example output:

An example output::



	1-th fault Y: res=-1/23

	2-th fault Y: res=-1/23

Documentation/fault-injection/index.rst

0 → 100644
+20 −0
Original line number Diff line number Diff line 
:orphan:



===============

fault-injection

===============



.. toctree::

    :maxdepth: 1



    fault-injection

    notifier-error-inject

    nvme-fault-injection

    provoke-crashes



.. only::  subproject and html



   Indices

   =======



   * :ref:`genindex`

Documentation/fault-injection/notifier-error-inject.txt→Documentation/fault-injection/notifier-error-inject.rst

+11 −7
Original line number Diff line number Diff line
@@ -14,6 +14,7 @@ modules that can be used to test the following notifiers. 
PM notifier error injection module

----------------------------------

This feature is controlled through debugfs interface



  /sys/kernel/debug/notifier-error-inject/pm/actions/<notifier event>/error



Possible PM notifier events to be failed are:
@@ -22,7 +23,7 @@ Possible PM notifier events to be failed are: 
 * PM_SUSPEND_PREPARE

 * PM_RESTORE_PREPARE



Example: Inject PM suspend error (-12 = -ENOMEM)

Example: Inject PM suspend error (-12 = -ENOMEM)::



	# cd /sys/kernel/debug/notifier-error-inject/pm/

	# echo -12 > actions/PM_SUSPEND_PREPARE/error
@@ -32,6 +33,7 @@ Example: Inject PM suspend error (-12 = -ENOMEM) 
Memory hotplug notifier error injection module

----------------------------------------------

This feature is controlled through debugfs interface



  /sys/kernel/debug/notifier-error-inject/memory/actions/<notifier event>/error



Possible memory notifier events to be failed are:
@@ -39,7 +41,7 @@ Possible memory notifier events to be failed are: 
 * MEM_GOING_ONLINE

 * MEM_GOING_OFFLINE



Example: Inject memory hotplug offline error (-12 == -ENOMEM)

Example: Inject memory hotplug offline error (-12 == -ENOMEM)::



	# cd /sys/kernel/debug/notifier-error-inject/memory

	# echo -12 > actions/MEM_GOING_OFFLINE/error
@@ -49,6 +51,7 @@ Example: Inject memory hotplug offline error (-12 == -ENOMEM) 
powerpc pSeries reconfig notifier error injection module

--------------------------------------------------------

This feature is controlled through debugfs interface



  /sys/kernel/debug/notifier-error-inject/pSeries-reconfig/actions/<notifier event>/error



Possible pSeries reconfig notifier events to be failed are:
@@ -61,6 +64,7 @@ Possible pSeries reconfig notifier events to be failed are: 
Netdevice notifier error injection module

----------------------------------------------

This feature is controlled through debugfs interface



  /sys/kernel/debug/notifier-error-inject/netdev/actions/<notifier event>/error



Netdevice notifier events which can be failed are:
@@ -75,7 +79,7 @@ Netdevice notifier events which can be failed are: 
 * NETDEV_PRECHANGEUPPER

 * NETDEV_CHANGEUPPER



Example: Inject netdevice mtu change error (-22 == -EINVAL)

Example: Inject netdevice mtu change error (-22 == -EINVAL)::



	# cd /sys/kernel/debug/notifier-error-inject/netdev

	# echo -22 > actions/NETDEV_CHANGEMTU/error

Documentation/fault-injection/nvme-fault-injection.rst

0 → 100644
+120 −0
Original line number Diff line number Diff line 
NVMe Fault Injection

====================

Linux's fault injection framework provides a systematic way to support

error injection via debugfs in the /sys/kernel/debug directory. When

enabled, the default NVME_SC_INVALID_OPCODE with no retry will be

injected into the nvme_end_request. Users can change the default status

code and no retry flag via the debugfs. The list of Generic Command

Status can be found in include/linux/nvme.h



Following examples show how to inject an error into the nvme.



First, enable CONFIG_FAULT_INJECTION_DEBUG_FS kernel config,

recompile the kernel. After booting up the kernel, do the

following.



Example 1: Inject default status code with no retry

---------------------------------------------------



::



  mount /dev/nvme0n1 /mnt

  echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times

  echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability

  cp a.file /mnt



Expected Result::



  cp: cannot stat ‘/mnt/a.file’: Input/output error



Message from dmesg::



  FAULT_INJECTION: forcing a failure.

  name fault_inject, interval 1, probability 100, space 0, times 1

  CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc8+ #2

  Hardware name: innotek GmbH VirtualBox/VirtualBox,

  BIOS VirtualBox 12/01/2006

  Call Trace:

    <IRQ>

    dump_stack+0x5c/0x7d

    should_fail+0x148/0x170

    nvme_should_fail+0x2f/0x50 [nvme_core]

    nvme_process_cq+0xe7/0x1d0 [nvme]

    nvme_irq+0x1e/0x40 [nvme]

    __handle_irq_event_percpu+0x3a/0x190

    handle_irq_event_percpu+0x30/0x70

    handle_irq_event+0x36/0x60

    handle_fasteoi_irq+0x78/0x120

    handle_irq+0xa7/0x130

    ? tick_irq_enter+0xa8/0xc0

    do_IRQ+0x43/0xc0

    common_interrupt+0xa2/0xa2

    </IRQ>

  RIP: 0010:native_safe_halt+0x2/0x10

  RSP: 0018:ffffffff82003e90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdd

  RAX: ffffffff817a10c0 RBX: ffffffff82012480 RCX: 0000000000000000

  RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000

  RBP: 0000000000000000 R08: 000000008e38ce64 R09: 0000000000000000

  R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff82012480

  R13: ffffffff82012480 R14: 0000000000000000 R15: 0000000000000000

    ? __sched_text_end+0x4/0x4

    default_idle+0x18/0xf0

    do_idle+0x150/0x1d0

    cpu_startup_entry+0x6f/0x80

    start_kernel+0x4c4/0x4e4

    ? set_init_arg+0x55/0x55

    secondary_startup_64+0xa5/0xb0

    print_req_error: I/O error, dev nvme0n1, sector 9240

  EXT4-fs error (device nvme0n1): ext4_find_entry:1436:

  inode #2: comm cp: reading directory lblock 0



Example 2: Inject default status code with retry

------------------------------------------------



::



  mount /dev/nvme0n1 /mnt

  echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times

  echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability

  echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/status

  echo 0 > /sys/kernel/debug/nvme0n1/fault_inject/dont_retry



  cp a.file /mnt



Expected Result::



  command success without error



Message from dmesg::



  FAULT_INJECTION: forcing a failure.

  name fault_inject, interval 1, probability 100, space 0, times 1

  CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.15.0-rc8+ #4

  Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006

  Call Trace:

    <IRQ>

    dump_stack+0x5c/0x7d

    should_fail+0x148/0x170

    nvme_should_fail+0x30/0x60 [nvme_core]

    nvme_loop_queue_response+0x84/0x110 [nvme_loop]

    nvmet_req_complete+0x11/0x40 [nvmet]

    nvmet_bio_done+0x28/0x40 [nvmet]

    blk_update_request+0xb0/0x310

    blk_mq_end_request+0x18/0x60

    flush_smp_call_function_queue+0x3d/0xf0

    smp_call_function_single_interrupt+0x2c/0xc0

    call_function_single_interrupt+0xa2/0xb0

    </IRQ>

  RIP: 0010:native_safe_halt+0x2/0x10

  RSP: 0018:ffffc9000068bec0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04

  RAX: ffffffff817a10c0 RBX: ffff88011a3c9680 RCX: 0000000000000000

  RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000

  RBP: 0000000000000001 R08: 000000008e38c131 R09: 0000000000000000

  R10: 0000000000000000 R11: 0000000000000000 R12: ffff88011a3c9680

  R13: ffff88011a3c9680 R14: 0000000000000000 R15: 0000000000000000

    ? __sched_text_end+0x4/0x4

    default_idle+0x18/0xf0

    do_idle+0x150/0x1d0

    cpu_startup_entry+0x6f/0x80

    start_secondary+0x187/0x1e0

    secondary_startup_64+0xa5/0xb0

Documentation/fault-injection/nvme-fault-injection.txt

deleted100644 → 0
+0 −116
Original line number Diff line number Diff line 
NVMe Fault Injection

====================

Linux's fault injection framework provides a systematic way to support

error injection via debugfs in the /sys/kernel/debug directory. When

enabled, the default NVME_SC_INVALID_OPCODE with no retry will be

injected into the nvme_end_request. Users can change the default status

code and no retry flag via the debugfs. The list of Generic Command

Status can be found in include/linux/nvme.h



Following examples show how to inject an error into the nvme.



First, enable CONFIG_FAULT_INJECTION_DEBUG_FS kernel config,

recompile the kernel. After booting up the kernel, do the

following.



Example 1: Inject default status code with no retry

---------------------------------------------------



mount /dev/nvme0n1 /mnt

echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times

echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability

cp a.file /mnt



Expected Result:



cp: cannot stat ‘/mnt/a.file’: Input/output error



Message from dmesg:



FAULT_INJECTION: forcing a failure.

name fault_inject, interval 1, probability 100, space 0, times 1

CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc8+ #2

Hardware name: innotek GmbH VirtualBox/VirtualBox,

BIOS VirtualBox 12/01/2006

Call Trace:

  <IRQ>

  dump_stack+0x5c/0x7d

  should_fail+0x148/0x170

  nvme_should_fail+0x2f/0x50 [nvme_core]

  nvme_process_cq+0xe7/0x1d0 [nvme]

  nvme_irq+0x1e/0x40 [nvme]

  __handle_irq_event_percpu+0x3a/0x190

  handle_irq_event_percpu+0x30/0x70

  handle_irq_event+0x36/0x60

  handle_fasteoi_irq+0x78/0x120

  handle_irq+0xa7/0x130

  ? tick_irq_enter+0xa8/0xc0

  do_IRQ+0x43/0xc0

  common_interrupt+0xa2/0xa2

  </IRQ>

RIP: 0010:native_safe_halt+0x2/0x10

RSP: 0018:ffffffff82003e90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdd

RAX: ffffffff817a10c0 RBX: ffffffff82012480 RCX: 0000000000000000

RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000

RBP: 0000000000000000 R08: 000000008e38ce64 R09: 0000000000000000

R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff82012480

R13: ffffffff82012480 R14: 0000000000000000 R15: 0000000000000000

  ? __sched_text_end+0x4/0x4

  default_idle+0x18/0xf0

  do_idle+0x150/0x1d0

  cpu_startup_entry+0x6f/0x80

  start_kernel+0x4c4/0x4e4

  ? set_init_arg+0x55/0x55

  secondary_startup_64+0xa5/0xb0

  print_req_error: I/O error, dev nvme0n1, sector 9240

EXT4-fs error (device nvme0n1): ext4_find_entry:1436:

inode #2: comm cp: reading directory lblock 0



Example 2: Inject default status code with retry

------------------------------------------------



mount /dev/nvme0n1 /mnt

echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times

echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability

echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/status

echo 0 > /sys/kernel/debug/nvme0n1/fault_inject/dont_retry



cp a.file /mnt



Expected Result:



command success without error



Message from dmesg:



FAULT_INJECTION: forcing a failure.

name fault_inject, interval 1, probability 100, space 0, times 1

CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.15.0-rc8+ #4

Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006

Call Trace:

  <IRQ>

  dump_stack+0x5c/0x7d

  should_fail+0x148/0x170

  nvme_should_fail+0x30/0x60 [nvme_core]

  nvme_loop_queue_response+0x84/0x110 [nvme_loop]

  nvmet_req_complete+0x11/0x40 [nvmet]

  nvmet_bio_done+0x28/0x40 [nvmet]

  blk_update_request+0xb0/0x310

  blk_mq_end_request+0x18/0x60

  flush_smp_call_function_queue+0x3d/0xf0

  smp_call_function_single_interrupt+0x2c/0xc0

  call_function_single_interrupt+0xa2/0xb0

  </IRQ>

RIP: 0010:native_safe_halt+0x2/0x10

RSP: 0018:ffffc9000068bec0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04

RAX: ffffffff817a10c0 RBX: ffff88011a3c9680 RCX: 0000000000000000

RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000

RBP: 0000000000000001 R08: 000000008e38c131 R09: 0000000000000000

R10: 0000000000000000 R11: 0000000000000000 R12: ffff88011a3c9680

R13: ffff88011a3c9680 R14: 0000000000000000 R15: 0000000000000000

  ? __sched_text_end+0x4/0x4

  default_idle+0x18/0xf0

  do_idle+0x150/0x1d0

  cpu_startup_entry+0x6f/0x80

  start_secondary+0x187/0x1e0

  secondary_startup_64+0xa5/0xb0