Merge "llkd: Use more inclusive language"

Android Live-LocK Daemon

<!--

========================

Project: /_project.yaml

Book: /_book.yaml

Introduction

{% include "_versions.html" %}

------------

-->

Android Live-LocK Daemon (llkd) is used to catch kernel deadlocks and mitigate.

<!--

  Copyright 2020 The Android Open Source Project

Code is structured to allow integration into another service as either as part

  Licensed under the Apache License, Version 2.0 (the "License");

of the main loop, or spun off as a thread should that be necessary.  A default

  you may not use this file except in compliance with the License.

standalone implementation is provided by llkd component.

  You may obtain a copy of the License at

The 'C' interface from libllkd component is thus:

      http://www.apache.org/licenses/LICENSE-2.0

    #include "llkd.h"

  Unless required by applicable law or agreed to in writing, software

    bool llkInit(const char* threadname) /* return true if enabled */

  distributed under the License is distributed on an "AS IS" BASIS,

    unsigned llkCheckMillseconds(void)   /* ms to sleep for next check */

  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  See the License for the specific language governing permissions and

  limitations under the License.

-->

If a threadname is provided, a thread will be automatically spawned, otherwise

# Android Live-LocK Daemon (llkd)

caller must call llkCheckMilliseconds in its main loop.  Function will return

the period of time before the next expected call to this handler.

Operations

Android 10 <!-- {{ androidQVersionNumber }} --> includes the Android Live-LocK Daemon

----------

(`llkd`), which is designed to catch and mitigate kernel deadlocks. The `llkd`

component provides a default standalone implementation, but you can

alternatively integrate the `llkd` code into another service, either as part of

the main loop or as a separate thread.

There are two detection scenarios. Persistent D or Z state, and persistent

## Detection scenarios <!-- {:#detection-scenarios} -->

The `llkd` has two detection scenarios: Persistent D or Z state, and persistent

stack signature.

stack signature.

If a thread is in D or Z state with no forward progress for longer than

### Persistent D or Z state <!-- {:#persistent-d-or-z-state} -->

ro.llk.timeout_ms, or ro.llk.[D|Z].timeout_ms, kill the process or parent

process respectively.  If another scan shows the same process continues to

If a thread is in D (uninterruptible sleep) or Z (zombie) state with no forward

exist, then have a confirmed live-lock condition and need to panic.  Panic

progress for longer than `ro.llk.timeout_ms or ro.llk.[D|Z].timeout_ms`, the

the kernel in a manner to provide the greatest bugreporting details as to the

`llkd` kills the process (or parent process). If a subsequent scan shows the

condition.  Add a alarm self watchdog should llkd ever get locked up that is

same process continues to exist, the `llkd` confirms a live-lock condition and

double the expected time to flow through the mainloop.  Sampling is every

panics the kernel in a manner that provides the most detailed bug report for the

ro.llk_sample_ms.

condition.

For usedebug releases only, persistent stack signature checking is enabled.

The `llkd` includes a self watchdog that alarms if `llkd` locks up; watchdog is

If a thread in any state but Z, has a persistent listed ro.llk.stack kernel

double the expected time to flow through the mainloop and sampling is every

symbol always being reported, even if there is forward scheduling progress, for

`ro.llk_sample_ms`.

longer than ro.llk.timeout_ms, or ro.llk.stack.timeout_ms, then issue a kill

to the process.  If another scan shows the same process continues to exist,

### Persistent stack signature <!-- {:#persistent-stack-signature} -->

then have a confirmed live-lock condition and need to panic.  There is no

ABA detection since forward scheduling progress is allowed, thus the condition

For userdebug releases, the `llkd` can detect kernel live-locks using persistent

for the symbols are:

stack signature checking. If a thread in any state except Z has a persistent

listed `ro.llk.stack` kernel symbol that is reported for longer than

- Check is looking for " __symbol__+0x" or " __symbol__.cfi+0x" in

`ro.llk.timeout_ms` or `ro.llk.stack.timeout_ms`, the `llkd` kills the process

  /proc/__pid__/stack.

(even if there is forward scheduling progress). If a subsequent scan shows the

- The __symbol__ should be rare and short lived enough that on a typical

same process continues to exist, the `llkd` confirms a live-lock condition and

  system the function is seen at most only once in a sample over the timeout

panics the kernel in a manner that provides the most detailed bug report for the

  period of ro.llk.stack.timeout_ms, samples occur every ro.llk.check_ms. This

condition.

  can be the only way to prevent a false trigger as there is no ABA protection.

- Persistent continuously when the live lock condition exists.

Note: Because forward scheduling progress is allowed, the `llkd` does not

- Should be just below the function that is calling the lock that could

perform [ABA detection](https://en.wikipedia.org/wiki/ABA_problem){:.external}.

  contend, because if the lock is below or in the symbol function, the

  symbol will show in all affected processes, not just the one that

The `lldk` check persists continuously when the live lock condition exists and

  caused the lockup.

looks for the composed strings `" symbol+0x"` or `" symbol.cfi+0x"` in the

`/proc/pid/stack` file on Linux. The list of symbols is in `ro.llk.stack` and

Default will not monitor init, or [kthreadd] and all that [kthreadd] spawns.

defaults to the comma-separated list of

This reduces the effectiveness of llkd by limiting its coverage.  If there is

"`cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable`".

value in covering [kthreadd] spawned threads, the requirement will be that

the drivers not remain in a persistent 'D' state, or that they have mechanisms

Symbols should be rare and short-lived enough that on a typical system the

to recover the thread should it be killed externally (this is good driver

function is seen only once in a sample over the timeout period of

coding hygiene, a common request to add such to publicly reviewed kernel.org

`ro.llk.stack.timeout_ms` (samples occur every `ro.llk.check_ms`). Due to lack

maintained drivers).  For instance use wait_event_interruptible() instead of

of ABA protection, this is the only way to prevent a false trigger. The symbol

wait_event().  The blacklists can be adjusted accordingly if these

function must appear below the function calling the lock that could contend. If

conditions are met to cover kernel components.  For the stack symbol checking,

the lock is below or in the symbol function, the symbol appears in all affected

there is an additional process blacklist so that we do not incide sepolicy

processes, not just the one that caused the lockup.

violations on services that block ptrace operations.

## Coverage <!-- {:#coverage} -->

An accompanying gTest set have been added, and will setup a persistent D or Z

process, with and without forward progress, but not in a live-lock state

The default implementation of `llkd` does not monitor `init`, `[kthreadd]`, or

because that would require a buggy kernel, or a module or kernel modification

`[kthreadd]` spawns. For the `llkd` to cover `[kthreadd]`-spawned threads:

to stimulate.  The test will check that llkd will mitigate first by killing

the appropriate process.  D state is setup by vfork() waiting for exec() in

* Drivers must not remain in a persistent D state,

child process.  Z state is setup by fork() and an un-waited for child process.

Should be noted that both of these conditions should never happen on Android

OR

on purpose, and llkd effectively sweeps up processes that create these

conditions.  If the test can, it will reconfigure llkd to expedite the test

* Drivers must have mechanisms to recover the thread should it be killed

duration by adjusting the ro.llk.* Android properties.  Tests run the D state

  externally. For example, use `wait_event_interruptible()` instead of

with some scheduling progress to ensure that ABA checking prevents false

  `wait_event()`.

triggers. If 100% reliable ABA on platform, then ro.llk.killtest can be

set to false; however this will result in some of the unit tests to panic

If one of the above conditions is met, the `llkd` ignorelist can be adjusted to

kernel instead of deal with more graceful kill operation.

cover kernel components.  Stack symbol checking involves an additional process

ignore list to prevent sepolicy violations on services that block `ptrace`

Android Properties

operations.

------------------

## Android properties <!-- {:#android-properties} -->

The following are the Android Properties llkd respond to.

*prop*_ms named properties are in milliseconds.

The `llkd` responds to several Android properties (listed below).

Properties that use comma (*,*) separator for lists, use a leading separator to

preserve default and add or subtract entries with (*optional*) plus (*+*) and

* Properties named `prop_ms` are in milliseconds.

minus (*-*) prefixes respectively.

* Properties that use comma (,) separator for lists use a leading separator to

For these lists, the string "*false*" is synonymous with an *empty* list,

  preserve the default entry, then add or subtract entries with optional plus

and *blank* or *missing* resorts to the specified *default* value.

  (+) and minus (-) prefixes respectively. For these lists, the string "false"

  is synonymous with an empty list, and blank or missing entries resort to the

#### ro.config.low_ram

  specified default value.

device is configured with limited memory.

### ro.config.low_ram <!-- {:#ro-config-low-ram} -->

#### ro.debuggable

device is configured for userdebug or eng build.

Device is configured with limited memory.

#### ro.llk.sysrq_t

### ro.debuggable <!-- {:#ro-debuggable} -->

default not ro.config.low_ram, or ro.debuggable if property is "eng".

if true do sysrq t (dump all threads).

Device is configured for userdebug or eng build.

#### ro.llk.enable

### ro.llk.sysrq_t <!-- {:#ro-llk-sysrq-t} -->

default false, allow live-lock daemon to be enabled.

If property is "eng", the default is not `ro.config.low_ram` or `ro.debuggable`.

#### llk.enable

If true, dump all threads (`sysrq t`).

default ro.llk.enable, and evaluated for eng.

### ro.llk.enable <!-- {:#ro-llk-enable} -->

#### ro.khungtask.enable

default false, allow [khungtask] daemon to be enabled.

Allow live-lock daemon to be enabled. Default is false.

#### khungtask.enable

### llk.enable <!-- {:#llk-enable} -->

default ro.khungtask.enable and evaluated for eng.

Evaluated for eng builds. Default is `ro.llk.enable`.

#### ro.llk.mlockall

default false, enable call to mlockall().

### ro.khungtask.enable <!-- {:#ro-khungtask-enable} -->

#### ro.khungtask.timeout

Allow `[khungtask]` daemon to be enabled. Default is false.

default value 12 minutes, [khungtask] maximum timelimit.

### khungtask.enable <!-- {:#khungtask-enable} -->

#### ro.llk.timeout_ms

default 10 minutes, D or Z maximum timelimit, double this value and it sets

Evaluated for eng builds. Default is `ro.khungtask.enable`.

the alarm watchdog for llkd.

### ro.llk.mlockall <!-- {:#ro-llk-mlockall} -->

#### ro.llk.D.timeout_ms

default ro.llk.timeout_ms, D maximum timelimit.

Enable call to `mlockall()`. Default is false.

#### ro.llk.Z.timeout_ms

### ro.khungtask.timeout <!-- {:#ro-khungtask-timeout} -->

default ro.llk.timeout_ms, Z maximum timelimit.

`[khungtask]` maximum time limit. Default is 12 minutes.

#### ro.llk.stack.timeout_ms

default ro.llk.timeout_ms,

### ro.llk.timeout_ms <!-- {:#ro-llk-timeout-ms} -->

checking for persistent stack symbols maximum timelimit.

Only active on userdebug or eng builds.

D or Z maximum time limit. Default is 10 minutes. Double this value to set the

alarm watchdog for `llkd`.

#### ro.llk.check_ms

default 2 minutes samples of threads for D or Z.

### ro.llk.D.timeout_ms <!-- {:#ro-llk-D-timeout-ms} -->

#### ro.llk.stack

D maximum time limit. Default is `ro.llk.timeout_ms`.

default cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable

comma separated list of kernel symbols.

### ro.llk.Z.timeout_ms <!-- {:#ro-llk-Z-timeout-ms} -->

Look for kernel stack symbols that if ever persistently present can

indicate a subsystem is locked up.

Z maximum time limit. Default is `ro.llk.timeout_ms`.

Beware, check does not on purpose do forward scheduling ABA except by polling

every ro.llk_check_ms over the period ro.llk.stack.timeout_ms, so stack symbol

### ro.llk.stack.timeout_ms <!-- {:#ro-llk-stack-timeout-ms} -->

should be exceptionally rare and fleeting.

One must be convinced that it is virtually *impossible* for symbol to show up

Checks for persistent stack symbols maximum time limit. Default is

persistently in all samples of the stack.

`ro.llk.timeout_ms`. **Active only on userdebug or eng builds**.

Again, looks for a match for either " **symbol**+0x" or " **symbol**.cfi+0x"

in stack expansion.

### ro.llk.check_ms <!-- {:#ro-llk-check-ms} -->

Only available on userdebug or eng builds, limited privileges due to security

concerns on user builds prevents this checking.

Samples of threads for D or Z. Default is two minutes.

#### ro.llk.blacklist.process

### ro.llk.stack <!-- {:#ro-llk-stack} -->

default 0,1,2 (kernel, init and [kthreadd]) plus process names

init,[kthreadd],[khungtaskd],lmkd,llkd,watchdogd,

Checks for kernel stack symbols that if persistently present can indicate a

[watchdogd],[watchdogd/0],...,[watchdogd/***get_nprocs**-1*].

subsystem is locked up. Default is

Do not watch these processes.  A process can be comm, cmdline or pid reference.

`cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable`

NB: automated default here can be larger than the current maximum property

comma-separated list of kernel symbols. The check doesn't do forward scheduling

size of 92.

ABA except by polling every `ro.llk_check_ms` over the period

NB: false is a very very very unlikely process to want to blacklist.

`ro.llk.stack.timeout_ms`, so stack symbols should be exceptionally rare and

fleeting (it is highly unlikely for a symbol to show up persistently in all

#### ro.llk.blacklist.parent

samples of the stack). Checks for a match for `" symbol+0x"` or

default 0,2,adbd&[setsid] (kernel, [kthreadd] and adbd *only for zombie setsid*).

`" symbol.cfi+0x"` in stack expansion. **Available only on userdebug or eng

Do not watch processes that have this parent.

builds**; security concerns on user builds result in limited privileges that

An ampersand (*&*) separator is used to specify that the parent is ignored

prevent this check.

only in combination with the target child process.

Ampersand was selected because it is never part of a process name,

### ro.llk.ignorelist.process <!-- {:#ro-llk-ignorelist-process} -->

however a setprop in the shell requires it to be escaped or quoted;

init rc file where this is normally specified does not have this issue.

The `llkd` does not watch the specified processes. Default is `0,1,2` (`kernel`,

A parent or target processes can be specified as comm, cmdline or pid reference.

`init`, and `[kthreadd]`) plus process names

`init,[kthreadd],[khungtaskd],lmkd,llkd,watchdogd, [watchdogd],[watchdogd/0],...,[watchdogd/get_nprocs-1]`.

#### ro.llk.blacklist.uid

A process can be a `comm`, `cmdline`, or `pid` reference. An automated default

default *empty* or false, comma separated list of uid numbers or names.

can be larger than the current maximum property size of 92.

Do not watch processes that match this uid.

Note: `false` is an extremely unlikely process to want to ignore.

#### ro.llk.blacklist.process.stack

default process names init,lmkd.llkd,llkd,keystore,ueventd,apexd,logd.

### ro.llk.ignorelist.parent <!-- {:#ro-llk-ignorelist-parent} -->

This subset of processes are not monitored for live lock stack signatures.

Also prevents the sepolicy violation associated with processes that block

The `llkd` does not watch processes that have the specified parent(s). Default

ptrace, as these can not be checked anyways.

is `0,2,adbd&[setsid]` (`kernel`, `[kthreadd]`, and `adbd` only for zombie

Only active on userdebug and eng builds.

`setsid`). An ampersand (&) separator specifies that the parent is ignored only

in combination with the target child process. Ampersand was selected because it

Architectural Concerns

is never part of a process name; however, a `setprop` in the shell requires the

----------------------

ampersand to be escaped or quoted, although the `init rc` file where this is

normally specified does not have this issue. A parent or target process can be a

- built-in [khungtask] daemon is too generic and trips on driver code that

`comm`, `cmdline`, or `pid` reference.

  sits around in D state too much.  To switch to S instead makes the task(s)

  killable, so the drivers should be able to resurrect them if needed.

### ro.llk.ignorelist.uid <!-- {:#ro-llk-ignorelist-uid} -->

- Properties are limited to 92 characters.

- Create kernel module and associated gTest to actually test panic.

The `llkd` does not watch processes that match the specified uid(s).

- Create gTest to test out blacklist (ro.llk.blacklist.*properties* generally

Comma-separated list of uid numbers or names. Default is empty or false.

  not be inputs).  Could require more test-only interfaces to libllkd.

- Speed up gTest using something else than ro.llk.*properties*, which should

### ro.llk.ignorelist.process.stack <!-- {:#ro-llk-ignorelist-process-stack} -->

  not be inputs as they should be baked into the product.

The `llkd` does not monitor the specified subset of processes for live lock stack

signatures. Default is process names

`init,lmkd.llkd,llkd,keystore,ueventd,apexd,logd`. Prevents the sepolicy

violation associated with processes that block `ptrace` (as these can't be

checked). **Active only on userdebug and eng builds**. For details on build

types, refer to [Building Android](/setup/build/building#choose-a-target).

## Architectural concerns <!-- {:#architectural-concerns} -->

* Properties are limited to 92 characters.  However, this is not limited for

  defaults defined in the `include/llkd.h` file in the sources.

* The built-in `[khungtask]` daemon is too generic and trips on driver code that

  sits around in D state too much. Switching drivers to sleep, or S state,

  would make task(s) killable, and need to be resurrectable by drivers on an

  as-need basis.

## Library interface (optional) <!-- {:#library-interface-optional} -->

You can optionally incorporate the `llkd` into another privileged daemon using

the following C interface from the `libllkd` component:

```

#include "llkd.h"

bool llkInit(const char* threadname) /* return true if enabled */

unsigned llkCheckMillseconds(void)   /* ms to sleep for next check */

```

If a threadname is provided, a thread automatically spawns, otherwise the caller

must call `llkCheckMilliseconds` in its main loop. The function returns the

period of time before the next expected call to this handler.

#define LLK_CHECK_STACK_PROPERTY        "ro.llk.stack"

#define LLK_CHECK_STACK_PROPERTY        "ro.llk.stack"

#define LLK_CHECK_STACK_DEFAULT         \

#define LLK_CHECK_STACK_DEFAULT         \

    "cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable"

    "cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable"

#define LLK_BLACKLIST_PROCESS_PROPERTY "ro.llk.blacklist.process"

#define LLK_IGNORELIST_PROCESS_PROPERTY "ro.llk.ignorelist.process"

#define LLK_BLACKLIST_PROCESS_DEFAULT  \

#define LLK_IGNORELIST_PROCESS_DEFAULT  \

    "0,1,2,init,[kthreadd],[khungtaskd],lmkd,llkd,watchdogd,[watchdogd],[watchdogd/0]"

    "0,1,2,init,[kthreadd],[khungtaskd],lmkd,llkd,watchdogd,[watchdogd],[watchdogd/0]"

#define LLK_BLACKLIST_PARENT_PROPERTY  "ro.llk.blacklist.parent"

#define LLK_IGNORELIST_PARENT_PROPERTY  "ro.llk.ignorelist.parent"

#define LLK_BLACKLIST_PARENT_DEFAULT   "0,2,[kthreadd],adbd&[setsid]"

#define LLK_IGNORELIST_PARENT_DEFAULT   "0,2,[kthreadd],adbd&[setsid]"

#define LLK_BLACKLIST_UID_PROPERTY     "ro.llk.blacklist.uid"

#define LLK_IGNORELIST_UID_PROPERTY     "ro.llk.ignorelist.uid"

#define LLK_BLACKLIST_UID_DEFAULT      ""

#define LLK_IGNORELIST_UID_DEFAULT      ""

#define LLK_BLACKLIST_STACK_PROPERTY   "ro.llk.blacklist.process.stack"

#define LLK_IGNORELIST_STACK_PROPERTY   "ro.llk.ignorelist.process.stack"

#define LLK_BLACKLIST_STACK_DEFAULT    "init,lmkd.llkd,llkd,keystore,ueventd,apexd"

#define LLK_IGNORELIST_STACK_DEFAULT    "init,lmkd.llkd,llkd,keystore,ueventd,apexd"

/* clang-format on */

/* clang-format on */

__END_DECLS

__END_DECLS