Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit efd78098 authored by android-build-team Robot's avatar android-build-team Robot
Browse files

Snap for 6617762 from 6a8602a8 to rvc-release

Change-Id: I1d3edd178c1609877cac3c30baaee4a39459af4a
parents 2cad8b10 6a8602a8
Loading
Loading
Loading
Loading
+1 −4
Original line number Diff line number Diff line
@@ -198,8 +198,7 @@ bool Modprobe::ParseBlocklistCallback(const std::vector<std::string>& args) {
    auto it = args.begin();
    const std::string& type = *it++;

    // +Legacy
    if ((type != "blocklist") && (type != "blacklist")) {
    if (type != "blocklist") {
        LOG(ERROR) << "non-blocklist line encountered in modules.blocklist";
        return false;
    }
@@ -334,8 +333,6 @@ Modprobe::Modprobe(const std::vector<std::string>& base_paths, const std::string

        auto blocklist_callback = std::bind(&Modprobe::ParseBlocklistCallback, this, _1);
        ParseCfg(base_path + "/modules.blocklist", blocklist_callback);
        // Legacy
        ParseCfg(base_path + "/modules.blacklist", blocklist_callback);
    }

    ParseKernelCmdlineOptions();
+227 −189
Original line number Diff line number Diff line
Android Live-LocK Daemon
========================
<!--
Project: /_project.yaml
Book: /_book.yaml

Introduction
------------
{% include "_versions.html" %}
-->

Android Live-LocK Daemon (llkd) is used to catch kernel deadlocks and mitigate.
<!--
  Copyright 2020 The Android Open Source Project

Code is structured to allow integration into another service as either as part
of the main loop, or spun off as a thread should that be necessary.  A default
standalone implementation is provided by llkd component.
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

The 'C' interface from libllkd component is thus:
      http://www.apache.org/licenses/LICENSE-2.0

    #include "llkd.h"
    bool llkInit(const char* threadname) /* return true if enabled */
    unsigned llkCheckMillseconds(void)   /* ms to sleep for next check */
  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->

If a threadname is provided, a thread will be automatically spawned, otherwise
caller must call llkCheckMilliseconds in its main loop.  Function will return
the period of time before the next expected call to this handler.
# Android Live-LocK Daemon (llkd)

Operations
----------
Android 10 <!-- {{ androidQVersionNumber }} --> includes the Android Live-LocK Daemon
(`llkd`), which is designed to catch and mitigate kernel deadlocks. The `llkd`
component provides a default standalone implementation, but you can
alternatively integrate the `llkd` code into another service, either as part of
the main loop or as a separate thread.

There are two detection scenarios. Persistent D or Z state, and persistent
## Detection scenarios <!-- {:#detection-scenarios} -->

The `llkd` has two detection scenarios: Persistent D or Z state, and persistent
stack signature.

If a thread is in D or Z state with no forward progress for longer than
ro.llk.timeout_ms, or ro.llk.[D|Z].timeout_ms, kill the process or parent
process respectively.  If another scan shows the same process continues to
exist, then have a confirmed live-lock condition and need to panic.  Panic
the kernel in a manner to provide the greatest bugreporting details as to the
condition.  Add a alarm self watchdog should llkd ever get locked up that is
double the expected time to flow through the mainloop.  Sampling is every
ro.llk_sample_ms.

For usedebug releases only, persistent stack signature checking is enabled.
If a thread in any state but Z, has a persistent listed ro.llk.stack kernel
symbol always being reported, even if there is forward scheduling progress, for
longer than ro.llk.timeout_ms, or ro.llk.stack.timeout_ms, then issue a kill
to the process.  If another scan shows the same process continues to exist,
then have a confirmed live-lock condition and need to panic.  There is no
ABA detection since forward scheduling progress is allowed, thus the condition
for the symbols are:

- Check is looking for " __symbol__+0x" or " __symbol__.cfi+0x" in
  /proc/__pid__/stack.
- The __symbol__ should be rare and short lived enough that on a typical
  system the function is seen at most only once in a sample over the timeout
  period of ro.llk.stack.timeout_ms, samples occur every ro.llk.check_ms. This
  can be the only way to prevent a false trigger as there is no ABA protection.
- Persistent continuously when the live lock condition exists.
- Should be just below the function that is calling the lock that could
  contend, because if the lock is below or in the symbol function, the
  symbol will show in all affected processes, not just the one that
  caused the lockup.

Default will not monitor init, or [kthreadd] and all that [kthreadd] spawns.
This reduces the effectiveness of llkd by limiting its coverage.  If there is
value in covering [kthreadd] spawned threads, the requirement will be that
the drivers not remain in a persistent 'D' state, or that they have mechanisms
to recover the thread should it be killed externally (this is good driver
coding hygiene, a common request to add such to publicly reviewed kernel.org
maintained drivers).  For instance use wait_event_interruptible() instead of
wait_event().  The blacklists can be adjusted accordingly if these
conditions are met to cover kernel components.  For the stack symbol checking,
there is an additional process blacklist so that we do not incide sepolicy
violations on services that block ptrace operations.

An accompanying gTest set have been added, and will setup a persistent D or Z
process, with and without forward progress, but not in a live-lock state
because that would require a buggy kernel, or a module or kernel modification
to stimulate.  The test will check that llkd will mitigate first by killing
the appropriate process.  D state is setup by vfork() waiting for exec() in
child process.  Z state is setup by fork() and an un-waited for child process.
Should be noted that both of these conditions should never happen on Android
on purpose, and llkd effectively sweeps up processes that create these
conditions.  If the test can, it will reconfigure llkd to expedite the test
duration by adjusting the ro.llk.* Android properties.  Tests run the D state
with some scheduling progress to ensure that ABA checking prevents false
triggers. If 100% reliable ABA on platform, then ro.llk.killtest can be
set to false; however this will result in some of the unit tests to panic
kernel instead of deal with more graceful kill operation.

Android Properties
------------------

The following are the Android Properties llkd respond to.
*prop*_ms named properties are in milliseconds.
Properties that use comma (*,*) separator for lists, use a leading separator to
preserve default and add or subtract entries with (*optional*) plus (*+*) and
minus (*-*) prefixes respectively.
For these lists, the string "*false*" is synonymous with an *empty* list,
and *blank* or *missing* resorts to the specified *default* value.

#### ro.config.low_ram
device is configured with limited memory.

#### ro.debuggable
device is configured for userdebug or eng build.

#### ro.llk.sysrq_t
default not ro.config.low_ram, or ro.debuggable if property is "eng".
if true do sysrq t (dump all threads).

#### ro.llk.enable
default false, allow live-lock daemon to be enabled.

#### llk.enable
default ro.llk.enable, and evaluated for eng.

#### ro.khungtask.enable
default false, allow [khungtask] daemon to be enabled.

#### khungtask.enable
default ro.khungtask.enable and evaluated for eng.

#### ro.llk.mlockall
default false, enable call to mlockall().

#### ro.khungtask.timeout
default value 12 minutes, [khungtask] maximum timelimit.

#### ro.llk.timeout_ms
default 10 minutes, D or Z maximum timelimit, double this value and it sets
the alarm watchdog for llkd.

#### ro.llk.D.timeout_ms
default ro.llk.timeout_ms, D maximum timelimit.

#### ro.llk.Z.timeout_ms
default ro.llk.timeout_ms, Z maximum timelimit.

#### ro.llk.stack.timeout_ms
default ro.llk.timeout_ms,
checking for persistent stack symbols maximum timelimit.
Only active on userdebug or eng builds.

#### ro.llk.check_ms
default 2 minutes samples of threads for D or Z.

#### ro.llk.stack
default cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable
comma separated list of kernel symbols.
Look for kernel stack symbols that if ever persistently present can
indicate a subsystem is locked up.
Beware, check does not on purpose do forward scheduling ABA except by polling
every ro.llk_check_ms over the period ro.llk.stack.timeout_ms, so stack symbol
should be exceptionally rare and fleeting.
One must be convinced that it is virtually *impossible* for symbol to show up
persistently in all samples of the stack.
Again, looks for a match for either " **symbol**+0x" or " **symbol**.cfi+0x"
in stack expansion.
Only available on userdebug or eng builds, limited privileges due to security
concerns on user builds prevents this checking.

#### ro.llk.blacklist.process
default 0,1,2 (kernel, init and [kthreadd]) plus process names
init,[kthreadd],[khungtaskd],lmkd,llkd,watchdogd,
[watchdogd],[watchdogd/0],...,[watchdogd/***get_nprocs**-1*].
Do not watch these processes.  A process can be comm, cmdline or pid reference.
NB: automated default here can be larger than the current maximum property
size of 92.
NB: false is a very very very unlikely process to want to blacklist.

#### ro.llk.blacklist.parent
default 0,2,adbd&[setsid] (kernel, [kthreadd] and adbd *only for zombie setsid*).
Do not watch processes that have this parent.
An ampersand (*&*) separator is used to specify that the parent is ignored
only in combination with the target child process.
Ampersand was selected because it is never part of a process name,
however a setprop in the shell requires it to be escaped or quoted;
init rc file where this is normally specified does not have this issue.
A parent or target processes can be specified as comm, cmdline or pid reference.

#### ro.llk.blacklist.uid
default *empty* or false, comma separated list of uid numbers or names.
Do not watch processes that match this uid.

#### ro.llk.blacklist.process.stack
default process names init,lmkd.llkd,llkd,keystore,ueventd,apexd,logd.
This subset of processes are not monitored for live lock stack signatures.
Also prevents the sepolicy violation associated with processes that block
ptrace, as these can not be checked anyways.
Only active on userdebug and eng builds.

Architectural Concerns
----------------------

- built-in [khungtask] daemon is too generic and trips on driver code that
  sits around in D state too much.  To switch to S instead makes the task(s)
  killable, so the drivers should be able to resurrect them if needed.
- Properties are limited to 92 characters.
- Create kernel module and associated gTest to actually test panic.
- Create gTest to test out blacklist (ro.llk.blacklist.*properties* generally
  not be inputs).  Could require more test-only interfaces to libllkd.
- Speed up gTest using something else than ro.llk.*properties*, which should
  not be inputs as they should be baked into the product.
### Persistent D or Z state <!-- {:#persistent-d-or-z-state} -->

If a thread is in D (uninterruptible sleep) or Z (zombie) state with no forward
progress for longer than `ro.llk.timeout_ms or ro.llk.[D|Z].timeout_ms`, the
`llkd` kills the process (or parent process). If a subsequent scan shows the
same process continues to exist, the `llkd` confirms a live-lock condition and
panics the kernel in a manner that provides the most detailed bug report for the
condition.

The `llkd` includes a self watchdog that alarms if `llkd` locks up; watchdog is
double the expected time to flow through the mainloop and sampling is every
`ro.llk_sample_ms`.

### Persistent stack signature <!-- {:#persistent-stack-signature} -->

For userdebug releases, the `llkd` can detect kernel live-locks using persistent
stack signature checking. If a thread in any state except Z has a persistent
listed `ro.llk.stack` kernel symbol that is reported for longer than
`ro.llk.timeout_ms` or `ro.llk.stack.timeout_ms`, the `llkd` kills the process
(even if there is forward scheduling progress). If a subsequent scan shows the
same process continues to exist, the `llkd` confirms a live-lock condition and
panics the kernel in a manner that provides the most detailed bug report for the
condition.

Note: Because forward scheduling progress is allowed, the `llkd` does not
perform [ABA detection](https://en.wikipedia.org/wiki/ABA_problem){:.external}.

The `lldk` check persists continuously when the live lock condition exists and
looks for the composed strings `" symbol+0x"` or `" symbol.cfi+0x"` in the
`/proc/pid/stack` file on Linux. The list of symbols is in `ro.llk.stack` and
defaults to the comma-separated list of
"`cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable`".

Symbols should be rare and short-lived enough that on a typical system the
function is seen only once in a sample over the timeout period of
`ro.llk.stack.timeout_ms` (samples occur every `ro.llk.check_ms`). Due to lack
of ABA protection, this is the only way to prevent a false trigger. The symbol
function must appear below the function calling the lock that could contend. If
the lock is below or in the symbol function, the symbol appears in all affected
processes, not just the one that caused the lockup.

## Coverage <!-- {:#coverage} -->

The default implementation of `llkd` does not monitor `init`, `[kthreadd]`, or
`[kthreadd]` spawns. For the `llkd` to cover `[kthreadd]`-spawned threads:

* Drivers must not remain in a persistent D state,

OR

* Drivers must have mechanisms to recover the thread should it be killed
  externally. For example, use `wait_event_interruptible()` instead of
  `wait_event()`.

If one of the above conditions is met, the `llkd` ignorelist can be adjusted to
cover kernel components.  Stack symbol checking involves an additional process
ignore list to prevent sepolicy violations on services that block `ptrace`
operations.

## Android properties <!-- {:#android-properties} -->

The `llkd` responds to several Android properties (listed below).

* Properties named `prop_ms` are in milliseconds.
* Properties that use comma (,) separator for lists use a leading separator to
  preserve the default entry, then add or subtract entries with optional plus
  (+) and minus (-) prefixes respectively. For these lists, the string "false"
  is synonymous with an empty list, and blank or missing entries resort to the
  specified default value.

### ro.config.low_ram <!-- {:#ro-config-low-ram} -->

Device is configured with limited memory.

### ro.debuggable <!-- {:#ro-debuggable} -->

Device is configured for userdebug or eng build.

### ro.llk.sysrq_t <!-- {:#ro-llk-sysrq-t} -->

If property is "eng", the default is not `ro.config.low_ram` or `ro.debuggable`.
If true, dump all threads (`sysrq t`).

### ro.llk.enable <!-- {:#ro-llk-enable} -->

Allow live-lock daemon to be enabled. Default is false.

### llk.enable <!-- {:#llk-enable} -->

Evaluated for eng builds. Default is `ro.llk.enable`.

### ro.khungtask.enable <!-- {:#ro-khungtask-enable} -->

Allow `[khungtask]` daemon to be enabled. Default is false.

### khungtask.enable <!-- {:#khungtask-enable} -->

Evaluated for eng builds. Default is `ro.khungtask.enable`.

### ro.llk.mlockall <!-- {:#ro-llk-mlockall} -->

Enable call to `mlockall()`. Default is false.

### ro.khungtask.timeout <!-- {:#ro-khungtask-timeout} -->

`[khungtask]` maximum time limit. Default is 12 minutes.

### ro.llk.timeout_ms <!-- {:#ro-llk-timeout-ms} -->

D or Z maximum time limit. Default is 10 minutes. Double this value to set the
alarm watchdog for `llkd`.

### ro.llk.D.timeout_ms <!-- {:#ro-llk-D-timeout-ms} -->

D maximum time limit. Default is `ro.llk.timeout_ms`.

### ro.llk.Z.timeout_ms <!-- {:#ro-llk-Z-timeout-ms} -->

Z maximum time limit. Default is `ro.llk.timeout_ms`.

### ro.llk.stack.timeout_ms <!-- {:#ro-llk-stack-timeout-ms} -->

Checks for persistent stack symbols maximum time limit. Default is
`ro.llk.timeout_ms`. **Active only on userdebug or eng builds**.

### ro.llk.check_ms <!-- {:#ro-llk-check-ms} -->

Samples of threads for D or Z. Default is two minutes.

### ro.llk.stack <!-- {:#ro-llk-stack} -->

Checks for kernel stack symbols that if persistently present can indicate a
subsystem is locked up. Default is
`cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable`
comma-separated list of kernel symbols. The check doesn't do forward scheduling
ABA except by polling every `ro.llk_check_ms` over the period
`ro.llk.stack.timeout_ms`, so stack symbols should be exceptionally rare and
fleeting (it is highly unlikely for a symbol to show up persistently in all
samples of the stack). Checks for a match for `" symbol+0x"` or
`" symbol.cfi+0x"` in stack expansion. **Available only on userdebug or eng
builds**; security concerns on user builds result in limited privileges that
prevent this check.

### ro.llk.ignorelist.process <!-- {:#ro-llk-ignorelist-process} -->

The `llkd` does not watch the specified processes. Default is `0,1,2` (`kernel`,
`init`, and `[kthreadd]`) plus process names
`init,[kthreadd],[khungtaskd],lmkd,llkd,watchdogd, [watchdogd],[watchdogd/0],...,[watchdogd/get_nprocs-1]`.
A process can be a `comm`, `cmdline`, or `pid` reference. An automated default
can be larger than the current maximum property size of 92.

Note: `false` is an extremely unlikely process to want to ignore.

### ro.llk.ignorelist.parent <!-- {:#ro-llk-ignorelist-parent} -->

The `llkd` does not watch processes that have the specified parent(s). Default
is `0,2,adbd&[setsid]` (`kernel`, `[kthreadd]`, and `adbd` only for zombie
`setsid`). An ampersand (&) separator specifies that the parent is ignored only
in combination with the target child process. Ampersand was selected because it
is never part of a process name; however, a `setprop` in the shell requires the
ampersand to be escaped or quoted, although the `init rc` file where this is
normally specified does not have this issue. A parent or target process can be a
`comm`, `cmdline`, or `pid` reference.

### ro.llk.ignorelist.uid <!-- {:#ro-llk-ignorelist-uid} -->

The `llkd` does not watch processes that match the specified uid(s).
Comma-separated list of uid numbers or names. Default is empty or false.

### ro.llk.ignorelist.process.stack <!-- {:#ro-llk-ignorelist-process-stack} -->

The `llkd` does not monitor the specified subset of processes for live lock stack
signatures. Default is process names
`init,lmkd.llkd,llkd,keystore,ueventd,apexd,logd`. Prevents the sepolicy
violation associated with processes that block `ptrace` (as these can't be
checked). **Active only on userdebug and eng builds**. For details on build
types, refer to [Building Android](/setup/build/building#choose-a-target).

## Architectural concerns <!-- {:#architectural-concerns} -->

* Properties are limited to 92 characters.  However, this is not limited for
  defaults defined in the `include/llkd.h` file in the sources.
* The built-in `[khungtask]` daemon is too generic and trips on driver code that
  sits around in D state too much. Switching drivers to sleep, or S state,
  would make task(s) killable, and need to be resurrectable by drivers on an
  as-need basis.

## Library interface (optional) <!-- {:#library-interface-optional} -->

You can optionally incorporate the `llkd` into another privileged daemon using
the following C interface from the `libllkd` component:

```
#include "llkd.h"
bool llkInit(const char* threadname) /* return true if enabled */
unsigned llkCheckMillseconds(void)   /* ms to sleep for next check */
```

If a threadname is provided, a thread automatically spawns, otherwise the caller
must call `llkCheckMilliseconds` in its main loop. The function returns the
period of time before the next expected call to this handler.
+28 −28
Original line number Diff line number Diff line
@@ -52,15 +52,15 @@ unsigned llkCheckMilliseconds(void);
#define LLK_CHECK_STACK_PROPERTY        "ro.llk.stack"
#define LLK_CHECK_STACK_DEFAULT         \
    "cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable"
#define LLK_BLACKLIST_PROCESS_PROPERTY "ro.llk.blacklist.process"
#define LLK_BLACKLIST_PROCESS_DEFAULT  \
#define LLK_IGNORELIST_PROCESS_PROPERTY "ro.llk.ignorelist.process"
#define LLK_IGNORELIST_PROCESS_DEFAULT  \
    "0,1,2,init,[kthreadd],[khungtaskd],lmkd,llkd,watchdogd,[watchdogd],[watchdogd/0]"
#define LLK_BLACKLIST_PARENT_PROPERTY  "ro.llk.blacklist.parent"
#define LLK_BLACKLIST_PARENT_DEFAULT   "0,2,[kthreadd],adbd&[setsid]"
#define LLK_BLACKLIST_UID_PROPERTY     "ro.llk.blacklist.uid"
#define LLK_BLACKLIST_UID_DEFAULT      ""
#define LLK_BLACKLIST_STACK_PROPERTY   "ro.llk.blacklist.process.stack"
#define LLK_BLACKLIST_STACK_DEFAULT    "init,lmkd.llkd,llkd,keystore,ueventd,apexd"
#define LLK_IGNORELIST_PARENT_PROPERTY  "ro.llk.ignorelist.parent"
#define LLK_IGNORELIST_PARENT_DEFAULT   "0,2,[kthreadd],adbd&[setsid]"
#define LLK_IGNORELIST_UID_PROPERTY     "ro.llk.ignorelist.uid"
#define LLK_IGNORELIST_UID_DEFAULT      ""
#define LLK_IGNORELIST_STACK_PROPERTY   "ro.llk.ignorelist.process.stack"
#define LLK_IGNORELIST_STACK_DEFAULT    "init,lmkd.llkd,llkd,keystore,ueventd,apexd"
/* clang-format on */

__END_DECLS
+65 −65

File changed.

Preview size limit exceeded, changes collapsed.

+6 −0
Original line number Diff line number Diff line
@@ -517,6 +517,12 @@ on post-fs

    mkdir /metadata/apex 0700 root system
    mkdir /metadata/apex/sessions 0700 root system
    # On some devices we see a weird behaviour in which /metadata/apex doesn't
    # have a correct label. To workaround this bug, explicitly call restorecon
    # on /metadata/apex. For most of the boot sequences /metadata/apex will
    # already have a correct selinux label, meaning that this call will be a
    # no-op.
    restorecon_recursive /metadata/apex

    mkdir /metadata/staged-install 0770 root system
on late-fs