Merge "llkd: add live-lock daemon" am: 656bc246

#define AID_INCIDENTD 1067       /* incidentd daemon */

#define AID_INCIDENTD 1067       /* incidentd daemon */

#define AID_SECURE_ELEMENT 1068  /* secure element subsystem */

#define AID_SECURE_ELEMENT 1068  /* secure element subsystem */

#define AID_LMKD 1069            /* low memory killer daemon */

#define AID_LMKD 1069            /* low memory killer daemon */

#define AID_LLKD 1070            /* live lock daemon */

/* Changes to this file must be made in AOSP, *not* in internal branches. */

/* Changes to this file must be made in AOSP, *not* in internal branches. */

#define AID_SHELL 2000 /* adb and debug shell user */

#define AID_SHELL 2000 /* adb and debug shell user */

cc_library_headers {

    name: "llkd_headers",

    export_include_dirs: ["include"],

}

cc_library_static {

    name: "libllkd",

    srcs: [

        "libllkd.cpp",

    ],

    shared_libs: [

        "libbase",

        "libcutils",

        "liblog",

    ],

    export_include_dirs: ["include"],

    cflags: ["-Werror"],

}

cc_binary {

    name: "llkd",

    srcs: [

        "llkd.cpp",

    ],

    shared_libs: [

        "libbase",

        "libcutils",

        "liblog",

    ],

    static_libs: [

        "libllkd",

    ],

    cflags: ["-Werror"],

    init_rc: ["llkd.rc"],

}

salyzyn@google.com

surenb@google.com

Android Live-LocK Daemon

========================

Introduction

------------

Android Live-LocK Daemon (llkd) is used to catch kernel deadlocks and mitigate.

Code is structured to allow integration into another service as either as part

of the main loop, or spun off as a thread should that be necessary.  A default

standalone implementation is provided by llkd component.

The 'C' interface from libllkd component is thus:

    #include "llkd.h"

    bool llkInit(const char* threadname) /* return true if enabled */

    unsigned llkCheckMillseconds(void)   /* ms to sleep for next check */

If a threadname is provided, a thread will be automatically spawned, otherwise

caller must call llkCheckMilliseconds in its main loop.  Function will return

the period of time before the next expected call to this handler.

Operations

----------

If a thread is in D or Z state with no forward progress for longer than

ro.llk.timeout_ms, or ro.llk.[D|Z].timeout_ms, kill the process or parent

process respectively.  If another scan shows the same process continues to

exist, then have a confirmed live-lock condition and need to panic.  Panic

the kernel in a manner to provide the greatest bugreporting details as to the

condition.  Add a alarm self watchdog should llkd ever get locked up that is

double the expected time to flow through the mainloop.  Sampling is every

ro.llk_sample_ms.

Default will not monitor init, or [kthreadd] and all that [kthreadd] spawns.

This reduces the effectiveness of llkd by limiting its coverage.  If there is

value in covering [kthreadd] spawned threads, the requirement will be that

the drivers not remain in a persistent 'D' state, or that they have mechanisms

to recover the thread should it be killed externally (this is good driver

coding hygiene, a common request to add such to publicly reviewed kernel.org

maintained drivers).  For instance use wait_event_interruptible() instead of

wait_event().  The blacklists can be adjusted accordingly if these

conditions are met to cover kernel components.

An accompanying gTest set have been added, and will setup a persistent D or Z

process, with and without forward progress, but not in a live-lock state

because that would require a buggy kernel, or a module or kernel modification

to stimulate.  The test will check that llkd will mitigate first by killing

the appropriate process.  D state is setup by vfork() waiting for exec() in

child process.  Z state is setup by fork() and an un-waited for child process.

Should be noted that both of these conditions should never happen on Android

on purpose, and llkd effectively sweeps up processes that create these

conditions.  If the test can, it will reconfigure llkd to expedite the test

duration by adjusting the ro.llk.* Android properties.  Tests run the D state

with some scheduling progress to ensure that ABA checking prevents false

triggers.

Android Properties

------------------

Android Properties llkd respond to (<prop>_ms parms are in milliseconds):

#### ro.config.low_ram

default false, if true do not sysrq t (dump all threads).

#### ro.llk.enable

default false, allow live-lock daemon to be enabled.

#### ro.khungtask.enable

default false, allow [khungtask] daemon to be enabled.

#### ro.llk.mlockall

default false, enable call to mlockall().

#### ro.khungtask.timeout

default value 12 minutes, [khungtask] maximum timelimit.

#### ro.llk.timeout_ms

default 10 minutes, D or Z maximum timelimit, double this value and it sets

the alarm watchdog for llkd.

#### ro.llk.D.timeout_ms

default ro.llk.timeout_ms, D maximum timelimit.

#### ro.llk.Z.timeout_ms

default ro.llk.timeout_ms, Z maximum timelimit.

#### ro.llk.check_ms

default 2 minutes samples of threads for D or Z.

#### ro.llk.blacklist.process

default 0,1,2 (kernel, init and [kthreadd]) plus process names

init,[kthreadd],[khungtaskd],lmkd,lmkd.llkd,llkd,watchdogd,

[watchdogd],[watchdogd/0],...,[watchdogd/<get_nprocs-1>].

#### ro.llk.blacklist.parent

default 0,2 (kernel and [kthreadd]).

#### ro.llk.blacklist.uid

default <empty>, comma separated list of uid numbers or names.

Architectural Concerns

----------------------

- Figure out how to communicate the kernel panic better to bootstat canonical

  boot reason determination.  This may require an alteration to bootstat, or

  some logging from llkd.  Would like to see boot reason to be

  watchdog,livelock as a minimum requirement.  Or more specifically would want

  watchdog,livelock,device or watchdog,livelock,zombie be reported.

  Currently reports panic,sysrq (user requested panic) or panic depending on

  system support of pstore.

- Create kernel module and associated gTest to actually test panic.

- Create gTest to test out blacklist (ro.llk.blacklist.<properties> generally

  not be inputs).  Could require more test-only interfaces to libllkd.

- Speed up gTest using something else than ro.llk.<properties>, which should

  not be inputs.

/*

 * Copyright (C) 2018 The Android Open Source Project

 *

 * Licensed under the Apache License, Version 2.0 (the "License");

 * you may not use this file except in compliance with the License.

 * You may obtain a copy of the License at

 *

 *      http://www.apache.org/licenses/LICENSE-2.0

 *

 * Unless required by applicable law or agreed to in writing, software

 * distributed under the License is distributed on an "AS IS" BASIS,

 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

 * See the License for the specific language governing permissions and

 * limitations under the License.

 */

#ifndef _LLKD_H_

#define _LLKD_H_

#ifndef LOG_TAG

#define LOG_TAG "livelock"

#endif

#include <stdbool.h>

#include <sys/cdefs.h>

__BEGIN_DECLS

bool llkInit(const char* threadname); /* threadname NULL, not spawned */

unsigned llkCheckMilliseconds(void);

/* clang-format off */

#define LLK_ENABLE_PROPERTY            "ro.llk.enable"

#define LLK_ENABLE_DEFAULT             false

#define KHT_ENABLE_PROPERTY            "ro.khungtask.enable"

#define LLK_MLOCKALL_PROPERTY          "ro.llk.mlockall"

#define LLK_MLOCKALL_DEFAULT           true

#define LLK_TIMEOUT_MS_PROPERTY        "ro.llk.timeout_ms"

#define KHT_TIMEOUT_PROPERTY           "ro.khungtask.timeout"

#define LLK_D_TIMEOUT_MS_PROPERTY      "ro.llk.D.timeout_ms"

#define LLK_Z_TIMEOUT_MS_PROPERTY      "ro.llk.Z.timeout_ms"

#define LLK_CHECK_MS_PROPERTY          "ro.llk.check_ms"

/* LLK_CHECK_MS_DEFAULT = actual timeout_ms / LLK_CHECKS_PER_TIMEOUT_DEFAULT */

#define LLK_CHECKS_PER_TIMEOUT_DEFAULT 5

#define LLK_BLACKLIST_PROCESS_PROPERTY "ro.llk.blacklist.process"

#define LLK_BLACKLIST_PROCESS_DEFAULT  \

    "0,1,2,init,[kthreadd],[khungtaskd],lmkd,lmkd.llkd,llkd,watchdogd,[watchdogd],[watchdogd/0]"

#define LLK_BLACKLIST_PARENT_PROPERTY  "ro.llk.blacklist.parent"

#define LLK_BLACKLIST_PARENT_DEFAULT   "0,2,[kthreadd]"

#define LLK_BLACKLIST_UID_PROPERTY     "ro.llk.blacklist.uid"

#define LLK_BLACKLIST_UID_DEFAULT      ""

/* clang-format on */

__END_DECLS

#ifdef __cplusplus

extern "C++" { /* In case this included wrapped with __BEGIN_DECLS */

#include <chrono>

__BEGIN_DECLS

/* C++ code allowed to not specify threadname argument for this C linkage */

bool llkInit(const char* threadname = nullptr);

__END_DECLS

std::chrono::milliseconds llkCheck(bool checkRunning = false);

/* clang-format off */

#define LLK_TIMEOUT_MS_DEFAULT  std::chrono::duration_cast<milliseconds>(std::chrono::minutes(10))

#define LLK_TIMEOUT_MS_MINIMUM  std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::seconds(10))

#define LLK_CHECK_MS_MINIMUM    std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::seconds(1))

/* clang-format on */

} /* extern "C++" */

#endif /* __cplusplus */

#endif /* _LLKD_H_ */