Merge "llkd: Add stack symbol checking" (aa31cfcd) · Commits · e / os / android_system_core

llkd/Android.bp

+11 −0

Original line number	Diff line number	Diff line
		@@ -20,6 +20,12 @@ cc_library_static {
		export_include_dirs: ["include"],

		cflags: ["-Werror"],

		product_variables: {
		debuggable: {
		cppflags: ["-D__PTRACE_ENABLED__"],
		},
		},
		}

		cc_binary {
		@@ -39,4 +45,9 @@ cc_binary {
		cflags: ["-Werror"],

		init_rc: ["llkd.rc"],
		product_variables: {
		debuggable: {
		init_rc: ["llkd-debuggable.rc"],
		},
		},
		}

llkd/README.md

+54 −4

Original line number	Diff line number	Diff line
		@@ -23,6 +23,9 @@ the period of time before the next expected call to this handler.
		Operations
		----------

		There are two detection scenarios. Persistent D or Z state, and persistent
		stack signature.

		If a thread is in D or Z state with no forward progress for longer than
		ro.llk.timeout_ms, or ro.llk.[D\|Z].timeout_ms, kill the process or parent
		process respectively. If another scan shows the same process continues to
		@@ -32,6 +35,26 @@ condition. Add a alarm self watchdog should llkd ever get locked up that is
		double the expected time to flow through the mainloop. Sampling is every
		ro.llk_sample_ms.

		For usedebug releases only, persistent stack signature checking is enabled.
		If a thread in any state but Z, has a persistent listed ro.llk.stack kernel
		symbol always being reported, even if there is forward scheduling progress, for
		longer than ro.llk.timeout_ms, or ro.llk.stack.timeout_ms, then issue a kill
		to the process. If another scan shows the same process continues to exist,
		then have a confirmed live-lock condition and need to panic. There is no
		ABA detection since forward scheduling progress is allowed, thus the condition
		for the symbols are:

		- Check is looking for " " + __symbol__+ "0x" in /proc/<pid>/stack.
		- The __symbol__ should be rare and short lived enough that on a typical
		system the function is seen at most only once in a sample over the timeout
		period of ro.llk.stack.timeout_ms, samples occur every ro.llk.check_ms. This
		can be the only way to prevent a false trigger as there is no ABA protection.
		- Persistent continuously when the live lock condition exists.
		- Should be just below the function that is calling the lock that could
		contend, because if the lock is below or in the symbol function, the
		symbol will show in all affected processes, not just the one that
		caused the lockup.

		Default will not monitor init, or [kthreadd] and all that [kthreadd] spawns.
		This reduces the effectiveness of llkd by limiting its coverage. If there is
		value in covering [kthreadd] spawned threads, the requirement will be that
		@@ -40,7 +63,9 @@ to recover the thread should it be killed externally (this is good driver
		coding hygiene, a common request to add such to publicly reviewed kernel.org
		maintained drivers). For instance use wait_event_interruptible() instead of
		wait_event(). The blacklists can be adjusted accordingly if these
		conditions are met to cover kernel components.
		conditions are met to cover kernel components. For the stack symbol checking,
		there is an additional process blacklist so that we do not incide sepolicy
		violations on services that block ptrace operations.

		An accompanying gTest set have been added, and will setup a persistent D or Z
		process, with and without forward progress, but not in a live-lock state
		@@ -93,14 +118,31 @@ default ro.llk.timeout_ms, D maximum timelimit.
		#### ro.llk.Z.timeout_ms
		default ro.llk.timeout_ms, Z maximum timelimit.

		#### ro.llk.stack.timeout_ms
		default ro.llk.timeout_ms,
		checking for persistent stack symbols maximum timelimit.
		Only active on userdebug and eng builds.

		#### ro.llk.check_ms
		default 2 minutes samples of threads for D or Z.

		#### ro.llk.stack
		default empty or false, comma separated list of kernel symbols.
		The string "false" is the equivalent to an empty list.
		Look for kernel stack symbols that if ever persistently present can
		indicate a subsystem is locked up.
		Beware, check does not on purpose do forward scheduling ABA except by polling
		every ro.llk_check_ms over the period ro.llk.stack.timeout_ms, so stack symbol
		should be exceptionally rare and fleeting.
		One must be convinced that it is virtually impossible for symbol to show up
		persistently in all samples of the stack.
		Only active on userdebug and eng builds.

		#### ro.llk.blacklist.process
		default 0,1,2 (kernel, init and [kthreadd]) plus process names
		init,[kthreadd],[khungtaskd],lmkd,lmkd.llkd,llkd,watchdogd,
		[watchdogd],[watchdogd/0],...,[watchdogd/*get_nprocs-1*].
		The string false is the equivalent to an empty list.
		The string "false" is the equivalent to an empty list.
		Do not watch these processes. A process can be comm, cmdline or pid reference.
		NB: automated default here can be larger than the current maximum property
		size of 92.
		@@ -108,15 +150,23 @@ NB: false is a very very very unlikely process to want to blacklist.

		#### ro.llk.blacklist.parent
		default 0,2 (kernel and [kthreadd]).
		The string false is the equivalent to an empty list.
		The string "false" is the equivalent to an empty list.
		Do not watch processes that have this parent.
		A parent process can be comm, cmdline or pid reference.

		#### ro.llk.blacklist.uid
		default empty or false, comma separated list of uid numbers or names.
		The string false is the equivalent to an empty list.
		The string "false" is the equivalent to an empty list.
		Do not watch processes that match this uid.

		#### ro.llk.blacklist.process.stack
		default process names init,lmkd,lmkd.llkd,llkd,keystore,logd.
		The string "false" is the equivalent to an empty list.
		This subset of processes are not monitored for live lock stack signatures.
		Also prevents the sepolicy violation associated with processes that block
		ptrace, as these can not be checked anyways.
		Only active on userdebug and eng builds.

		Architectural Concerns
		----------------------

llkd/include/llkd.h

+5 −0

Original line number	Diff line number	Diff line
		@@ -43,9 +43,12 @@ unsigned llkCheckMilliseconds(void);
		#define KHT_TIMEOUT_PROPERTY "ro.khungtask.timeout"
		#define LLK_D_TIMEOUT_MS_PROPERTY "ro.llk.D.timeout_ms"
		#define LLK_Z_TIMEOUT_MS_PROPERTY "ro.llk.Z.timeout_ms"
		#define LLK_STACK_TIMEOUT_MS_PROPERTY "ro.llk.stack.timeout_ms"
		#define LLK_CHECK_MS_PROPERTY "ro.llk.check_ms"
		/* LLK_CHECK_MS_DEFAULT = actual timeout_ms / LLK_CHECKS_PER_TIMEOUT_DEFAULT */
		#define LLK_CHECKS_PER_TIMEOUT_DEFAULT 5
		#define LLK_CHECK_STACK_PROPERTY "ro.llk.stack"
		#define LLK_CHECK_STACK_DEFAULT ""
		#define LLK_BLACKLIST_PROCESS_PROPERTY "ro.llk.blacklist.process"
		#define LLK_BLACKLIST_PROCESS_DEFAULT \
		"0,1,2,init,[kthreadd],[khungtaskd],lmkd,lmkd.llkd,llkd,watchdogd,[watchdogd],[watchdogd/0]"
		@@ -53,6 +56,8 @@ unsigned llkCheckMilliseconds(void);
		#define LLK_BLACKLIST_PARENT_DEFAULT "0,2,[kthreadd]"
		#define LLK_BLACKLIST_UID_PROPERTY "ro.llk.blacklist.uid"
		#define LLK_BLACKLIST_UID_DEFAULT ""
		#define LLK_BLACKLIST_STACK_PROPERTY "ro.llk.blacklist.process.stack"
		#define LLK_BLACKLIST_STACK_DEFAULT "init,lmkd.llkd,llkd,keystore,/system/bin/keystore"
		/* clang-format on */

		__END_DECLS

llkd/libllkd.cpp

+129 −12

Original line number	Diff line number	Diff line
		@@ -73,7 +73,14 @@ bool llkRunning = false; // thread is running
		bool llkMlockall = LLK_MLOCKALL_DEFAULT; // run mlocked
		bool llkTestWithKill = LLK_KILLTEST_DEFAULT; // issue test kills
		milliseconds llkTimeoutMs = LLK_TIMEOUT_MS_DEFAULT; // default timeout
		enum { llkStateD, llkStateZ, llkNumStates }; // state indexes
		enum { // enum of state indexes
		llkStateD, // Persistent 'D' state
		llkStateZ, // Persistent 'Z' state
		#ifdef __PTRACE_ENABLED__ // Extra privileged states
		llkStateStack, // stack signature
		#endif // End of extra privilege
		llkNumStates, // Maxumum number of states
		}; // state indexes
		milliseconds llkStateTimeoutMs[llkNumStates]; // timeout override for each detection state
		milliseconds llkCheckMs; // checking interval to inspect any
		// persistent live-locked states
		@@ -83,6 +90,10 @@ bool khtEnable = LLK_ENABLE_DEFAULT; // [khungtaskd] panic
		// Provides a wide angle of margin b/c khtTimeout is also its granularity.
		seconds khtTimeout = duration_cast<seconds>(llkTimeoutMs * (1 + LLK_CHECKS_PER_TIMEOUT_DEFAULT) /
		LLK_CHECKS_PER_TIMEOUT_DEFAULT);
		#ifdef __PTRACE_ENABLED__
		// list of stack symbols to search for persistence.
		std::unordered_set<std::string> llkCheckStackSymbols;
		#endif

		// Blacklist variables, initialized with comma separated lists of high false
		// positive and/or dangerous references, e.g. without self restart, for pid,
		@@ -97,6 +108,11 @@ std::unordered_set<std::string> llkBlacklistProcess;
		std::unordered_set<std::string> llkBlacklistParent;
		// list of uids, and uid names, to skip, default nothing
		std::unordered_set<std::string> llkBlacklistUid;
		#ifdef __PTRACE_ENABLED__
		// list of names to skip stack checking. "init", "lmkd", "llkd", "keystore" or
		// "logd" (if not userdebug).
		std::unordered_set<std::string> llkBlacklistStack;
		#endif

		class dir {
		public:
		@@ -263,6 +279,9 @@ struct proc {
		// forward scheduling progress.
		milliseconds update; // llkUpdate millisecond signature of last.
		milliseconds count; // duration in state.
		#ifdef __PTRACE_ENABLED__ // Privileged state checking
		milliseconds count_stack; // duration where stack is stagnant.
		#endif // End privilege
		pid_t pid; // /proc/<pid> before iterating through
		// /proc/<pid>/task/<tid> for threads.
		pid_t ppid; // /proc/<tid>/stat field 4 parent pid.
		@@ -272,6 +291,9 @@ struct proc {
		std::string cmdline; // cached /cmdline content
		char state; // /proc/<tid>/stat field 3: Z or D
		// (others we do not monitor: S, R, T or ?)
		#ifdef __PTRACE_ENABLED__ // Privileged state checking
		char stack; // index in llkCheckStackSymbols for matches
		#endif // and with maximum index PROP_VALUE_MAX/2.
		char comm[TASK_COMM_LEN + 3]; // space for adding '[' and ']'
		bool exeMissingValid; // exeMissing has been cached
		bool cmdlineValid; // cmdline has been cached
		@@ -286,11 +308,17 @@ struct proc {
		nrSwitches(0),
		update(llkUpdate),
		count(0ms),
		#ifdef __PTRACE_ENABLED__
		count_stack(0ms),
		#endif
		pid(pid),
		ppid(ppid),
		uid(-1),
		time(time),
		state(state),
		#ifdef __PTRACE_ENABLED__
		stack(-1),
		#endif
		exeMissingValid(false),
		cmdlineValid(false),
		updated(true),
		@@ -343,6 +371,10 @@ struct proc {
		void reset(void) { // reset cache, if we detected pid rollover
		uid = -1;
		state = '?';
		#ifdef __PTRACE_ENABLED__
		count_stack = 0ms;
		stack = -1;
		#endif
		cmdline = "";
		comm[0] = '\0';
		exeMissingValid = false;
		@@ -667,6 +699,48 @@ long long getSchedValue(const std::string& schedString, const char* key) {
		return ret;
		}

		#ifdef __PTRACE_ENABLED__
		bool llkCheckStack(proc* procp, const std::string& piddir) {
		if (llkCheckStackSymbols.empty()) return false;
		if (procp->state == 'Z') { // No brains for Zombies
		procp->stack = -1;
		procp->count_stack = 0ms;
		return false;
		}

		// Don't check process that are known to block ptrace, save sepolicy noise.
		if (llkSkipName(std::to_string(procp->pid), llkBlacklistStack)) return false;
		if (llkSkipName(procp->getComm(), llkBlacklistStack)) return false;
		if (llkSkipName(procp->getCmdline(), llkBlacklistStack)) return false;

		auto kernel_stack = ReadFile(piddir + "/stack");
		if (kernel_stack.empty()) {
		LOG(INFO) << piddir << "/stack empty comm=" << procp->getComm()
		<< " cmdline=" << procp->getCmdline();
		return false;
		}
		// A scheduling incident that should not reset count_stack
		if (kernel_stack.find(" cpu_worker_pools+0x") != std::string::npos) return false;
		char idx = -1;
		char match = -1;
		for (const auto& stack : llkCheckStackSymbols) {
		if (++idx < 0) break;
		if (kernel_stack.find(" "s + stack + "+0x") != std::string::npos) {
		match = idx;
		break;
		}
		}
		if (procp->stack != match) {
		procp->stack = match;
		procp->count_stack = 0ms;
		return false;
		}
		if (match == char(-1)) return false;
		procp->count_stack += llkCycle;
		return procp->count_stack >= llkStateTimeoutMs[llkStateStack];
		}
		#endif

		// Primary ABA mitigation watching last time schedule activity happened
		void llkCheckSchedUpdate(proc* procp, const std::string& piddir) {
		// Audit finds /proc/<tid>/sched is just over 1K, and
		@@ -731,7 +805,15 @@ void llkLogConfig(void) {
		<< LLK_TIMEOUT_MS_PROPERTY "=" << llkFormat(llkTimeoutMs) << "\n"
		<< LLK_D_TIMEOUT_MS_PROPERTY "=" << llkFormat(llkStateTimeoutMs[llkStateD]) << "\n"
		<< LLK_Z_TIMEOUT_MS_PROPERTY "=" << llkFormat(llkStateTimeoutMs[llkStateZ]) << "\n"
		#ifdef __PTRACE_ENABLED__
		<< LLK_STACK_TIMEOUT_MS_PROPERTY "=" << llkFormat(llkStateTimeoutMs[llkStateStack])
		<< "\n"
		#endif
		<< LLK_CHECK_MS_PROPERTY "=" << llkFormat(llkCheckMs) << "\n"
		#ifdef __PTRACE_ENABLED__
		<< LLK_CHECK_STACK_PROPERTY "=" << llkFormat(llkCheckStackSymbols) << "\n"
		<< LLK_BLACKLIST_STACK_PROPERTY "=" << llkFormat(llkBlacklistStack) << "\n"
		#endif
		<< LLK_BLACKLIST_PROCESS_PROPERTY "=" << llkFormat(llkBlacklistProcess) << "\n"
		<< LLK_BLACKLIST_PARENT_PROPERTY "=" << llkFormat(llkBlacklistParent) << "\n"
		<< LLK_BLACKLIST_UID_PROPERTY "=" << llkFormat(llkBlacklistUid);
		@@ -892,9 +974,14 @@ milliseconds llkCheck(bool checkRunning) {
		if (pid == myPid) {
		break;
		}
		if (!llkIsMonitorState(state)) {
		#ifdef __PTRACE_ENABLED__
		// if no stack monitoring, we can quickly exit here
		if (!llkIsMonitorState(state) && llkCheckStackSymbols.empty()) {
		continue;
		}
		#else
		if (!llkIsMonitorState(state)) continue;
		#endif
		if ((tid == myTid) \|\| llkSkipPid(tid)) {
		continue;
		}
		@@ -925,12 +1012,26 @@ milliseconds llkCheck(bool checkRunning) {
		// ABA mitigation watching last time schedule activity happened
		llkCheckSchedUpdate(procp, piddir);

		// Can only fall through to here if registered D or Z state !!!
		if (procp->count < llkStateTimeoutMs[(state == 'Z') ? llkStateZ : llkStateD]) {
		#ifdef __PTRACE_ENABLED__
		auto stuck = llkCheckStack(procp, piddir);
		if (llkIsMonitorState(state)) {
		if (procp->count >= llkStateTimeoutMs[(state == 'Z') ? llkStateZ : llkStateD]) {
		stuck = true;
		} else if (procp->count != 0ms) {
		LOG(VERBOSE) << state << ' ' << llkFormat(procp->count) << ' ' << ppid << "->"
		<< pid << "->" << tid << ' ' << procp->getComm();
		}
		}
		if (!stuck) continue;
		#else
		if (procp->count >= llkStateTimeoutMs[(state == 'Z') ? llkStateZ : llkStateD]) {
		if (procp->count != 0ms) {
		LOG(VERBOSE) << state << ' ' << llkFormat(procp->count) << ' ' << ppid << "->"
		<< pid << "->" << tid << ' ' << procp->getComm();
		}
		continue;
		}
		#endif

		// We have to kill it to determine difference between live lock
		// and persistent state blocked on a resource. Is there something
		@@ -969,12 +1070,13 @@ milliseconds llkCheck(bool checkRunning) {
		// not working is we kill a process that likes to
		// stay in 'D' state, instead of panicing the
		// kernel (worse).
		LOG(WARNING) << "D " << llkFormat(procp->count) << ' ' << pid << "->" << tid
		<< ' ' << procp->getComm() << " [kill]";
		default:
		LOG(WARNING) << state << ' ' << llkFormat(procp->count) << ' ' << pid
		<< "->" << tid << ' ' << procp->getComm() << " [kill]";
		if ((llkKillOneProcess(llkTidLookup(pid), procp) >= 0) \|\|
		(llkKillOneProcess(pid, 'D', tid) >= 0) \|\|
		(llkKillOneProcess(pid, state, tid) >= 0) \|\|
		(llkKillOneProcess(procp, procp) >= 0) \|\|
		(llkKillOneProcess(tid, 'D', tid) >= 0)) {
		(llkKillOneProcess(tid, state, tid) >= 0)) {
		continue;
		}
		break;
		@@ -983,7 +1085,8 @@ milliseconds llkCheck(bool checkRunning) {
		// We are here because we have confirmed kernel live-lock
		LOG(ERROR) << state << ' ' << llkFormat(procp->count) << ' ' << ppid << "->" << pid
		<< "->" << tid << ' ' << procp->getComm() << " [panic]";
		llkPanicKernel(true, tid, (state == 'Z') ? "zombie" : "driver");
		llkPanicKernel(true, tid,
		(state == 'Z') ? "zombie" : (state == 'D') ? "driver" : "sleeping");
		}
		LOG(VERBOSE) << "+closedir()";
		}
		@@ -1041,8 +1144,9 @@ unsigned llkCheckMilliseconds() {
		}

		bool llkInit(const char* threadname) {
		auto debuggable = android::base::GetBoolProperty("ro.debuggable", false);
		llkLowRam = android::base::GetBoolProperty("ro.config.low_ram", false);
		if (!LLK_ENABLE_DEFAULT && android::base::GetBoolProperty("ro.debuggable", false)) {
		if (!LLK_ENABLE_DEFAULT && debuggable) {
		llkEnable = android::base::GetProperty(LLK_ENABLE_PROPERTY, "eng") == "eng";
		khtEnable = android::base::GetProperty(KHT_ENABLE_PROPERTY, "eng") == "eng";
		}
		@@ -1069,8 +1173,21 @@ bool llkInit(const char* threadname) {
		llkValidate(); // validate llkTimeoutMs, llkCheckMs and llkCycle
		llkStateTimeoutMs[llkStateD] = GetUintProperty(LLK_D_TIMEOUT_MS_PROPERTY, llkTimeoutMs);
		llkStateTimeoutMs[llkStateZ] = GetUintProperty(LLK_Z_TIMEOUT_MS_PROPERTY, llkTimeoutMs);
		#ifdef __PTRACE_ENABLED__
		llkStateTimeoutMs[llkStateStack] = GetUintProperty(LLK_STACK_TIMEOUT_MS_PROPERTY, llkTimeoutMs);
		#endif
		llkCheckMs = GetUintProperty(LLK_CHECK_MS_PROPERTY, llkCheckMs);
		llkValidate(); // validate all (effectively minus llkTimeoutMs)
		#ifdef __PTRACE_ENABLED__
		if (debuggable) {
		llkCheckStackSymbols = llkSplit(
		android::base::GetProperty(LLK_CHECK_STACK_PROPERTY, LLK_CHECK_STACK_DEFAULT));
		}
		std::string defaultBlacklistStack(LLK_BLACKLIST_STACK_DEFAULT);
		if (!debuggable) defaultBlacklistStack += ",logd,/system/bin/logd";
		llkBlacklistStack = llkSplit(
		android::base::GetProperty(LLK_BLACKLIST_STACK_PROPERTY, defaultBlacklistStack));
		#endif
		std::string defaultBlacklistProcess(
		std::to_string(kernelPid) + "," + std::to_string(initPid) + "," +
		std::to_string(kthreaddPid) + "," + std::to_string(::getpid()) + "," +

llkd/llkd-debuggable.rc

0 → 100644

+19 −0

Original line number	Diff line number	Diff line
		on property:ro.debuggable=1
		setprop llk.enable ${ro.llk.enable:-1}
		setprop khungtask.enable ${ro.khungtask.enable:-1}

		on property:ro.llk.enable=eng
		setprop llk.enable ${ro.debuggable:-0}

		on property:ro.khungtask.enable=eng
		setprop khungtask.enable ${ro.debuggable:-0}

		service llkd-1 /system/bin/llkd
		class late_start
		disabled
		user llkd
		group llkd readproc
		capabilities KILL IPC_LOCK SYS_PTRACE DAC_OVERRIDE
		file /dev/kmsg w
		file /proc/sysrq-trigger w
		writepid /dev/cpuset/system-background/tasks