Merge "Merge remote-tracking branch '318/dev/msm-3.18-sched' into msm318"

     2) time spent waiting on a runqueue

     3) # of timeslices run on this cpu

/proc/sys/kernel/{sched_latency_warn_threshold_us,sched_latency_panic_threshold_us}

----------------

schedstats provides procfs nodes /proc/sys/kernel/sched_latency_warn_threshold_us

and /proc/sys/kernel/sched_latency_panic_threshold_us.  These can be configured

to detect unreasonably high scheduling latency.

Set sched_latency_warn_threshold_us or sched_latency_panic_threshold_us with

non-zero threshold to warn or panic system when scheduling latency higher than

configured threshold is detected.  Default is 0 (disabled) for both.

/proc/sys/kernel/sched_max_latency_us

----------------

/proc/sys/kernel/sched_max_latency_us shows the maximum scheduling latency seen

accross the CPUs.  The file shows the maximum latency seen in microseconds along

with the cpu number or cpu id and the task that incurred maximum latency on that

cpu.  The maximum latency can be reset by writing any value to the file.

A program could be easily written to make use of these extra fields to

report on how well a particular process or set of processes is faring

under the scheduler's policies.  A simple version of such a program is

have been runnable for over some period of time within the tick. For such

tasks the scheduler informs the governor of imminent need for high frequency.

If there exists a task on the runqueue at the tick that has been runnable

for greater than sched_early_detection_duration amount of time, it notifies

for greater than SCHED_EARLY_DETECTION_DURATION amount of time, it notifies

the governor with a fabricated load of the full window at the highest

frequency. The fabricated load is maintained until the task is no longer

runnable or until the next tick.

nt_prev_runnable_sum.

"cpu demand" of a task includes its execution time and can also include its

wait time. 'sched_freq_account_wait_time' tunable controls whether task's wait

wait time. 'SCHED_FREQ_ACCOUNT_WAIT_TIME' controls whether task's wait

time is included in its 'curr_window' and 'prev_window' counters or not.

Needless to say, curr_runnable_sum counter of a cpu is derived from curr_window

	This represents beginning of execution for a task. Provided the task

	refers to a non-idle task, a portion of task's wait time that

	corresponds to the current window being tracked on a cpu is added to

	task's curr_window counter, provided sched_freq_account_wait_time is

	task's curr_window counter, provided SCHED_FREQ_ACCOUNT_WAIT_TIME is

	set. The same quantum is also added to cpu's curr_runnable_sum counter.

	The remaining portion, which corresponds to task's wait time in previous

	window is added to task's prev_window and cpu's prev_runnable_sum

	this event reflects actions taken under PICK_NEXT_TASK (i.e its

	wait time is added to task's curr/prev_window counters as well

	as src_cpu's curr/prev_runnable_sum counters, provided

	sched_freq_account_wait_time tunable is non-zero). After that update,

	SCHED_FREQ_ACCOUNT_WAIT_TIME is non-zero). After that update,

	src_cpu's curr_runnable_sum is reduced by task's curr_window value

	and dst_cpu's curr_runnable_sum is increased by task's curr_window

	value, provided sched_migration_fixup = 1. Similarly, src_cpu's

	prev_runnable_sum is reduced by task's prev_window value and dst_cpu's

	prev_runnable_sum is increased by task's prev_window value,

	provided sched_migration_fixup = 1

	value. Similarly, src_cpu's prev_runnable_sum is reduced by task's

	prev_window value and dst_cpu's prev_runnable_sum is increased by

	task's prev_window value.

IRQ_UPDATE

	This event signifies end of execution of an interrupt handler. This

See also notes on 'cpu.upmigrate_discourage' tunable.

*** 7.6 sched_enable_power_aware

Appears at: /proc/sys/kernel/sched_enable_power_aware

Default value: 0

Controls whether or not per-CPU power values are used in determining

task placement. If this is disabled, tasks are simply placed on the

least capacity CPU that will adequately meet the task's needs as

determined by the task load tracking mechanism. If this is enabled,

after a set of CPUs are determined which will meet the task's

performance needs, a CPU is selected which is reported to have the

lowest power consumption at that time.

*** 7.7 sched_ravg_hist_size

*** 7.6 sched_ravg_hist_size

Appears at: /proc/sys/kernel/sched_ravg_hist_size

This tunable controls the number of samples used from task's sum_history[]

array for determination of its demand.

*** 7.8 sched_window_stats_policy

*** 7.7 sched_window_stats_policy

Appears at: /proc/sys/kernel/sched_window_stats_policy

   samples), where M = sysctl_sched_ravg_hist_size

3. Use average of first M samples, where M = sysctl_sched_ravg_hist_size

*** 7.9 sched_ravg_window

*** 7.8 sched_ravg_window

Appears at: kernel command line argument

currently be set at boot time on the kernel command line (or the

default value of 10ms can be used).

*** 7.10 RAVG_HIST_SIZE

*** 7.9 RAVG_HIST_SIZE

Appears at: compile time only (see RAVG_HIST_SIZE in include/linux/sched.h)

both this and sched_ravg_window then a total of 50ms of task history

would be maintained in 5 10ms windows.

*** 7.11 sched_account_wait_time

Appears at: /proc/sys/kernel/sched_account_wait_time

Default value: 1

This controls whether a task's wait time is accounted as its demand for cpu

and thus the values found in its sum, sum_history[] and demand attributes.

*** 7.12 sched_freq_account_wait_time

Appears at: /proc/sys/kernel/sched_freq_account_wait_time

Default value: 0

This controls whether a task's wait time is accounted in its curr_window and

prev_window attributes and thus in a cpu's curr_runnable_sum and

prev_runnable_sum counters.

*** 7.13 sched_migration_fixup

Appears at: /proc/sys/kernel/sched_migration_fixup

Default value: 1

This controls whether a cpu's busy time counters are adjusted during task

migration.

*** 7.14 sched_freq_inc_notify

*** 7.10 sched_freq_inc_notify

Appears at: /proc/sys/kernel/sched_freq_inc_notify

by scheduler to meet current task demand. Note that sched_freq_inc_notify is

specified in kHz units.

*** 7.15 sched_freq_dec_notify

*** 7.11 sched_freq_dec_notify

Appears at: /proc/sys/kernel/sched_freq_dec_notify

by scheduler to meet current task demand. Note that sched_freq_dec_notify is

specified in kHz units.

** 7.16 sched_heavy_task

Appears at: /proc/sys/kernel/sched_heavy_task

Default value: 0

This tunable can be used to specify a demand value for tasks above which task

are classified as "heavy" tasks. Task's ravg.demand attribute is used for this

comparison. Scheduler will request a raise in cpu frequency when heavy tasks

wakeup after at least one window of sleep, where window size is defined by

sched_ravg_window. Value 0 will disable this feature.

*** 7.17 sched_cpu_high_irqload

*** 7.12 sched_cpu_high_irqload

Appears at: /proc/sys/kernel/sched_cpu_high_irqload

described above, causing the scheduler to try and steer tasks away from

the CPU.

** 7.18 cpu.upmigrate_discourage

*** 7.13 cpu.upmigrate_discourage

Default value : 0

how overcommitment threshold is defined and also notes on

'sched_upmigrate_min_nice' tunable.

*** 7.19 sched_static_cpu_pwr_cost

*** 7.14 sched_static_cpu_pwr_cost

Default value: 0

needed.

*** 7.20 sched_static_cluster_pwr_cost

*** 7.15 sched_static_cluster_pwr_cost

Default value: 0

the worst case power cost of the highest D-state. It is means of biasing task

placement away from idle clusters when necessary.

***7.23 sched_early_detection_duration

Default value: 9500000

Appears at /proc/sys/kernel/sched_early_detection_duration

This governs the time in microseconds that a task has to runnable within one

tick for it to be eligible for the scheduler's early detection feature

under scheduler boost. For more information on the feature itself please

refer to section 5.2.1.

*** 7.24 sched_restrict_cluster_spill

*** 7.16 sched_restrict_cluster_spill

Default value: 0

restricted to the lowest possible power cluster.

*** 7.25 sched_downmigrate

*** 7.17 sched_downmigrate

Appears at: /proc/sys/kernel/sched_downmigrate

(sched_downmigrate).

*** 7.26 sched_small_wakee_task_load

*** 7.18 sched_small_wakee_task_load

Appears at: /proc/sys/kernel/sched_small_wakee_task_load

waker's cluster.

*** 7.27 sched_big_waker_task_load

*** 7.19 sched_big_waker_task_load

Appears at: /proc/sys/kernel/sched_big_waker_task_load

task.  Scheduler places small wakee tasks woken up by big sync waker on the

waker's cluster.

*** 7.28 sched_prefer_sync_wakee_to_waker

*** 7.25 sched_prefer_sync_wakee_to_waker

Appears at: /proc/sys/kernel/sched_prefer_sync_wakee_to_waker

* sched_ravg_window (See Sec 2)

* sched_window_stats_policy (See Sec 2.4)

* sched_account_wait_time (See Sec 7.15)

* sched_ravg_hist_size (See Sec 7.11)

* sched_migration_fixup (See Sec 7.17)

* sched_freq_account_wait_time (See Sec 7.16)

<task>-0     [004] d.h4 12700.711489: sched_reset_all_windows_stats: time_taken 1123 window_start 0 window_size 0 reason POLICY_CHANGE old_val 0 new_val 1

extern unsigned int sysctl_sched_wake_to_idle;

extern unsigned int sysctl_sched_wakeup_load_threshold;

extern unsigned int sysctl_sched_window_stats_policy;

extern unsigned int sysctl_sched_account_wait_time;

extern unsigned int sysctl_sched_ravg_hist_size;

extern unsigned int sysctl_sched_cpu_high_irqload;

extern unsigned int sysctl_sched_freq_account_wait_time;

extern unsigned int sysctl_sched_migration_fixup;

extern unsigned int sysctl_sched_heavy_task_pct;

extern unsigned int sysctl_sched_enable_power_aware;

#if defined(CONFIG_SCHED_FREQ_INPUT) || defined(CONFIG_SCHED_HMP)

extern unsigned int sysctl_sched_init_task_load_pct;

extern unsigned int sysctl_sched_downmigrate_pct;

extern int sysctl_sched_upmigrate_min_nice;

extern unsigned int sysctl_sched_boost;

extern unsigned int sysctl_early_detection_duration;

extern unsigned int sysctl_sched_small_wakee_task_load_pct;

extern unsigned int sysctl_sched_big_waker_task_load_pct;

extern unsigned int sysctl_sched_prefer_sync_wakee_to_waker;

extern unsigned int sysctl_sched_min_runtime;

extern unsigned int sysctl_sched_small_task_pct;

extern unsigned int sysctl_sched_restrict_tasks_spread;

extern unsigned int sysctl_sched_account_wait_time;

extern unsigned int sysctl_sched_freq_account_wait_time;

extern unsigned int sysctl_sched_enable_power_aware;

extern unsigned int sysctl_sched_migration_fixup;

extern unsigned int sysctl_sched_heavy_task_pct;

#else

extern unsigned int sysctl_sched_select_prev_cpu_us;

extern unsigned int sysctl_sched_enable_colocation;

#endif

#ifdef CONFIG_SCHEDSTATS

#ifdef CONFIG_SCHED_QHMP

extern unsigned int sysctl_sched_latency_panic_threshold;

extern unsigned int sysctl_sched_latency_warn_threshold;

extern int sched_max_latency_sysctl(struct ctl_table *table, int write,

				    void __user *buffer, size_t *lenp,

				    loff_t *ppos);

#endif

#endif

extern int sched_rr_timeslice;

struct sched_cluster *sched_cluster[NR_CPUS];

int num_clusters;

unsigned int max_power_cost = 1;

static struct sched_cluster init_cluster = {

	.list			=	LIST_HEAD_INIT(init_cluster.list),

	.id			=	0,

{

	struct sched_cluster *cluster;

	struct list_head new_head;

	unsigned int tmp_max = 1;

	INIT_LIST_HEAD(&new_head);

							       max_task_load());

		cluster->min_power_cost = power_cost(cluster_first_cpu(cluster),

							       0);

		if (cluster->max_power_cost > tmp_max)

			tmp_max = cluster->max_power_cost;

	}

	max_power_cost = tmp_max;

	move_list(&new_head, &cluster_head, true);

early_param("sched_enable_hmp", set_sched_enable_hmp);

static int __init set_sched_enable_power_aware(char *str)

{

	int enable_power_aware = 0;

	get_option(&str, &enable_power_aware);

	sysctl_sched_enable_power_aware = !!enable_power_aware;

	return 0;

}

early_param("sched_enable_power_aware", set_sched_enable_power_aware);

static inline int got_boost_kick(void)

{

	int cpu = smp_processor_id();

#if defined(CONFIG_SCHED_HMP)

/*

 * sched_window_stats_policy, sched_account_wait_time, sched_ravg_hist_size,

 * sched_migration_fixup, sched_freq_account_wait_time have a 'sysctl' copy

 * sched_window_stats_policy and sched_ravg_hist_size have a 'sysctl' copy

 * associated with them. This is required for atomic update of those variables

 * when being modifed via sysctl interface.

 *

/*

 * Tasks that are runnable continuously for a period greather than

 * sysctl_early_detection_duration can be flagged early as potential

 * EARLY_DETECTION_DURATION can be flagged early as potential

 * high load tasks.

 */

__read_mostly unsigned int sysctl_early_detection_duration = 9500000;

#define EARLY_DETECTION_DURATION 9500000

static __read_mostly unsigned int sched_ravg_hist_size = 5;

__read_mostly unsigned int sysctl_sched_ravg_hist_size = 5;

__read_mostly unsigned int sysctl_sched_window_stats_policy =

	WINDOW_STATS_MAX_RECENT_AVG;

static __read_mostly unsigned int sched_account_wait_time = 1;

__read_mostly unsigned int sysctl_sched_account_wait_time = 1;

#define SCHED_ACCOUNT_WAIT_TIME 1

__read_mostly unsigned int sysctl_sched_cpu_high_irqload = (10 * NSEC_PER_MSEC);

__read_mostly unsigned int sysctl_sched_new_task_windows = 5;

static __read_mostly unsigned int sched_migration_fixup = 1;

__read_mostly unsigned int sysctl_sched_migration_fixup = 1;

static __read_mostly unsigned int sched_freq_account_wait_time;

__read_mostly unsigned int sysctl_sched_freq_account_wait_time;

#define SCHED_FREQ_ACCOUNT_WAIT_TIME 0

/*

 * For increase, send notification if

		if (rq->curr == p)

			return 1;

		return p->on_rq ? sched_freq_account_wait_time : 0;

		return p->on_rq ? SCHED_FREQ_ACCOUNT_WAIT_TIME : 0;

	}

	/* TASK_MIGRATE, PICK_NEXT_TASK left */

	return sched_freq_account_wait_time;

}

static inline int

heavy_task_wakeup(struct task_struct *p, struct rq *rq, int event)

{

	u32 task_demand = p->ravg.demand;

	if (!sched_heavy_task || event != TASK_WAKE ||

	    task_demand < sched_heavy_task || exiting_task(p))

		return 0;

	if (p->ravg.mark_start > rq->window_start)

		return 0;

	/* has a full window elapsed since task slept? */

	return (rq->window_start - p->ravg.mark_start > sched_ravg_window);

	return SCHED_FREQ_ACCOUNT_WAIT_TIME;

}

static inline bool is_new_task(struct task_struct *p)

		return;

	if (event != PUT_PREV_TASK && event != TASK_UPDATE &&

			(!sched_freq_account_wait_time ||

			(!SCHED_FREQ_ACCOUNT_WAIT_TIME ||

			 (event != TASK_MIGRATE &&

			 event != PICK_NEXT_TASK)))

		return;

	 * related groups

	 */

	if (event == TASK_UPDATE) {

		if (!p->on_rq && !sched_freq_account_wait_time)

		if (!p->on_rq && !SCHED_FREQ_ACCOUNT_WAIT_TIME)

			return;

	}

		if (p_is_curr_task) {

			/* p is idle task */

			BUG_ON(p != rq->idle);

		} else if (heavy_task_wakeup(p, rq, event)) {

			/* A new window has started. If p is a waking

			 * heavy task its prev_window contribution is faked

			 * to be its window-based demand. Note that this can

			 * introduce phantom load into the system depending

			 * on the window policy and task behavior. This feature

			 * can be controlled via the sched_heavy_task

			 * tunable. */

			p->ravg.prev_window = p->ravg.demand;

			*prev_runnable_sum += p->ravg.demand;

			if (new_task)

				*nt_prev_runnable_sum += p->ravg.demand;

		}

		return;

	u32 hfreq;

	int hpct;

	if (!per_cpu_info || !sysctl_sched_enable_power_aware)

	if (!per_cpu_info)

		return 0;

	spin_lock_irqsave(&freq_max_load_lock, flags);

	 * time. Likewise, if wait time is not treated as busy time, then

	 * when a task begins to run or is migrated, it is not running and

	 * is completing a segment of non-busy time. */

	if (event == TASK_WAKE || (!sched_account_wait_time &&

	if (event == TASK_WAKE || (!SCHED_ACCOUNT_WAIT_TIME &&

			 (event == PICK_NEXT_TASK || event == TASK_MIGRATE)))

		return 0;

enum reset_reason_code {

	WINDOW_CHANGE,

	POLICY_CHANGE,

	ACCOUNT_WAIT_TIME_CHANGE,

	HIST_SIZE_CHANGE,

	MIGRATION_FIXUP_CHANGE,

	FREQ_ACCOUNT_WAIT_TIME_CHANGE,

	FREQ_AGGREGATE_CHANGE,

};

const char *sched_window_reset_reasons[] = {

	"WINDOW_CHANGE",

	"POLICY_CHANGE",

	"ACCOUNT_WAIT_TIME_CHANGE",

	"HIST_SIZE_CHANGE",

	"MIGRATION_FIXUP_CHANGE",

	"FREQ_ACCOUNT_WAIT_TIME_CHANGE"};

};

/* Called with IRQs enabled */

void reset_all_window_stats(u64 window_start, unsigned int window_size)

		old = sched_window_stats_policy;

		new = sysctl_sched_window_stats_policy;

		sched_window_stats_policy = sysctl_sched_window_stats_policy;

	} else if (sched_account_wait_time != sysctl_sched_account_wait_time) {

		reason = ACCOUNT_WAIT_TIME_CHANGE;

		old = sched_account_wait_time;

		new = sysctl_sched_account_wait_time;

		sched_account_wait_time = sysctl_sched_account_wait_time;

	} else if (sched_ravg_hist_size != sysctl_sched_ravg_hist_size) {

		reason = HIST_SIZE_CHANGE;

		old = sched_ravg_hist_size;

		sched_ravg_hist_size = sysctl_sched_ravg_hist_size;

	}

#ifdef CONFIG_SCHED_FREQ_INPUT

	else if (sched_migration_fixup != sysctl_sched_migration_fixup) {

		reason = MIGRATION_FIXUP_CHANGE;

		old = sched_migration_fixup;

		new = sysctl_sched_migration_fixup;

		sched_migration_fixup = sysctl_sched_migration_fixup;

	} else if (sched_freq_account_wait_time !=

					sysctl_sched_freq_account_wait_time) {

		reason = FREQ_ACCOUNT_WAIT_TIME_CHANGE;

		old = sched_freq_account_wait_time;

		new = sysctl_sched_freq_account_wait_time;

		sched_freq_account_wait_time =

				 sysctl_sched_freq_account_wait_time;

	} else if (sched_freq_aggregate !=

	else if (sched_freq_aggregate !=

					sysctl_sched_freq_aggregate) {

		reason = FREQ_AGGREGATE_CHANGE;

		old = sched_freq_aggregate;

	bool new_task;

	struct related_thread_group *grp;

	if (!sched_enable_hmp || !sched_migration_fixup ||

		 (!p->on_rq && p->state != TASK_WAKING))

	if (!sched_enable_hmp || (!p->on_rq && p->state != TASK_WAKING))

		return;

	if (exiting_task(p)) {

static inline void fixup_busy_time(struct task_struct *p, int new_cpu) { }

static inline int

heavy_task_wakeup(struct task_struct *p, struct rq *rq, int event)

{

	return 0;

}

#endif	/* CONFIG_SCHED_FREQ_INPUT */

#define sched_up_down_migrate_auto_update 1

static inline void fixup_busy_time(struct task_struct *p, int new_cpu) { }

static inline int

heavy_task_wakeup(struct task_struct *p, struct rq *rq, int event)

{

	return 0;

}

static struct cpu_cycle

update_task_ravg(struct task_struct *p, struct rq *rq,

			 int event, u64 wallclock, u64 irqtime)

	int cpu, src_cpu, success = 0;

	int notify = 0;

	struct migration_notify_data mnd;

	int heavy_task = 0;

#ifdef CONFIG_SMP

	unsigned int old_load;

	struct rq *rq;

	old_load = task_load(p);

	wallclock = sched_ktime_clock();

	update_task_ravg(rq->curr, rq, TASK_UPDATE, wallclock, 0);

	heavy_task = heavy_task_wakeup(p, rq, TASK_WAKE);

	update_task_ravg(p, rq, TASK_WAKE, wallclock, 0);

	raw_spin_unlock(&rq->lock);

						false, check_group);

			check_for_freq_change(cpu_rq(src_cpu),

						false, check_group);

		} else if (heavy_task) {

			check_for_freq_change(cpu_rq(cpu), false, false);

		} else if (success) {

			check_for_freq_change(cpu_rq(cpu), true, false);

		}

		if (!loop_max)

			break;

		if (wallclock - p->last_wake_ts >=

				sysctl_early_detection_duration) {

		if (wallclock - p->last_wake_ts >= EARLY_DETECTION_DURATION) {

			rq->ed_task = p;

			return 1;

		}