Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit a1e01829 authored by Kirill Tkhai's avatar Kirill Tkhai Committed by Ingo Molnar
Browse files

sched: Remove double_rq_lock() from __migrate_task() 



Avoid double_rq_lock() and use TASK_ON_RQ_MIGRATING for
__migrate_task(). The advantage is (obviously) not holding two
rq->lock's at the same time and thereby increasing parallelism.

The important point to note is that because we acquire dst->lock
immediately after releasing src->lock the potential wait time of
task_rq_lock() callers on TASK_ON_RQ_MIGRATING is not longer
than it would have been in the double rq lock scenario.

Signed-off-by: default avatarKirill Tkhai <ktkhai@parallels.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Paul Turner <pjt@google.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mike Galbraith <umgwanakikbuti@gmail.com>
Cc: Kirill Tkhai <tkhai@yandex.ru>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Nicolas Pitre <nicolas.pitre@linaro.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Link: http://lkml.kernel.org/r/1408528070.23412.89.camel@tkhai


Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
parent cca26e80

kernel/sched/core.c

+15 −8
Original line number Diff line number Diff line
@@ -4679,20 +4679,20 @@ EXPORT_SYMBOL_GPL(set_cpus_allowed_ptr); 
 */

static int __migrate_task(struct task_struct *p, int src_cpu, int dest_cpu)

{

	struct rq *rq_dest, *rq_src;

	struct rq *rq;

	int ret = 0;



	if (unlikely(!cpu_active(dest_cpu)))

		return ret;



	rq_src = cpu_rq(src_cpu);

	rq_dest = cpu_rq(dest_cpu);

	rq = cpu_rq(src_cpu);



	raw_spin_lock(&p->pi_lock);

	double_rq_lock(rq_src, rq_dest);

	raw_spin_lock(&rq->lock);

	/* Already moved. */

	if (task_cpu(p) != src_cpu)

		goto done;



	/* Affinity changed (again). */

	if (!cpumask_test_cpu(dest_cpu, tsk_cpus_allowed(p)))

		goto fail;
@@ -4702,15 +4702,22 @@ static int __migrate_task(struct task_struct *p, int src_cpu, int dest_cpu) 
	 * placed properly.

	 */

	if (task_on_rq_queued(p)) {

		dequeue_task(rq_src, p, 0);

		dequeue_task(rq, p, 0);

		p->on_rq = TASK_ON_RQ_MIGRATING;

		set_task_cpu(p, dest_cpu);

		enqueue_task(rq_dest, p, 0);

		check_preempt_curr(rq_dest, p, 0);

		raw_spin_unlock(&rq->lock);



		rq = cpu_rq(dest_cpu);

		raw_spin_lock(&rq->lock);

		BUG_ON(task_rq(p) != rq);

		p->on_rq = TASK_ON_RQ_QUEUED;

		enqueue_task(rq, p, 0);

		check_preempt_curr(rq, p, 0);

	}

done:

	ret = 1;

fail:

	double_rq_unlock(rq_src, rq_dest);

	raw_spin_unlock(&rq->lock);

	raw_spin_unlock(&p->pi_lock);

	return ret;

}