Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit fd99f91a authored by Peter Zijlstra's avatar Peter Zijlstra Committed by Ingo Molnar
Browse files

sched/idle: Avoid spurious wakeup IPIs 



Because mwait_idle_with_hints() gets called from !idle context it must
call current_clr_polling(). This however means that resched_task() is
very likely to send an IPI even when we were polling:

  CPU0					CPU1

  if (current_set_polling_and_test())
    goto out;

  __monitor(&ti->flags);
  if (!need_resched())
    __mwait(eax, ecx);
					set_tsk_need_resched(p);
					smp_mb();
out:
  current_clr_polling();
					if (!tsk_is_polling(p))
					  smp_send_reschedule(cpu);

So while it is correct (extra IPIs aren't a problem, whereas a missed
IPI would be) it is a performance problem (for some).

Avoid this issue by using fetch_or() to atomically set NEED_RESCHED
and test if POLLING_NRFLAG is set.

Since a CPU stuck in mwait is unlikely to modify the flags word,
contention on the cmpxchg is unlikely and thus we should mostly
succeed in a single go.

Signed-off-by: default avatarPeter Zijlstra <peterz@infradead.org>
Acked-by: default avatarNicolas Pitre <nico@linaro.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Link: http://lkml.kernel.org/n/tip-kf5suce6njh5xf5d3od13rr0@git.kernel.org


Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
parent 69dd0f84

kernel/sched/core.c

+36 −5
Original line number Diff line number Diff line
@@ -505,6 +505,39 @@ static inline void init_hrtick(void) 
}

#endif	/* CONFIG_SCHED_HRTICK */



/*

 * cmpxchg based fetch_or, macro so it works for different integer types

 */

#define fetch_or(ptr, val)						\

({	typeof(*(ptr)) __old, __val = *(ptr);				\

 	for (;;) {							\

 		__old = cmpxchg((ptr), __val, __val | (val));		\

 		if (__old == __val)					\

 			break;						\

 		__val = __old;						\

 	}								\

 	__old;								\

})



#ifdef TIF_POLLING_NRFLAG

/*

 * Atomically set TIF_NEED_RESCHED and test for TIF_POLLING_NRFLAG,

 * this avoids any races wrt polling state changes and thereby avoids

 * spurious IPIs.

 */

static bool set_nr_and_not_polling(struct task_struct *p)

{

	struct thread_info *ti = task_thread_info(p);

	return !(fetch_or(&ti->flags, _TIF_NEED_RESCHED) & _TIF_POLLING_NRFLAG);

}

#else

static bool set_nr_and_not_polling(struct task_struct *p)

{

	set_tsk_need_resched(p);

	return true;

}

#endif



/*

 * resched_task - mark a task 'to be rescheduled now'.

 *
@@ -521,17 +554,15 @@ void resched_task(struct task_struct *p) 
	if (test_tsk_need_resched(p))

		return;



	set_tsk_need_resched(p);



	cpu = task_cpu(p);



	if (cpu == smp_processor_id()) {

		set_tsk_need_resched(p);

		set_preempt_need_resched();

		return;

	}



	/* NEED_RESCHED must be visible before we test polling */

	smp_mb();

	if (!tsk_is_polling(p))

	if (set_nr_and_not_polling(p))

		smp_send_reschedule(cpu);

}