Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit f5e6a975 authored by Andy Lutomirski's avatar Andy Lutomirski Committed by Ingo Molnar
Browse files

x86/entry: Split and inline syscall_return_slowpath() 



GCC is unable to properly optimize functions that have a very
short likely case and a longer and register-heavier cold part --
it fails to sink all of the register saving and stack frame
setup code into the unlikely part.

Help it out with syscall_return_slowpath() by splitting it into
two parts and inline the hot part.

Saves 6 cycles for compat syscalls.

Signed-off-by: default avatarAndy Lutomirski <luto@kernel.org>
Cc: Andy Lutomirski <luto@amacapital.net>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Link: http://lkml.kernel.org/r/0f773a894ab15c589ac794c2d34ca6ba9b5335c9.1444091585.git.luto@kernel.org


Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
parent 39b48e57

arch/x86/entry/common.c

+29 −21
Original line number Diff line number Diff line
@@ -284,28 +284,14 @@ __visible inline void prepare_exit_to_usermode(struct pt_regs *regs) 
	user_enter();

}



/*

 * Called with IRQs on and fully valid regs.  Returns with IRQs off in a

 * state such that we can immediately switch to user mode.

 */

__visible void syscall_return_slowpath(struct pt_regs *regs)

#define SYSCALL_EXIT_WORK_FLAGS				\

	(_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT |	\

	 _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT)



static void syscall_slow_exit_work(struct pt_regs *regs, u32 cached_flags)

{

	struct thread_info *ti = pt_regs_to_thread_info(regs);

	u32 cached_flags = READ_ONCE(ti->flags);

	bool step;



	CT_WARN_ON(ct_state() != CONTEXT_KERNEL);



	if (IS_ENABLED(CONFIG_PROVE_LOCKING) &&

	    WARN(irqs_disabled(), "syscall %ld left IRQs disabled", regs->orig_ax))

		local_irq_enable();



	/*

	 * First do one-time work.  If these work items are enabled, we

	 * want to run them exactly once per syscall exit with IRQs on.

	 */

	if (cached_flags & (_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT |

			    _TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT)) {

	audit_syscall_exit(regs);



	if (cached_flags & _TIF_SYSCALL_TRACEPOINT)
@@ -324,6 +310,28 @@ __visible void syscall_return_slowpath(struct pt_regs *regs) 
		tracehook_report_syscall_exit(regs, step);

}



/*

 * Called with IRQs on and fully valid regs.  Returns with IRQs off in a

 * state such that we can immediately switch to user mode.

 */

__visible inline void syscall_return_slowpath(struct pt_regs *regs)

{

	struct thread_info *ti = pt_regs_to_thread_info(regs);

	u32 cached_flags = READ_ONCE(ti->flags);



	CT_WARN_ON(ct_state() != CONTEXT_KERNEL);



	if (IS_ENABLED(CONFIG_PROVE_LOCKING) &&

	    WARN(irqs_disabled(), "syscall %ld left IRQs disabled", regs->orig_ax))

		local_irq_enable();



	/*

	 * First do one-time work.  If these work items are enabled, we

	 * want to run them exactly once per syscall exit with IRQs on.

	 */

	if (unlikely(cached_flags & SYSCALL_EXIT_WORK_FLAGS))

		syscall_slow_exit_work(regs, cached_flags);



#ifdef CONFIG_COMPAT

	/*

	 * Compat syscalls set TS_COMPAT.  Make sure we clear it before