Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 21e94459 authored by Peter Zijlstra's avatar Peter Zijlstra Committed by Ingo Molnar
Browse files

x86/mm: Optimize RESTORE_CR3 



Most NMI/paranoid exceptions will not in fact change pagetables and would
thus not require TLB flushing, however RESTORE_CR3 uses flushing CR3
writes.

Restores to kernel PCIDs can be NOFLUSH, because we explicitly flush the
kernel mappings and now that we track which user PCIDs need flushing we can
avoid those too when possible.

This does mean RESTORE_CR3 needs an additional scratch_reg, luckily both
sites have plenty available.

Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Laight <David.Laight@aculab.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: Eduardo Valentin <eduval@amazon.com>
Cc: Greg KH <gregkh@linuxfoundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Juergen Gross <jgross@suse.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: aliguori@amazon.com
Cc: daniel.gruss@iaik.tugraz.at
Cc: hughd@google.com
Cc: keescook@google.com
Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
parent 6fd166aa

arch/x86/entry/calling.h

+28 −2
Original line number Diff line number Diff line
@@ -281,8 +281,34 @@ For 32-bit we have the following conventions - kernel is built with 
.Ldone_\@:

.endm



.macro RESTORE_CR3 save_reg:req

.macro RESTORE_CR3 scratch_reg:req save_reg:req

	ALTERNATIVE "jmp .Lend_\@", "", X86_FEATURE_PTI



	ALTERNATIVE "jmp .Lwrcr3_\@", "", X86_FEATURE_PCID



	/*

	 * KERNEL pages can always resume with NOFLUSH as we do

	 * explicit flushes.

	 */

	bt	$X86_CR3_PTI_SWITCH_BIT, \save_reg

	jnc	.Lnoflush_\@



	/*

	 * Check if there's a pending flush for the user ASID we're

	 * about to set.

	 */

	movq	\save_reg, \scratch_reg

	andq	$(0x7FF), \scratch_reg

	bt	\scratch_reg, THIS_CPU_user_pcid_flush_mask

	jnc	.Lnoflush_\@



	btr	\scratch_reg, THIS_CPU_user_pcid_flush_mask

	jmp	.Lwrcr3_\@



.Lnoflush_\@:

	SET_NOFLUSH_BIT \save_reg



.Lwrcr3_\@:

	/*

	 * The CR3 write could be avoided when not changing its value,

	 * but would require a CR3 read *and* a scratch register.
@@ -301,7 +327,7 @@ For 32-bit we have the following conventions - kernel is built with 
.endm

.macro SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg:req save_reg:req

.endm

.macro RESTORE_CR3 save_reg:req

.macro RESTORE_CR3 scratch_reg:req save_reg:req

.endm



#endif

arch/x86/entry/entry_64.S

+2 −2
Original line number Diff line number Diff line
@@ -1288,7 +1288,7 @@ ENTRY(paranoid_exit) 
	testl	%ebx, %ebx			/* swapgs needed? */

	jnz	.Lparanoid_exit_no_swapgs

	TRACE_IRQS_IRETQ

	RESTORE_CR3	save_reg=%r14

	RESTORE_CR3	scratch_reg=%rbx save_reg=%r14

	SWAPGS_UNSAFE_STACK

	jmp	.Lparanoid_exit_restore

.Lparanoid_exit_no_swapgs:
@@ -1730,7 +1730,7 @@ end_repeat_nmi: 
	movq	$-1, %rsi

	call	do_nmi



	RESTORE_CR3 save_reg=%r14

	RESTORE_CR3 scratch_reg=%r15 save_reg=%r14



	testl	%ebx, %ebx			/* swapgs needed? */

	jnz	nmi_restore