Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit fa28ba21 authored by Jeremy Fitzhardinge's avatar Jeremy Fitzhardinge Committed by Ingo Molnar
Browse files

x86: defer cr3 reload when doing pud_clear() 



PAE mode requires that we reload cr3 in order to guarantee that
changes to the pgd will be noticed by the processor.  This means that
in principle pud_clear needs to reload cr3 every time.  However,
because reloading cr3 implies a tlb flush, we want to avoid it where
possible.

pud_clear() is only used in a couple of places:
 - in free_pmd_range(), when pulling down a range of process address space, and
 - huge_pmd_unshare()

In both cases, the calling code will do a a tlb flush anyway, so
there's no need to do it within pud_clear().

In free_pmd_range(), the pud_clear is immediately followed by
pmd_free_tlb(); we can hook that to make the mmu_gather do an
unconditional full flush to make sure cr3 gets reloaded.

In huge_pmd_unshare, it is followed by flush_tlb_range, which always
results in a full cr3-reload tlb flush.

Signed-off-by: default avatarJeremy Fitzhardinge <jeremy@xensource.com>
Cc: Andi Kleen <ak@suse.de>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: William Irwin <wli@holomorphy.com>
Signed-off-by: default avatarIngo Molnar <mingo@elte.hu>
Signed-off-by: default avatarThomas Gleixner <tglx@linutronix.de>
parent f212ec4b

include/asm-x86/pgalloc_32.h

+7 −0
Original line number Diff line number Diff line
@@ -74,6 +74,13 @@ static inline void pmd_free(pmd_t *pmd) 


static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)

{

	/* This is called just after the pmd has been detached from

	   the pgd, which requires a full tlb flush to be recognized

	   by the CPU.  Rather than incurring multiple tlb flushes

	   while the address space is being pulled down, make the tlb

	   gathering machinery do a full flush when we're done. */

	tlb->fullmm = 1;



	paravirt_release_pd(__pa(pmd) >> PAGE_SHIFT);

	tlb_remove_page(tlb, virt_to_page(pmd));

}

include/asm-x86/pgtable-3level.h

+15 −6
Original line number Diff line number Diff line
@@ -96,14 +96,23 @@ static inline void pud_clear(pud_t *pudp) 
	set_pud(pudp, __pud(0));



	/*

	 * Pentium-II erratum A13: in PAE mode we explicitly have to flush

	 * the TLB via cr3 if the top-level pgd is changed...

	 * In principle we need to do a cr3 reload here to make sure

	 * the processor recognizes the changed pgd.  In practice, all

	 * the places where pud_clear() gets called are followed by

	 * full tlb flushes anyway, so we can defer the cost here.

	 *

	 * XXX I don't think we need to worry about this here, since

	 * when clearing the pud, the calling code needs to flush the

	 * tlb anyway.  But do it now for safety's sake. - jsgf

	 * Specifically:

	 *

	 * mm/memory.c:free_pmd_range() - immediately after the

	 * pud_clear() it does a pmd_free_tlb().  We change the

	 * mmu_gather structure to do a full tlb flush (which has the

	 * effect of reloading cr3) when the pagetable free is

	 * complete.

	 *

	 * arch/x86/mm/hugetlbpage.c:huge_pmd_unshare() - the call to

	 * this is followed by a flush_tlb_range, which on x86 does a

	 * full tlb flush.

	 */

	write_cr3(read_cr3());

}



#define pud_page(pud) \