Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 6c1051ff authored by Joerg Roedel's avatar Joerg Roedel Committed by Greg Kroah-Hartman
Browse files

x86/mm: split vmalloc_sync_all() 



commit 763802b53a427ed3cbd419dbba255c414fdd9e7c upstream.

Commit 3f8fd02b1bf1 ("mm/vmalloc: Sync unmappings in
__purge_vmap_area_lazy()") introduced a call to vmalloc_sync_all() in
the vunmap() code-path.  While this change was necessary to maintain
correctness on x86-32-pae kernels, it also adds additional cycles for
architectures that don't need it.

Specifically on x86-64 with CONFIG_VMAP_STACK=y some people reported
severe performance regressions in micro-benchmarks because it now also
calls the x86-64 implementation of vmalloc_sync_all() on vunmap().  But
the vmalloc_sync_all() implementation on x86-64 is only needed for newly
created mappings.

To avoid the unnecessary work on x86-64 and to gain the performance
back, split up vmalloc_sync_all() into two functions:

	* vmalloc_sync_mappings(), and
	* vmalloc_sync_unmappings()

Most call-sites to vmalloc_sync_all() only care about new mappings being
synchronized.  The only exception is the new call-site added in the
above mentioned commit.

Shile Zhang directed us to a report of an 80% regression in reaim
throughput.

Fixes: 3f8fd02b1bf1 ("mm/vmalloc: Sync unmappings in __purge_vmap_area_lazy()")
Reported-by: default avatarkernel test robot <oliver.sang@intel.com>
Reported-by: default avatarShile Zhang <shile.zhang@linux.alibaba.com>
Signed-off-by: default avatarJoerg Roedel <jroedel@suse.de>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Tested-by: default avatarBorislav Petkov <bp@suse.de>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>	[GHES]
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: <stable@vger.kernel.org>
Link: http://lkml.kernel.org/r/20191009124418.8286-1-joro@8bytes.org
Link: https://lists.01.org/hyperkitty/list/lkp@lists.01.org/thread/4D3JPPHBNOSPFK2KEPC6KGKS6J25AIDB/
Link: http://lkml.kernel.org/r/20191113095530.228959-1-shile.zhang@linux.alibaba.com


Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
parent 51a4b00e

arch/x86/mm/fault.c

+24 −2
Original line number Diff line number Diff line
@@ -273,7 +273,7 @@ static inline pmd_t *vmalloc_sync_one(pgd_t *pgd, unsigned long address) 
	return pmd_k;

}



void vmalloc_sync_all(void)

static void vmalloc_sync(void)

{

	unsigned long address;
@@ -300,6 +300,16 @@ void vmalloc_sync_all(void) 
	}

}



void vmalloc_sync_mappings(void)

{

	vmalloc_sync();

}



void vmalloc_sync_unmappings(void)

{

	vmalloc_sync();

}



/*

 * 32-bit:

 *
@@ -402,11 +412,23 @@ static void dump_pagetable(unsigned long address) 


#else /* CONFIG_X86_64: */



void vmalloc_sync_all(void)

void vmalloc_sync_mappings(void)

{

	/*

	 * 64-bit mappings might allocate new p4d/pud pages

	 * that need to be propagated to all tasks' PGDs.

	 */

	sync_global_pgds(VMALLOC_START & PGDIR_MASK, VMALLOC_END);

}



void vmalloc_sync_unmappings(void)

{

	/*

	 * Unmappings never allocate or free p4d/pud pages.

	 * No work is required here.

	 */

}



/*

 * 64-bit:

 *

drivers/acpi/apei/ghes.c

+1 −1
Original line number Diff line number Diff line
@@ -201,7 +201,7 @@ static int ghes_estatus_pool_expand(unsigned long len) 
	 * New allocation must be visible in all pgd before it can be found by

	 * an NMI allocating from the pool.

	 */

	vmalloc_sync_all();

	vmalloc_sync_mappings();



	return gen_pool_add(ghes_estatus_pool, addr, PAGE_ALIGN(len), -1);

}

include/linux/vmalloc.h

+3 −2
Original line number Diff line number Diff line
@@ -107,7 +107,8 @@ extern int remap_vmalloc_range_partial(struct vm_area_struct *vma, 


extern int remap_vmalloc_range(struct vm_area_struct *vma, void *addr,

							unsigned long pgoff);

void vmalloc_sync_all(void);

void vmalloc_sync_mappings(void);

void vmalloc_sync_unmappings(void);



/*

 *	Lowlevel-APIs (not for driver use!)

kernel/notifier.c

+1 −1
Original line number Diff line number Diff line
@@ -552,7 +552,7 @@ NOKPROBE_SYMBOL(notify_die); 


int register_die_notifier(struct notifier_block *nb)

{

	vmalloc_sync_all();

	vmalloc_sync_mappings();

	return atomic_notifier_chain_register(&die_chain, nb);

}

EXPORT_SYMBOL_GPL(register_die_notifier);

mm/nommu.c

+7 −3
Original line number Diff line number Diff line
@@ -446,10 +446,14 @@ void vm_unmap_aliases(void) 
EXPORT_SYMBOL_GPL(vm_unmap_aliases);



/*

 * Implement a stub for vmalloc_sync_all() if the architecture chose not to

 * have one.

 * Implement a stub for vmalloc_sync_[un]mapping() if the architecture

 * chose not to have one.

 */

void __weak vmalloc_sync_all(void)

void __weak vmalloc_sync_mappings(void)

{

}



void __weak vmalloc_sync_unmappings(void)

{

}