Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 39732ca5 authored by Eric Wong's avatar Eric Wong Committed by Linus Torvalds
Browse files

epoll: trim epitem by one cache line 



It is common for epoll users to have thousands of epitems, so saving a
cache line on every allocation leads to large memory savings.

Since epitem allocations are cache-aligned, reducing sizeof(struct
epitem) from 136 bytes to 128 bytes will allow it to squeeze under a
cache line boundary on x86_64.

Via /sys/kernel/slab/eventpoll_epi, I see the following changes on my
x86_64 Core2 Duo (which has 64-byte cache alignment):

	object_size  :  192 => 128
	objs_per_slab:   21 =>  32

Also, add a BUILD_BUG_ON() to check for future accidental breakage.

[akpm@linux-foundation.org: use __packed, for all architectures]
Signed-off-by: default avatarEric Wong <normalperson@yhbt.net>
Cc: Davide Libenzi <davidel@xmailserver.org>
Cc: Al Viro <viro@ZenIV.linux.org.uk>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent 4a22f166

fs/eventpoll.c

+9 −1
Original line number Diff line number Diff line
@@ -104,7 +104,7 @@ 
struct epoll_filefd {

	struct file *file;

	int fd;

};

} __packed;



/*

 * Structure used to track possible nested calls, for too deep recursions
@@ -128,6 +128,8 @@ struct nested_calls { 
/*

 * Each file descriptor added to the eventpoll interface will

 * have an entry of this type linked to the "rbr" RB tree.

 * Avoid increasing the size of this struct, there can be many thousands

 * of these on a server and we do not want this to take another cache line.

 */

struct epitem {

	/* RB tree node used to link this structure to the eventpoll RB tree */
@@ -1964,6 +1966,12 @@ static int __init eventpoll_init(void) 
	/* Initialize the structure used to perform file's f_op->poll() calls */

	ep_nested_calls_init(&poll_readywalk_ncalls);



	/*

	 * We can have many thousands of epitems, so prevent this from

	 * using an extra cache line on 64-bit (and smaller) CPUs

	 */

	BUILD_BUG_ON(sizeof(void *) <= 8 && sizeof(struct epitem) > 128);



	/* Allocates slab cache used to allocate "struct epitem" items */

	epi_cache = kmem_cache_create("eventpoll_epi", sizeof(struct epitem),

			0, SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL);