Commit d2ee40ea authored Jun 04, 2014 by Jianyu Zhan Committed by Linus Torvalds Jun 04, 2014

mm: introdule compound_head_by_tail()



Currently, in put_compound_page(), we have

======
if (likely(!PageTail(page))) {                  <------  (1)
        if (put_page_testzero(page)) {
                 /*
                 ¦* By the time all refcounts have been released
                 ¦* split_huge_page cannot run anymore from under us.
                 ¦*/
                 if (PageHead(page))
                         __put_compound_page(page);
                 else
                         __put_single_page(page);
         }
         return;
}

/* __split_huge_page_refcount can run under us */
page_head = compound_head(page);        <------------ (2)
======

if at (1) ,  we fail the check, this means page is *likely* a tail page.

Then at (2), as compoud_head(page) is inlined, it is :

======
static inline struct page *compound_head(struct page *page)
{
          if (unlikely(PageTail(page))) {           <----------- (3)
              struct page *head = page->first_page;

                smp_rmb();
                if (likely(PageTail(page)))
                        return head;
        }
        return page;
}
======

here, the (3) unlikely in the case is a negative hint, because it is
*likely* a tail page.  So the check (3) in this case is not good, so I
introduce a helper for this case.

So this patch introduces compound_head_by_tail() which deals with a
possible tail page(though it could be spilt by a racy thread), and make
compound_head() a wrapper on it.

This patch has no functional change, and it reduces the object
size slightly:
   text    data     bss     dec     hex  filename
  11003    1328      16   12347    303b  mm/swap.o.orig
  10971    1328      16   12315    301b  mm/swap.o.patched

I've ran "perf top -e branch-miss" to observe branch-miss in this case.
As Michael points out, it's a slow path, so only very few times this case
happens.  But I grep'ed the code base, and found there still are some
other call sites could be benifited from this helper.  And given that it
only bloating up the source by only 5 lines, but with a reduced object
size.  I still believe this helper deserves to exsit.

Signed-off-by: Jianyu Zhan <nasa4836@gmail.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Jiang Liu <liuj97@gmail.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Sasha Levin <sasha.levin@oracle.com>
Cc: Wanpeng Li <liwanp@linux.vnet.ibm.com>
Cc: Hugh Dickins <hughd@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

parent 4bd3e8f7

include/linux/mm.h

+17 −12

Original line number	Original line	Diff line number	Diff line
	@@ -407,10 +407,9 @@ static inline void compound_unlock_irqrestore(struct page *page,
	#endif		#endif
	}		}

	static inline struct page compound_head(struct page page)		static inline struct page compound_head_by_tail(struct page tail)
	{		{
	if (unlikely(PageTail(page))) {		struct page *head = tail->first_page;
	struct page *head = page->first_page;

	/*		/*
	* page->first_page may be a dangling pointer to an old		* page->first_page may be a dangling pointer to an old
	@@ -418,9 +417,15 @@ static inline struct page compound_head(struct page page)
	* page before returning.		* page before returning.
	*/		*/
	smp_rmb();		smp_rmb();
	if (likely(PageTail(page)))		if (likely(PageTail(tail)))
	return head;		return head;
			return tail;
	}		}

			static inline struct page compound_head(struct page page)
			{
			if (unlikely(PageTail(page)))
			return compound_head_by_tail(page);
	return page;		return page;
	}		}

mm/swap.c

+1 −1

Original line number	Original line	Diff line number	Diff line
	@@ -253,7 +253,7 @@ static void put_compound_page(struct page *page)
	* Case 3 is possible, as we may race with		* Case 3 is possible, as we may race with
	* __split_huge_page_refcount tearing down a THP page.		* __split_huge_page_refcount tearing down a THP page.
	*/		*/
	page_head = compound_head(page);		page_head = compound_head_by_tail(page);
	if (!__compound_tail_refcounted(page_head))		if (!__compound_tail_refcounted(page_head))
	put_unrefcounted_compound_page(page_head, page);		put_unrefcounted_compound_page(page_head, page);
	else		else