Improving the Performance of Transparent Huge Pages in Linux, by Khalid Aziz

static void put_compound_page(struct page *page)
{
if (unlikely(PageTail(page))) {
/* __split_huge_page_refcount can run under us */
struct page *page_head = page->first_page;
smp_rmb();
/*
 * If PageTail is still set after smp_rmb() we can be sure
 * that the page->first_page we read wasn't a dangling pointer.
 * See __split_huge_page_refcount() smp_wmb().
 */
if (likely(PageTail(page) && get_page_unless_zero(page_head))) {
unsigned long flags;
/*
 * Verify that our page_head wasn't converted
 * to a a regular page before we got a
 * reference on it.
 */
if (unlikely(!PageHead(page_head))) {
/* PageHead is cleared after PageTail */
smp_rmb();
VM_BUG_ON(PageTail(page));
goto out_put_head;
}
/*
 * Only run compound_lock on a valid PageHead,
 * after having it pinned with
 * get_page_unless_zero() above.
 */
smp_mb();
/* page_head wasn't a dangling pointer */
flags = compound_lock_irqsave(page_head);
if (unlikely(!PageTail(page))) {
/* __split_huge_page_refcount run before us */
compound_unlock_irqrestore(page_head, flags);
VM_BUG_ON(PageHead(page_head));
out_put_head:
if (put_page_testzero(page_head))
__put_single_page(page_head);
out_put_single:
if (put_page_testzero(page))
__put_single_page(page);
return;
}
VM_BUG_ON(page_head != page->first_page);
/*
 * We can release the refcount taken by
 * get_page_unless_zero now that
 * split_huge_page_refcount is blocked on the
 * compound_lock.
 */
if (put_page_testzero(page_head))
VM_BUG_ON(1);
/* __split_huge_page_refcount will wait now */
VM_BUG_ON(atomic_read(&page->_count) <= 0);
atomic_dec(&page->_count);
VM_BUG_ON(atomic_read(&page_head->_count) <= 0);
compound_unlock_irqrestore(page_head, flags);
if (put_page_testzero(page_head)) {
if (PageHead(page_head))
__put_compound_page(page_head);
else
__put_single_page(page_head);
}
} else {
/* page_head is a dangling pointer */
VM_BUG_ON(PageTail(page));
goto out_put_single;
}
} else if (put_page_testzero(page)) {
if (PageHead(page))
__put_compound_page(page);
else
__put_single_page(page);
}
}

The level of complexity of code
went up significantly. This complexity guaranteed correctness but
sacrificed performance.

Large database applications read
large chunks of database into memory using AIO. When databases
started using hugepages for these reads into memory, performance went
up significantly due to the benefit of much lower number of TLB
misses and significantly smaller amount of memory being used up by
page table resulting in lower swapping activity. When a database
application reads data from disk into memory using AIO, pages from the
hugepages pool are allocated for the read and the block I/O subsystem
grabs reference to these pages for read and later releases reference
to these pages when read is done. This causes traversal of the code
referenced above starting with call to put_page().
With the newly introduced THP code, the additional overhead added up
to significant performance penalty.

Over the next several kernel
releases, the THP code was refined and optimized which helped slightly in
some cases while performance got worse in other cases. Subsequent
refinements to THP code to do accurate accounting of tail pages
introduced the routine __get_page_tail() which is called by get_page()
to grab tail pages for the hugepage. This added further performance
impact to AIO into hugetlbfs pages. All of this code stays in the
code path as long as kernel was compiled with CONFIG_TRANSPARENT_HUGEPAGE. Running echo never > /sys/kernel/mm/transparent_hugepage/enabled does not bypass this new additional code to support THP. Results from
a database performance benchmark run using two common read sizes used
by databases show this performance degradation clearly:

This amounts to 22% degradation for
1M read and 45% degradation for 64K read! perf
top during benchmark runs showed CPU spending more than 40% of
cycles in __get_page_tail()
and put_compound_page().

The Solution

An
Immediate solution to the performance degradation comes from the fact
that hugetlbfs pages can never be split and hence all the overhead
added for THP can be bypassed. I added code to
__get_page_tail() and
put_compound_page()to check
for hugetlbfs page up front and bypass all the additional checks for
those pages:

static void put_compound_page(struct page *page) 
{ 
      if (PageHuge(page)) { 
              page = compound_head(page); 
              if (put_page_testzero(page)) 
                      __put_compound_page(page); 
              return; 
      } 
...
bool __get_page_tail(struct page *page)
{
...

      if (PageHuge(page)) { 
              page_head = compound_head(page); 
              atomic_inc(&page_head->_count); 
             got = true; 
      } else { 
...

This
resulted in immediate performance gain. Running the same benchmark as
before with THP enabled, the new performance numbers for aio reads
are below:

This
patch was sent to linux-mm and linux kernel mailing lists in August
2013 [link]
and was subsequently integrated into kernel version 3.12. This is a
significant performance boost for database applications.

Further
review of the original patch by Andrea Arcangeli during integration
of this patch into stable kernels exposed issues with refcounting of
pages and revealed this patch had introduced a subtle bug where a
page pointer could become a dangling link under certain
circumstances. Andrea Arcangeli and author worked to address these
issues and revised the code in __get_page_tail() and put_compound_page() to
eliminate extraneous locks and memory barriers, fixed incorrect
refcounting of tail pages and eliminate some of the inefficiencies in
the code.

Andrea sent out an initial series of patches to address all
of these issues [link].

Further discussions and refinements led to the final version of these
patches which were integrated into kernel version 3.13 [link],[link].