From 05c415bcf6f15a1864cab1133b5ae29a96a19d0f Mon Sep 17 00:00:00 2001 From: Alexander Duyck Date: Sat, 15 Feb 2020 15:19:57 +1100 Subject: [PATCH 0864/2944] mm/page_reporting: rotate reported pages to the tail of the list to #26589565 Rather than walking over the same pages again and again to get to the pages that have yet to be reported we can save ourselves a significant amount of time by simply rotating the list so that when we have a full list of reported pages the head of the list is pointing to the next non-reported page. Doing this should save us some significant time when processing each free list. This doesn't gain us much in the standard case as all of the non-reported pages should be near the top of the list already. However in the case of page shuffling this results in a noticeable improvement. Below are the will-it-scale page_fault1 w/ THP numbers for 16 tasks with and without this patch. Without: tasks processes processes_idle threads threads_idle 16 8093776.25 0.17 5393242.00 38.20 With: tasks processes processes_idle threads threads_idle 16 8283274.75 0.17 5594261.00 38.15 Link: http://lkml.kernel.org/r/20200211224708.29318.16862.stgit@localhost.localdomain Signed-off-by: Alexander Duyck Cc: Andrea Arcangeli Cc: Dan Williams Cc: Dave Hansen Cc: David Hildenbrand Cc: Konrad Rzeszutek Wilk Cc: Luiz Capitulino Cc: Matthew Wilcox Cc: Mel Gorman Cc: Michael S. Tsirkin Cc: Michal Hocko Cc: Nitesh Narayan Lal Cc: Oscar Salvador Cc: Pankaj Gupta Cc: Paolo Bonzini Cc: Rik van Riel Cc: Vlastimil Babka Cc: Wei Wang Cc: Yang Zhang Cc: wei qi Signed-off-by: Andrew Morton Signed-off-by: Stephen Rothwell Signed-off-by: Yang Shi Reviewed-by: Xunlei Pang Signed-off-by: Shile Zhang --- mm/page_reporting.c | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/mm/page_reporting.c b/mm/page_reporting.c index 1047c68..6885e74 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -131,17 +131,27 @@ void __page_reporting_notify(void) if (PageReported(page)) continue; - /* Attempt to pull page from list */ - if (!__isolate_free_page(page, order)) - break; + /* Attempt to pull page from list and place in scatterlist */ + if (*offset) { + if (!__isolate_free_page(page, order)) { + next = page; + break; + } - /* Add page to scatter list */ - --(*offset); - sg_set_page(&sgl[*offset], page, page_len, 0); + /* Add page to scatter list */ + --(*offset); + sg_set_page(&sgl[*offset], page, page_len, 0); - /* If scatterlist isn't full grab more pages */ - if (*offset) continue; + } + + /* + * Make the first non-processed page in the free list + * the new head of the free list before we release the + * zone lock. + */ + if (&page->lru != list && !list_is_first(&page->lru, list)) + list_rotate_to_front(&page->lru, list); /* release lock before waiting on report processing */ spin_unlock_irq(&zone->lock); @@ -169,6 +179,10 @@ void __page_reporting_notify(void) break; } + /* Rotate any leftover pages to the head of the freelist */ + if (&next->lru != list && !list_is_first(&next->lru, list)) + list_rotate_to_front(&next->lru, list); + spin_unlock_irq(&zone->lock); return err; -- 1.8.3.1