License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 08:07:57 -06:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2016-07-26 16:26:24 -06:00
|
|
|
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
|
|
|
|
|
|
|
|
#include <linux/mm.h>
|
|
|
|
#include <linux/sched.h>
|
2017-02-08 10:51:29 -07:00
|
|
|
#include <linux/sched/mm.h>
|
2017-02-08 10:51:30 -07:00
|
|
|
#include <linux/sched/coredump.h>
|
2016-07-26 16:26:24 -06:00
|
|
|
#include <linux/mmu_notifier.h>
|
|
|
|
#include <linux/rmap.h>
|
|
|
|
#include <linux/swap.h>
|
|
|
|
#include <linux/mm_inline.h>
|
|
|
|
#include <linux/kthread.h>
|
|
|
|
#include <linux/khugepaged.h>
|
|
|
|
#include <linux/freezer.h>
|
|
|
|
#include <linux/mman.h>
|
|
|
|
#include <linux/hashtable.h>
|
|
|
|
#include <linux/userfaultfd_k.h>
|
|
|
|
#include <linux/page_idle.h>
|
|
|
|
#include <linux/swapops.h>
|
2016-07-26 16:26:32 -06:00
|
|
|
#include <linux/shmem_fs.h>
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
#include <asm/tlb.h>
|
|
|
|
#include <asm/pgalloc.h>
|
|
|
|
#include "internal.h"
|
|
|
|
|
|
|
|
enum scan_result {
|
|
|
|
SCAN_FAIL,
|
|
|
|
SCAN_SUCCEED,
|
|
|
|
SCAN_PMD_NULL,
|
|
|
|
SCAN_EXCEED_NONE_PTE,
|
|
|
|
SCAN_PTE_NON_PRESENT,
|
|
|
|
SCAN_PAGE_RO,
|
2016-07-26 16:26:46 -06:00
|
|
|
SCAN_LACK_REFERENCED_PAGE,
|
2016-07-26 16:26:24 -06:00
|
|
|
SCAN_PAGE_NULL,
|
|
|
|
SCAN_SCAN_ABORT,
|
|
|
|
SCAN_PAGE_COUNT,
|
|
|
|
SCAN_PAGE_LRU,
|
|
|
|
SCAN_PAGE_LOCK,
|
|
|
|
SCAN_PAGE_ANON,
|
|
|
|
SCAN_PAGE_COMPOUND,
|
|
|
|
SCAN_ANY_PROCESS,
|
|
|
|
SCAN_VMA_NULL,
|
|
|
|
SCAN_VMA_CHECK,
|
|
|
|
SCAN_ADDRESS_RANGE,
|
|
|
|
SCAN_SWAP_CACHE_PAGE,
|
|
|
|
SCAN_DEL_PAGE_LRU,
|
|
|
|
SCAN_ALLOC_HUGE_PAGE_FAIL,
|
|
|
|
SCAN_CGROUP_CHARGE_FAIL,
|
2016-07-26 16:26:32 -06:00
|
|
|
SCAN_EXCEED_SWAP_PTE,
|
|
|
|
SCAN_TRUNCATED,
|
2019-09-23 16:38:00 -06:00
|
|
|
SCAN_PAGE_HAS_PRIVATE,
|
2016-07-26 16:26:24 -06:00
|
|
|
};
|
|
|
|
|
|
|
|
#define CREATE_TRACE_POINTS
|
|
|
|
#include <trace/events/huge_memory.h>
|
|
|
|
|
|
|
|
/* default scan 8*512 pte (or vmas) every 30 second */
|
|
|
|
static unsigned int khugepaged_pages_to_scan __read_mostly;
|
|
|
|
static unsigned int khugepaged_pages_collapsed;
|
|
|
|
static unsigned int khugepaged_full_scans;
|
|
|
|
static unsigned int khugepaged_scan_sleep_millisecs __read_mostly = 10000;
|
|
|
|
/* during fragmentation poll the hugepage allocator once every minute */
|
|
|
|
static unsigned int khugepaged_alloc_sleep_millisecs __read_mostly = 60000;
|
|
|
|
static unsigned long khugepaged_sleep_expire;
|
|
|
|
static DEFINE_SPINLOCK(khugepaged_mm_lock);
|
|
|
|
static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait);
|
|
|
|
/*
|
|
|
|
* default collapse hugepages if there is at least one pte mapped like
|
|
|
|
* it would have happened if the vma was large enough during page
|
|
|
|
* fault.
|
|
|
|
*/
|
|
|
|
static unsigned int khugepaged_max_ptes_none __read_mostly;
|
|
|
|
static unsigned int khugepaged_max_ptes_swap __read_mostly;
|
|
|
|
|
|
|
|
#define MM_SLOTS_HASH_BITS 10
|
|
|
|
static __read_mostly DEFINE_HASHTABLE(mm_slots_hash, MM_SLOTS_HASH_BITS);
|
|
|
|
|
|
|
|
static struct kmem_cache *mm_slot_cache __read_mostly;
|
|
|
|
|
2019-09-23 16:38:30 -06:00
|
|
|
#define MAX_PTE_MAPPED_THP 8
|
|
|
|
|
2016-07-26 16:26:24 -06:00
|
|
|
/**
|
|
|
|
* struct mm_slot - hash lookup from mm to mm_slot
|
|
|
|
* @hash: hash collision list
|
|
|
|
* @mm_node: khugepaged scan list headed in khugepaged_scan.mm_head
|
|
|
|
* @mm: the mm that this information is valid for
|
|
|
|
*/
|
|
|
|
struct mm_slot {
|
|
|
|
struct hlist_node hash;
|
|
|
|
struct list_head mm_node;
|
|
|
|
struct mm_struct *mm;
|
2019-09-23 16:38:30 -06:00
|
|
|
|
|
|
|
/* pte-mapped THP in this mm */
|
|
|
|
int nr_pte_mapped_thp;
|
|
|
|
unsigned long pte_mapped_thp[MAX_PTE_MAPPED_THP];
|
2016-07-26 16:26:24 -06:00
|
|
|
};
|
|
|
|
|
|
|
|
/**
|
|
|
|
* struct khugepaged_scan - cursor for scanning
|
|
|
|
* @mm_head: the head of the mm list to scan
|
|
|
|
* @mm_slot: the current mm_slot we are scanning
|
|
|
|
* @address: the next address inside that to be scanned
|
|
|
|
*
|
|
|
|
* There is only the one khugepaged_scan instance of this cursor structure.
|
|
|
|
*/
|
|
|
|
struct khugepaged_scan {
|
|
|
|
struct list_head mm_head;
|
|
|
|
struct mm_slot *mm_slot;
|
|
|
|
unsigned long address;
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct khugepaged_scan khugepaged_scan = {
|
|
|
|
.mm_head = LIST_HEAD_INIT(khugepaged_scan.mm_head),
|
|
|
|
};
|
|
|
|
|
2016-11-30 16:54:02 -07:00
|
|
|
#ifdef CONFIG_SYSFS
|
2016-07-26 16:26:24 -06:00
|
|
|
static ssize_t scan_sleep_millisecs_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
char *buf)
|
|
|
|
{
|
|
|
|
return sprintf(buf, "%u\n", khugepaged_scan_sleep_millisecs);
|
|
|
|
}
|
|
|
|
|
|
|
|
static ssize_t scan_sleep_millisecs_store(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
const char *buf, size_t count)
|
|
|
|
{
|
|
|
|
unsigned long msecs;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
err = kstrtoul(buf, 10, &msecs);
|
|
|
|
if (err || msecs > UINT_MAX)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
khugepaged_scan_sleep_millisecs = msecs;
|
|
|
|
khugepaged_sleep_expire = 0;
|
|
|
|
wake_up_interruptible(&khugepaged_wait);
|
|
|
|
|
|
|
|
return count;
|
|
|
|
}
|
|
|
|
static struct kobj_attribute scan_sleep_millisecs_attr =
|
|
|
|
__ATTR(scan_sleep_millisecs, 0644, scan_sleep_millisecs_show,
|
|
|
|
scan_sleep_millisecs_store);
|
|
|
|
|
|
|
|
static ssize_t alloc_sleep_millisecs_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
char *buf)
|
|
|
|
{
|
|
|
|
return sprintf(buf, "%u\n", khugepaged_alloc_sleep_millisecs);
|
|
|
|
}
|
|
|
|
|
|
|
|
static ssize_t alloc_sleep_millisecs_store(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
const char *buf, size_t count)
|
|
|
|
{
|
|
|
|
unsigned long msecs;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
err = kstrtoul(buf, 10, &msecs);
|
|
|
|
if (err || msecs > UINT_MAX)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
khugepaged_alloc_sleep_millisecs = msecs;
|
|
|
|
khugepaged_sleep_expire = 0;
|
|
|
|
wake_up_interruptible(&khugepaged_wait);
|
|
|
|
|
|
|
|
return count;
|
|
|
|
}
|
|
|
|
static struct kobj_attribute alloc_sleep_millisecs_attr =
|
|
|
|
__ATTR(alloc_sleep_millisecs, 0644, alloc_sleep_millisecs_show,
|
|
|
|
alloc_sleep_millisecs_store);
|
|
|
|
|
|
|
|
static ssize_t pages_to_scan_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
char *buf)
|
|
|
|
{
|
|
|
|
return sprintf(buf, "%u\n", khugepaged_pages_to_scan);
|
|
|
|
}
|
|
|
|
static ssize_t pages_to_scan_store(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
const char *buf, size_t count)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
unsigned long pages;
|
|
|
|
|
|
|
|
err = kstrtoul(buf, 10, &pages);
|
|
|
|
if (err || !pages || pages > UINT_MAX)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
khugepaged_pages_to_scan = pages;
|
|
|
|
|
|
|
|
return count;
|
|
|
|
}
|
|
|
|
static struct kobj_attribute pages_to_scan_attr =
|
|
|
|
__ATTR(pages_to_scan, 0644, pages_to_scan_show,
|
|
|
|
pages_to_scan_store);
|
|
|
|
|
|
|
|
static ssize_t pages_collapsed_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
char *buf)
|
|
|
|
{
|
|
|
|
return sprintf(buf, "%u\n", khugepaged_pages_collapsed);
|
|
|
|
}
|
|
|
|
static struct kobj_attribute pages_collapsed_attr =
|
|
|
|
__ATTR_RO(pages_collapsed);
|
|
|
|
|
|
|
|
static ssize_t full_scans_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
char *buf)
|
|
|
|
{
|
|
|
|
return sprintf(buf, "%u\n", khugepaged_full_scans);
|
|
|
|
}
|
|
|
|
static struct kobj_attribute full_scans_attr =
|
|
|
|
__ATTR_RO(full_scans);
|
|
|
|
|
|
|
|
static ssize_t khugepaged_defrag_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr, char *buf)
|
|
|
|
{
|
|
|
|
return single_hugepage_flag_show(kobj, attr, buf,
|
|
|
|
TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG);
|
|
|
|
}
|
|
|
|
static ssize_t khugepaged_defrag_store(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
const char *buf, size_t count)
|
|
|
|
{
|
|
|
|
return single_hugepage_flag_store(kobj, attr, buf, count,
|
|
|
|
TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG);
|
|
|
|
}
|
|
|
|
static struct kobj_attribute khugepaged_defrag_attr =
|
|
|
|
__ATTR(defrag, 0644, khugepaged_defrag_show,
|
|
|
|
khugepaged_defrag_store);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* max_ptes_none controls if khugepaged should collapse hugepages over
|
|
|
|
* any unmapped ptes in turn potentially increasing the memory
|
|
|
|
* footprint of the vmas. When max_ptes_none is 0 khugepaged will not
|
|
|
|
* reduce the available free memory in the system as it
|
|
|
|
* runs. Increasing max_ptes_none will instead potentially reduce the
|
|
|
|
* free memory in the system during the khugepaged scan.
|
|
|
|
*/
|
|
|
|
static ssize_t khugepaged_max_ptes_none_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
char *buf)
|
|
|
|
{
|
|
|
|
return sprintf(buf, "%u\n", khugepaged_max_ptes_none);
|
|
|
|
}
|
|
|
|
static ssize_t khugepaged_max_ptes_none_store(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
const char *buf, size_t count)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
unsigned long max_ptes_none;
|
|
|
|
|
|
|
|
err = kstrtoul(buf, 10, &max_ptes_none);
|
|
|
|
if (err || max_ptes_none > HPAGE_PMD_NR-1)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
khugepaged_max_ptes_none = max_ptes_none;
|
|
|
|
|
|
|
|
return count;
|
|
|
|
}
|
|
|
|
static struct kobj_attribute khugepaged_max_ptes_none_attr =
|
|
|
|
__ATTR(max_ptes_none, 0644, khugepaged_max_ptes_none_show,
|
|
|
|
khugepaged_max_ptes_none_store);
|
|
|
|
|
|
|
|
static ssize_t khugepaged_max_ptes_swap_show(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
char *buf)
|
|
|
|
{
|
|
|
|
return sprintf(buf, "%u\n", khugepaged_max_ptes_swap);
|
|
|
|
}
|
|
|
|
|
|
|
|
static ssize_t khugepaged_max_ptes_swap_store(struct kobject *kobj,
|
|
|
|
struct kobj_attribute *attr,
|
|
|
|
const char *buf, size_t count)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
unsigned long max_ptes_swap;
|
|
|
|
|
|
|
|
err = kstrtoul(buf, 10, &max_ptes_swap);
|
|
|
|
if (err || max_ptes_swap > HPAGE_PMD_NR-1)
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
khugepaged_max_ptes_swap = max_ptes_swap;
|
|
|
|
|
|
|
|
return count;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct kobj_attribute khugepaged_max_ptes_swap_attr =
|
|
|
|
__ATTR(max_ptes_swap, 0644, khugepaged_max_ptes_swap_show,
|
|
|
|
khugepaged_max_ptes_swap_store);
|
|
|
|
|
|
|
|
static struct attribute *khugepaged_attr[] = {
|
|
|
|
&khugepaged_defrag_attr.attr,
|
|
|
|
&khugepaged_max_ptes_none_attr.attr,
|
|
|
|
&pages_to_scan_attr.attr,
|
|
|
|
&pages_collapsed_attr.attr,
|
|
|
|
&full_scans_attr.attr,
|
|
|
|
&scan_sleep_millisecs_attr.attr,
|
|
|
|
&alloc_sleep_millisecs_attr.attr,
|
|
|
|
&khugepaged_max_ptes_swap_attr.attr,
|
|
|
|
NULL,
|
|
|
|
};
|
|
|
|
|
|
|
|
struct attribute_group khugepaged_attr_group = {
|
|
|
|
.attrs = khugepaged_attr,
|
|
|
|
.name = "khugepaged",
|
|
|
|
};
|
2016-11-30 16:54:02 -07:00
|
|
|
#endif /* CONFIG_SYSFS */
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
int hugepage_madvise(struct vm_area_struct *vma,
|
|
|
|
unsigned long *vm_flags, int advice)
|
|
|
|
{
|
|
|
|
switch (advice) {
|
|
|
|
case MADV_HUGEPAGE:
|
|
|
|
#ifdef CONFIG_S390
|
|
|
|
/*
|
|
|
|
* qemu blindly sets MADV_HUGEPAGE on all allocations, but s390
|
|
|
|
* can't handle this properly after s390_enable_sie, so we simply
|
|
|
|
* ignore the madvise to prevent qemu from causing a SIGSEGV.
|
|
|
|
*/
|
|
|
|
if (mm_has_pgste(vma->vm_mm))
|
|
|
|
return 0;
|
|
|
|
#endif
|
|
|
|
*vm_flags &= ~VM_NOHUGEPAGE;
|
|
|
|
*vm_flags |= VM_HUGEPAGE;
|
|
|
|
/*
|
|
|
|
* If the vma become good for khugepaged to scan,
|
|
|
|
* register it here without waiting a page fault that
|
|
|
|
* may not happen any time soon.
|
|
|
|
*/
|
|
|
|
if (!(*vm_flags & VM_NO_KHUGEPAGED) &&
|
|
|
|
khugepaged_enter_vma_merge(vma, *vm_flags))
|
|
|
|
return -ENOMEM;
|
|
|
|
break;
|
|
|
|
case MADV_NOHUGEPAGE:
|
|
|
|
*vm_flags &= ~VM_HUGEPAGE;
|
|
|
|
*vm_flags |= VM_NOHUGEPAGE;
|
|
|
|
/*
|
|
|
|
* Setting VM_NOHUGEPAGE will prevent khugepaged from scanning
|
|
|
|
* this vma even if we leave the mm registered in khugepaged if
|
|
|
|
* it got registered before VM_NOHUGEPAGE was set.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int __init khugepaged_init(void)
|
|
|
|
{
|
|
|
|
mm_slot_cache = kmem_cache_create("khugepaged_mm_slot",
|
|
|
|
sizeof(struct mm_slot),
|
|
|
|
__alignof__(struct mm_slot), 0, NULL);
|
|
|
|
if (!mm_slot_cache)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
khugepaged_pages_to_scan = HPAGE_PMD_NR * 8;
|
|
|
|
khugepaged_max_ptes_none = HPAGE_PMD_NR - 1;
|
|
|
|
khugepaged_max_ptes_swap = HPAGE_PMD_NR / 8;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
void __init khugepaged_destroy(void)
|
|
|
|
{
|
|
|
|
kmem_cache_destroy(mm_slot_cache);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline struct mm_slot *alloc_mm_slot(void)
|
|
|
|
{
|
|
|
|
if (!mm_slot_cache) /* initialization failed */
|
|
|
|
return NULL;
|
|
|
|
return kmem_cache_zalloc(mm_slot_cache, GFP_KERNEL);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void free_mm_slot(struct mm_slot *mm_slot)
|
|
|
|
{
|
|
|
|
kmem_cache_free(mm_slot_cache, mm_slot);
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct mm_slot *get_mm_slot(struct mm_struct *mm)
|
|
|
|
{
|
|
|
|
struct mm_slot *mm_slot;
|
|
|
|
|
|
|
|
hash_for_each_possible(mm_slots_hash, mm_slot, hash, (unsigned long)mm)
|
|
|
|
if (mm == mm_slot->mm)
|
|
|
|
return mm_slot;
|
|
|
|
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void insert_to_mm_slots_hash(struct mm_struct *mm,
|
|
|
|
struct mm_slot *mm_slot)
|
|
|
|
{
|
|
|
|
mm_slot->mm = mm;
|
|
|
|
hash_add(mm_slots_hash, &mm_slot->hash, (long)mm);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int khugepaged_test_exit(struct mm_struct *mm)
|
|
|
|
{
|
|
|
|
return atomic_read(&mm->mm_users) == 0;
|
|
|
|
}
|
|
|
|
|
mm: thp: pass correct vm_flags to hugepage_vma_check()
khugepaged_enter_vma_merge() passes a stale vma->vm_flags to
hugepage_vma_check(). The argument vm_flags contains the latest value.
Therefore, it is necessary to pass this vm_flags into
hugepage_vma_check().
With this bug, madvise(MADV_HUGEPAGE) for mmap files in shmem fails to
put memory in huge pages. Here is an example of failed madvise():
/* mount /dev/shm with huge=advise:
* mount -o remount,huge=advise /dev/shm */
/* create file /dev/shm/huge */
#define HUGE_FILE "/dev/shm/huge"
fd = open(HUGE_FILE, O_RDONLY);
ptr = mmap(NULL, FILE_SIZE, PROT_READ, MAP_PRIVATE, fd, 0);
ret = madvise(ptr, FILE_SIZE, MADV_HUGEPAGE);
madvise() will return 0, but this memory region is never put in huge
page (check from /proc/meminfo: ShmemHugePages).
Link: http://lkml.kernel.org/r/20180629181752.792831-1-songliubraving@fb.com
Fixes: 02b75dc8160d ("mm: thp: register mm for khugepaged when merging vma for shmem")
Signed-off-by: Song Liu <songliubraving@fb.com>
Reviewed-by: Rik van Riel <riel@surriel.com>
Reviewed-by: Yang Shi <yang.shi@linux.alibaba.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-17 16:47:00 -06:00
|
|
|
static bool hugepage_vma_check(struct vm_area_struct *vma,
|
|
|
|
unsigned long vm_flags)
|
2018-08-17 16:45:26 -06:00
|
|
|
{
|
mm: thp: pass correct vm_flags to hugepage_vma_check()
khugepaged_enter_vma_merge() passes a stale vma->vm_flags to
hugepage_vma_check(). The argument vm_flags contains the latest value.
Therefore, it is necessary to pass this vm_flags into
hugepage_vma_check().
With this bug, madvise(MADV_HUGEPAGE) for mmap files in shmem fails to
put memory in huge pages. Here is an example of failed madvise():
/* mount /dev/shm with huge=advise:
* mount -o remount,huge=advise /dev/shm */
/* create file /dev/shm/huge */
#define HUGE_FILE "/dev/shm/huge"
fd = open(HUGE_FILE, O_RDONLY);
ptr = mmap(NULL, FILE_SIZE, PROT_READ, MAP_PRIVATE, fd, 0);
ret = madvise(ptr, FILE_SIZE, MADV_HUGEPAGE);
madvise() will return 0, but this memory region is never put in huge
page (check from /proc/meminfo: ShmemHugePages).
Link: http://lkml.kernel.org/r/20180629181752.792831-1-songliubraving@fb.com
Fixes: 02b75dc8160d ("mm: thp: register mm for khugepaged when merging vma for shmem")
Signed-off-by: Song Liu <songliubraving@fb.com>
Reviewed-by: Rik van Riel <riel@surriel.com>
Reviewed-by: Yang Shi <yang.shi@linux.alibaba.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-17 16:47:00 -06:00
|
|
|
if ((!(vm_flags & VM_HUGEPAGE) && !khugepaged_always()) ||
|
|
|
|
(vm_flags & VM_NOHUGEPAGE) ||
|
2018-08-17 16:45:26 -06:00
|
|
|
test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags))
|
|
|
|
return false;
|
2019-09-23 16:38:00 -06:00
|
|
|
|
|
|
|
if (shmem_file(vma->vm_file) ||
|
|
|
|
(IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) &&
|
|
|
|
vma->vm_file &&
|
|
|
|
(vm_flags & VM_DENYWRITE))) {
|
2018-08-17 16:45:26 -06:00
|
|
|
if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE))
|
|
|
|
return false;
|
|
|
|
return IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) - vma->vm_pgoff,
|
|
|
|
HPAGE_PMD_NR);
|
|
|
|
}
|
|
|
|
if (!vma->anon_vma || vma->vm_ops)
|
|
|
|
return false;
|
|
|
|
if (is_vma_temporary_stack(vma))
|
|
|
|
return false;
|
mm: thp: pass correct vm_flags to hugepage_vma_check()
khugepaged_enter_vma_merge() passes a stale vma->vm_flags to
hugepage_vma_check(). The argument vm_flags contains the latest value.
Therefore, it is necessary to pass this vm_flags into
hugepage_vma_check().
With this bug, madvise(MADV_HUGEPAGE) for mmap files in shmem fails to
put memory in huge pages. Here is an example of failed madvise():
/* mount /dev/shm with huge=advise:
* mount -o remount,huge=advise /dev/shm */
/* create file /dev/shm/huge */
#define HUGE_FILE "/dev/shm/huge"
fd = open(HUGE_FILE, O_RDONLY);
ptr = mmap(NULL, FILE_SIZE, PROT_READ, MAP_PRIVATE, fd, 0);
ret = madvise(ptr, FILE_SIZE, MADV_HUGEPAGE);
madvise() will return 0, but this memory region is never put in huge
page (check from /proc/meminfo: ShmemHugePages).
Link: http://lkml.kernel.org/r/20180629181752.792831-1-songliubraving@fb.com
Fixes: 02b75dc8160d ("mm: thp: register mm for khugepaged when merging vma for shmem")
Signed-off-by: Song Liu <songliubraving@fb.com>
Reviewed-by: Rik van Riel <riel@surriel.com>
Reviewed-by: Yang Shi <yang.shi@linux.alibaba.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-17 16:47:00 -06:00
|
|
|
return !(vm_flags & VM_NO_KHUGEPAGED);
|
2018-08-17 16:45:26 -06:00
|
|
|
}
|
|
|
|
|
2016-07-26 16:26:24 -06:00
|
|
|
int __khugepaged_enter(struct mm_struct *mm)
|
|
|
|
{
|
|
|
|
struct mm_slot *mm_slot;
|
|
|
|
int wakeup;
|
|
|
|
|
|
|
|
mm_slot = alloc_mm_slot();
|
|
|
|
if (!mm_slot)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
/* __khugepaged_exit() must not run from under us */
|
|
|
|
VM_BUG_ON_MM(khugepaged_test_exit(mm), mm);
|
|
|
|
if (unlikely(test_and_set_bit(MMF_VM_HUGEPAGE, &mm->flags))) {
|
|
|
|
free_mm_slot(mm_slot);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
spin_lock(&khugepaged_mm_lock);
|
|
|
|
insert_to_mm_slots_hash(mm, mm_slot);
|
|
|
|
/*
|
|
|
|
* Insert just behind the scanning cursor, to let the area settle
|
|
|
|
* down a little.
|
|
|
|
*/
|
|
|
|
wakeup = list_empty(&khugepaged_scan.mm_head);
|
|
|
|
list_add_tail(&mm_slot->mm_node, &khugepaged_scan.mm_head);
|
|
|
|
spin_unlock(&khugepaged_mm_lock);
|
|
|
|
|
2017-02-27 15:30:07 -07:00
|
|
|
mmgrab(mm);
|
2016-07-26 16:26:24 -06:00
|
|
|
if (wakeup)
|
|
|
|
wake_up_interruptible(&khugepaged_wait);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int khugepaged_enter_vma_merge(struct vm_area_struct *vma,
|
|
|
|
unsigned long vm_flags)
|
|
|
|
{
|
|
|
|
unsigned long hstart, hend;
|
2018-08-17 16:45:26 -06:00
|
|
|
|
|
|
|
/*
|
2019-09-23 16:38:00 -06:00
|
|
|
* khugepaged only supports read-only files for non-shmem files.
|
|
|
|
* khugepaged does not yet work on special mappings. And
|
|
|
|
* file-private shmem THP is not supported.
|
2018-08-17 16:45:26 -06:00
|
|
|
*/
|
mm: thp: pass correct vm_flags to hugepage_vma_check()
khugepaged_enter_vma_merge() passes a stale vma->vm_flags to
hugepage_vma_check(). The argument vm_flags contains the latest value.
Therefore, it is necessary to pass this vm_flags into
hugepage_vma_check().
With this bug, madvise(MADV_HUGEPAGE) for mmap files in shmem fails to
put memory in huge pages. Here is an example of failed madvise():
/* mount /dev/shm with huge=advise:
* mount -o remount,huge=advise /dev/shm */
/* create file /dev/shm/huge */
#define HUGE_FILE "/dev/shm/huge"
fd = open(HUGE_FILE, O_RDONLY);
ptr = mmap(NULL, FILE_SIZE, PROT_READ, MAP_PRIVATE, fd, 0);
ret = madvise(ptr, FILE_SIZE, MADV_HUGEPAGE);
madvise() will return 0, but this memory region is never put in huge
page (check from /proc/meminfo: ShmemHugePages).
Link: http://lkml.kernel.org/r/20180629181752.792831-1-songliubraving@fb.com
Fixes: 02b75dc8160d ("mm: thp: register mm for khugepaged when merging vma for shmem")
Signed-off-by: Song Liu <songliubraving@fb.com>
Reviewed-by: Rik van Riel <riel@surriel.com>
Reviewed-by: Yang Shi <yang.shi@linux.alibaba.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-17 16:47:00 -06:00
|
|
|
if (!hugepage_vma_check(vma, vm_flags))
|
2016-07-26 16:26:24 -06:00
|
|
|
return 0;
|
2018-08-17 16:45:26 -06:00
|
|
|
|
2016-07-26 16:26:24 -06:00
|
|
|
hstart = (vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK;
|
|
|
|
hend = vma->vm_end & HPAGE_PMD_MASK;
|
|
|
|
if (hstart < hend)
|
|
|
|
return khugepaged_enter(vma, vm_flags);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
void __khugepaged_exit(struct mm_struct *mm)
|
|
|
|
{
|
|
|
|
struct mm_slot *mm_slot;
|
|
|
|
int free = 0;
|
|
|
|
|
|
|
|
spin_lock(&khugepaged_mm_lock);
|
|
|
|
mm_slot = get_mm_slot(mm);
|
|
|
|
if (mm_slot && khugepaged_scan.mm_slot != mm_slot) {
|
|
|
|
hash_del(&mm_slot->hash);
|
|
|
|
list_del(&mm_slot->mm_node);
|
|
|
|
free = 1;
|
|
|
|
}
|
|
|
|
spin_unlock(&khugepaged_mm_lock);
|
|
|
|
|
|
|
|
if (free) {
|
|
|
|
clear_bit(MMF_VM_HUGEPAGE, &mm->flags);
|
|
|
|
free_mm_slot(mm_slot);
|
|
|
|
mmdrop(mm);
|
|
|
|
} else if (mm_slot) {
|
|
|
|
/*
|
|
|
|
* This is required to serialize against
|
|
|
|
* khugepaged_test_exit() (which is guaranteed to run
|
|
|
|
* under mmap sem read mode). Stop here (after we
|
|
|
|
* return all pagetables will be destroyed) until
|
|
|
|
* khugepaged has finished working on the pagetables
|
|
|
|
* under the mmap_sem.
|
|
|
|
*/
|
|
|
|
down_write(&mm->mmap_sem);
|
|
|
|
up_write(&mm->mmap_sem);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void release_pte_page(struct page *page)
|
|
|
|
{
|
2017-05-03 15:52:26 -06:00
|
|
|
dec_node_page_state(page, NR_ISOLATED_ANON + page_is_file_cache(page));
|
2016-07-26 16:26:24 -06:00
|
|
|
unlock_page(page);
|
|
|
|
putback_lru_page(page);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void release_pte_pages(pte_t *pte, pte_t *_pte)
|
|
|
|
{
|
|
|
|
while (--_pte >= pte) {
|
|
|
|
pte_t pteval = *_pte;
|
|
|
|
if (!pte_none(pteval) && !is_zero_pfn(pte_pfn(pteval)))
|
|
|
|
release_pte_page(pte_page(pteval));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
|
|
|
|
unsigned long address,
|
|
|
|
pte_t *pte)
|
|
|
|
{
|
|
|
|
struct page *page = NULL;
|
|
|
|
pte_t *_pte;
|
2016-07-26 16:26:46 -06:00
|
|
|
int none_or_zero = 0, result = 0, referenced = 0;
|
|
|
|
bool writable = false;
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
for (_pte = pte; _pte < pte+HPAGE_PMD_NR;
|
|
|
|
_pte++, address += PAGE_SIZE) {
|
|
|
|
pte_t pteval = *_pte;
|
|
|
|
if (pte_none(pteval) || (pte_present(pteval) &&
|
|
|
|
is_zero_pfn(pte_pfn(pteval)))) {
|
|
|
|
if (!userfaultfd_armed(vma) &&
|
|
|
|
++none_or_zero <= khugepaged_max_ptes_none) {
|
|
|
|
continue;
|
|
|
|
} else {
|
|
|
|
result = SCAN_EXCEED_NONE_PTE;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!pte_present(pteval)) {
|
|
|
|
result = SCAN_PTE_NON_PRESENT;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
page = vm_normal_page(vma, address, pteval);
|
|
|
|
if (unlikely(!page)) {
|
|
|
|
result = SCAN_PAGE_NULL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2018-03-22 17:17:28 -06:00
|
|
|
/* TODO: teach khugepaged to collapse THP mapped with pte */
|
|
|
|
if (PageCompound(page)) {
|
|
|
|
result = SCAN_PAGE_COMPOUND;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2016-07-26 16:26:24 -06:00
|
|
|
VM_BUG_ON_PAGE(!PageAnon(page), page);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We can do it before isolate_lru_page because the
|
|
|
|
* page can't be freed from under us. NOTE: PG_lock
|
|
|
|
* is needed to serialize against split_huge_page
|
|
|
|
* when invoked from the VM.
|
|
|
|
*/
|
|
|
|
if (!trylock_page(page)) {
|
|
|
|
result = SCAN_PAGE_LOCK;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* cannot use mapcount: can't collapse if there's a gup pin.
|
|
|
|
* The page must only be referenced by the scanned process
|
|
|
|
* and page swap cache.
|
|
|
|
*/
|
2017-05-03 15:53:35 -06:00
|
|
|
if (page_count(page) != 1 + PageSwapCache(page)) {
|
2016-07-26 16:26:24 -06:00
|
|
|
unlock_page(page);
|
|
|
|
result = SCAN_PAGE_COUNT;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
if (pte_write(pteval)) {
|
|
|
|
writable = true;
|
|
|
|
} else {
|
|
|
|
if (PageSwapCache(page) &&
|
|
|
|
!reuse_swap_page(page, NULL)) {
|
|
|
|
unlock_page(page);
|
|
|
|
result = SCAN_SWAP_CACHE_PAGE;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* Page is not in the swap cache. It can be collapsed
|
|
|
|
* into a THP.
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Isolate the page to avoid collapsing an hugepage
|
|
|
|
* currently in use by the VM.
|
|
|
|
*/
|
|
|
|
if (isolate_lru_page(page)) {
|
|
|
|
unlock_page(page);
|
|
|
|
result = SCAN_DEL_PAGE_LRU;
|
|
|
|
goto out;
|
|
|
|
}
|
2017-05-03 15:52:26 -06:00
|
|
|
inc_node_page_state(page,
|
|
|
|
NR_ISOLATED_ANON + page_is_file_cache(page));
|
2016-07-26 16:26:24 -06:00
|
|
|
VM_BUG_ON_PAGE(!PageLocked(page), page);
|
|
|
|
VM_BUG_ON_PAGE(PageLRU(page), page);
|
|
|
|
|
2016-07-26 16:26:46 -06:00
|
|
|
/* There should be enough young pte to collapse the page */
|
2016-07-26 16:26:24 -06:00
|
|
|
if (pte_young(pteval) ||
|
|
|
|
page_is_young(page) || PageReferenced(page) ||
|
|
|
|
mmu_notifier_test_young(vma->vm_mm, address))
|
2016-07-26 16:26:46 -06:00
|
|
|
referenced++;
|
2016-07-26 16:26:24 -06:00
|
|
|
}
|
|
|
|
if (likely(writable)) {
|
|
|
|
if (likely(referenced)) {
|
|
|
|
result = SCAN_SUCCEED;
|
|
|
|
trace_mm_collapse_huge_page_isolate(page, none_or_zero,
|
|
|
|
referenced, writable, result);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
result = SCAN_PAGE_RO;
|
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
|
|
|
release_pte_pages(pte, _pte);
|
|
|
|
trace_mm_collapse_huge_page_isolate(page, none_or_zero,
|
|
|
|
referenced, writable, result);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __collapse_huge_page_copy(pte_t *pte, struct page *page,
|
|
|
|
struct vm_area_struct *vma,
|
|
|
|
unsigned long address,
|
|
|
|
spinlock_t *ptl)
|
|
|
|
{
|
|
|
|
pte_t *_pte;
|
2017-05-12 16:47:03 -06:00
|
|
|
for (_pte = pte; _pte < pte + HPAGE_PMD_NR;
|
|
|
|
_pte++, page++, address += PAGE_SIZE) {
|
2016-07-26 16:26:24 -06:00
|
|
|
pte_t pteval = *_pte;
|
|
|
|
struct page *src_page;
|
|
|
|
|
|
|
|
if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
|
|
|
|
clear_user_highpage(page, address);
|
|
|
|
add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1);
|
|
|
|
if (is_zero_pfn(pte_pfn(pteval))) {
|
|
|
|
/*
|
|
|
|
* ptl mostly unnecessary.
|
|
|
|
*/
|
|
|
|
spin_lock(ptl);
|
|
|
|
/*
|
|
|
|
* paravirt calls inside pte_clear here are
|
|
|
|
* superfluous.
|
|
|
|
*/
|
|
|
|
pte_clear(vma->vm_mm, address, _pte);
|
|
|
|
spin_unlock(ptl);
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
src_page = pte_page(pteval);
|
|
|
|
copy_user_highpage(page, src_page, address, vma);
|
|
|
|
VM_BUG_ON_PAGE(page_mapcount(src_page) != 1, src_page);
|
|
|
|
release_pte_page(src_page);
|
|
|
|
/*
|
|
|
|
* ptl mostly unnecessary, but preempt has to
|
|
|
|
* be disabled to update the per-cpu stats
|
|
|
|
* inside page_remove_rmap().
|
|
|
|
*/
|
|
|
|
spin_lock(ptl);
|
|
|
|
/*
|
|
|
|
* paravirt calls inside pte_clear here are
|
|
|
|
* superfluous.
|
|
|
|
*/
|
|
|
|
pte_clear(vma->vm_mm, address, _pte);
|
|
|
|
page_remove_rmap(src_page, false);
|
|
|
|
spin_unlock(ptl);
|
|
|
|
free_page_and_swap_cache(src_page);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void khugepaged_alloc_sleep(void)
|
|
|
|
{
|
|
|
|
DEFINE_WAIT(wait);
|
|
|
|
|
|
|
|
add_wait_queue(&khugepaged_wait, &wait);
|
|
|
|
freezable_schedule_timeout_interruptible(
|
|
|
|
msecs_to_jiffies(khugepaged_alloc_sleep_millisecs));
|
|
|
|
remove_wait_queue(&khugepaged_wait, &wait);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int khugepaged_node_load[MAX_NUMNODES];
|
|
|
|
|
|
|
|
static bool khugepaged_scan_abort(int nid)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/*
|
2016-07-28 16:46:32 -06:00
|
|
|
* If node_reclaim_mode is disabled, then no extra effort is made to
|
2016-07-26 16:26:24 -06:00
|
|
|
* allocate memory locally.
|
|
|
|
*/
|
2016-07-28 16:46:32 -06:00
|
|
|
if (!node_reclaim_mode)
|
2016-07-26 16:26:24 -06:00
|
|
|
return false;
|
|
|
|
|
|
|
|
/* If there is a count for this node already, it must be acceptable */
|
|
|
|
if (khugepaged_node_load[nid])
|
|
|
|
return false;
|
|
|
|
|
|
|
|
for (i = 0; i < MAX_NUMNODES; i++) {
|
|
|
|
if (!khugepaged_node_load[i])
|
|
|
|
continue;
|
2019-08-08 13:53:01 -06:00
|
|
|
if (node_distance(nid, i) > node_reclaim_distance)
|
2016-07-26 16:26:24 -06:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Defrag for khugepaged will enter direct reclaim/compaction if necessary */
|
|
|
|
static inline gfp_t alloc_hugepage_khugepaged_gfpmask(void)
|
|
|
|
{
|
mm, thp: remove __GFP_NORETRY from khugepaged and madvised allocations
After the previous patch, we can distinguish costly allocations that
should be really lightweight, such as THP page faults, with
__GFP_NORETRY. This means we don't need to recognize khugepaged
allocations via PF_KTHREAD anymore. We can also change THP page faults
in areas where madvise(MADV_HUGEPAGE) was used to try as hard as
khugepaged, as the process has indicated that it benefits from THP's and
is willing to pay some initial latency costs.
We can also make the flags handling less cryptic by distinguishing
GFP_TRANSHUGE_LIGHT (no reclaim at all, default mode in page fault) from
GFP_TRANSHUGE (only direct reclaim, khugepaged default). Adding
__GFP_NORETRY or __GFP_KSWAPD_RECLAIM is done where needed.
The patch effectively changes the current GFP_TRANSHUGE users as
follows:
* get_huge_zero_page() - the zero page lifetime should be relatively
long and it's shared by multiple users, so it's worth spending some
effort on it. We use GFP_TRANSHUGE, and __GFP_NORETRY is not added.
This also restores direct reclaim to this allocation, which was
unintentionally removed by commit e4a49efe4e7e ("mm: thp: set THP defrag
by default to madvise and add a stall-free defrag option")
* alloc_hugepage_khugepaged_gfpmask() - this is khugepaged, so latency
is not an issue. So if khugepaged "defrag" is enabled (the default), do
reclaim via GFP_TRANSHUGE without __GFP_NORETRY. We can remove the
PF_KTHREAD check from page alloc.
As a side-effect, khugepaged will now no longer check if the initial
compaction was deferred or contended. This is OK, as khugepaged sleep
times between collapsion attempts are long enough to prevent noticeable
disruption, so we should allow it to spend some effort.
* migrate_misplaced_transhuge_page() - already was masking out
__GFP_RECLAIM, so just convert to GFP_TRANSHUGE_LIGHT which is
equivalent.
* alloc_hugepage_direct_gfpmask() - vma's with VM_HUGEPAGE (via madvise)
are now allocating without __GFP_NORETRY. Other vma's keep using
__GFP_NORETRY if direct reclaim/compaction is at all allowed (by default
it's allowed only for madvised vma's). The rest is conversion to
GFP_TRANSHUGE(_LIGHT).
[mhocko@suse.com: suggested GFP_TRANSHUGE_LIGHT]
Link: http://lkml.kernel.org/r/20160721073614.24395-7-vbabka@suse.cz
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Michal Hocko <mhocko@suse.com>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-07-28 16:49:25 -06:00
|
|
|
return khugepaged_defrag() ? GFP_TRANSHUGE : GFP_TRANSHUGE_LIGHT;
|
2016-07-26 16:26:24 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
static int khugepaged_find_target_node(void)
|
|
|
|
{
|
|
|
|
static int last_khugepaged_target_node = NUMA_NO_NODE;
|
|
|
|
int nid, target_node = 0, max_value = 0;
|
|
|
|
|
|
|
|
/* find first node with max normal pages hit */
|
|
|
|
for (nid = 0; nid < MAX_NUMNODES; nid++)
|
|
|
|
if (khugepaged_node_load[nid] > max_value) {
|
|
|
|
max_value = khugepaged_node_load[nid];
|
|
|
|
target_node = nid;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* do some balance if several nodes have the same hit record */
|
|
|
|
if (target_node <= last_khugepaged_target_node)
|
|
|
|
for (nid = last_khugepaged_target_node + 1; nid < MAX_NUMNODES;
|
|
|
|
nid++)
|
|
|
|
if (max_value == khugepaged_node_load[nid]) {
|
|
|
|
target_node = nid;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
last_khugepaged_target_node = target_node;
|
|
|
|
return target_node;
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool khugepaged_prealloc_page(struct page **hpage, bool *wait)
|
|
|
|
{
|
|
|
|
if (IS_ERR(*hpage)) {
|
|
|
|
if (!*wait)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
*wait = false;
|
|
|
|
*hpage = NULL;
|
|
|
|
khugepaged_alloc_sleep();
|
|
|
|
} else if (*hpage) {
|
|
|
|
put_page(*hpage);
|
|
|
|
*hpage = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct page *
|
2016-07-26 16:26:26 -06:00
|
|
|
khugepaged_alloc_page(struct page **hpage, gfp_t gfp, int node)
|
2016-07-26 16:26:24 -06:00
|
|
|
{
|
|
|
|
VM_BUG_ON_PAGE(*hpage, *hpage);
|
|
|
|
|
|
|
|
*hpage = __alloc_pages_node(node, gfp, HPAGE_PMD_ORDER);
|
|
|
|
if (unlikely(!*hpage)) {
|
|
|
|
count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
|
|
|
|
*hpage = ERR_PTR(-ENOMEM);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
prep_transhuge_page(*hpage);
|
|
|
|
count_vm_event(THP_COLLAPSE_ALLOC);
|
|
|
|
return *hpage;
|
|
|
|
}
|
|
|
|
#else
|
|
|
|
static int khugepaged_find_target_node(void)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline struct page *alloc_khugepaged_hugepage(void)
|
|
|
|
{
|
|
|
|
struct page *page;
|
|
|
|
|
|
|
|
page = alloc_pages(alloc_hugepage_khugepaged_gfpmask(),
|
|
|
|
HPAGE_PMD_ORDER);
|
|
|
|
if (page)
|
|
|
|
prep_transhuge_page(page);
|
|
|
|
return page;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct page *khugepaged_alloc_hugepage(bool *wait)
|
|
|
|
{
|
|
|
|
struct page *hpage;
|
|
|
|
|
|
|
|
do {
|
|
|
|
hpage = alloc_khugepaged_hugepage();
|
|
|
|
if (!hpage) {
|
|
|
|
count_vm_event(THP_COLLAPSE_ALLOC_FAILED);
|
|
|
|
if (!*wait)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
*wait = false;
|
|
|
|
khugepaged_alloc_sleep();
|
|
|
|
} else
|
|
|
|
count_vm_event(THP_COLLAPSE_ALLOC);
|
|
|
|
} while (unlikely(!hpage) && likely(khugepaged_enabled()));
|
|
|
|
|
|
|
|
return hpage;
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool khugepaged_prealloc_page(struct page **hpage, bool *wait)
|
|
|
|
{
|
|
|
|
if (!*hpage)
|
|
|
|
*hpage = khugepaged_alloc_hugepage(wait);
|
|
|
|
|
|
|
|
if (unlikely(!*hpage))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct page *
|
2016-07-26 16:26:26 -06:00
|
|
|
khugepaged_alloc_page(struct page **hpage, gfp_t gfp, int node)
|
2016-07-26 16:26:24 -06:00
|
|
|
{
|
|
|
|
VM_BUG_ON(!*hpage);
|
|
|
|
|
|
|
|
return *hpage;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If mmap_sem temporarily dropped, revalidate vma
|
|
|
|
* before taking mmap_sem.
|
|
|
|
* Return 0 if succeeds, otherwise return none-zero
|
|
|
|
* value (scan code).
|
|
|
|
*/
|
|
|
|
|
2016-09-19 15:44:01 -06:00
|
|
|
static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,
|
|
|
|
struct vm_area_struct **vmap)
|
2016-07-26 16:26:24 -06:00
|
|
|
{
|
|
|
|
struct vm_area_struct *vma;
|
|
|
|
unsigned long hstart, hend;
|
|
|
|
|
|
|
|
if (unlikely(khugepaged_test_exit(mm)))
|
|
|
|
return SCAN_ANY_PROCESS;
|
|
|
|
|
2016-09-19 15:44:01 -06:00
|
|
|
*vmap = vma = find_vma(mm, address);
|
2016-07-26 16:26:24 -06:00
|
|
|
if (!vma)
|
|
|
|
return SCAN_VMA_NULL;
|
|
|
|
|
|
|
|
hstart = (vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK;
|
|
|
|
hend = vma->vm_end & HPAGE_PMD_MASK;
|
|
|
|
if (address < hstart || address + HPAGE_PMD_SIZE > hend)
|
|
|
|
return SCAN_ADDRESS_RANGE;
|
mm: thp: pass correct vm_flags to hugepage_vma_check()
khugepaged_enter_vma_merge() passes a stale vma->vm_flags to
hugepage_vma_check(). The argument vm_flags contains the latest value.
Therefore, it is necessary to pass this vm_flags into
hugepage_vma_check().
With this bug, madvise(MADV_HUGEPAGE) for mmap files in shmem fails to
put memory in huge pages. Here is an example of failed madvise():
/* mount /dev/shm with huge=advise:
* mount -o remount,huge=advise /dev/shm */
/* create file /dev/shm/huge */
#define HUGE_FILE "/dev/shm/huge"
fd = open(HUGE_FILE, O_RDONLY);
ptr = mmap(NULL, FILE_SIZE, PROT_READ, MAP_PRIVATE, fd, 0);
ret = madvise(ptr, FILE_SIZE, MADV_HUGEPAGE);
madvise() will return 0, but this memory region is never put in huge
page (check from /proc/meminfo: ShmemHugePages).
Link: http://lkml.kernel.org/r/20180629181752.792831-1-songliubraving@fb.com
Fixes: 02b75dc8160d ("mm: thp: register mm for khugepaged when merging vma for shmem")
Signed-off-by: Song Liu <songliubraving@fb.com>
Reviewed-by: Rik van Riel <riel@surriel.com>
Reviewed-by: Yang Shi <yang.shi@linux.alibaba.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-17 16:47:00 -06:00
|
|
|
if (!hugepage_vma_check(vma, vma->vm_flags))
|
2016-07-26 16:26:24 -06:00
|
|
|
return SCAN_VMA_CHECK;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Bring missing pages in from swap, to complete THP collapse.
|
|
|
|
* Only done if khugepaged_scan_pmd believes it is worthwhile.
|
|
|
|
*
|
|
|
|
* Called and returns without pte mapped or spinlocks held,
|
|
|
|
* but with mmap_sem held to protect against vma changes.
|
|
|
|
*/
|
|
|
|
|
|
|
|
static bool __collapse_huge_page_swapin(struct mm_struct *mm,
|
|
|
|
struct vm_area_struct *vma,
|
2016-07-26 16:26:46 -06:00
|
|
|
unsigned long address, pmd_t *pmd,
|
|
|
|
int referenced)
|
2016-07-26 16:26:24 -06:00
|
|
|
{
|
2018-08-23 18:01:36 -06:00
|
|
|
int swapped_in = 0;
|
|
|
|
vm_fault_t ret = 0;
|
2016-12-14 16:06:58 -07:00
|
|
|
struct vm_fault vmf = {
|
2016-07-26 16:26:24 -06:00
|
|
|
.vma = vma,
|
|
|
|
.address = address,
|
|
|
|
.flags = FAULT_FLAG_ALLOW_RETRY,
|
|
|
|
.pmd = pmd,
|
2016-12-14 16:07:04 -07:00
|
|
|
.pgoff = linear_page_index(vma, address),
|
2016-07-26 16:26:24 -06:00
|
|
|
};
|
|
|
|
|
2016-09-19 15:44:04 -06:00
|
|
|
/* we only decide to swapin, if there is enough young ptes */
|
|
|
|
if (referenced < HPAGE_PMD_NR/2) {
|
|
|
|
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
|
|
|
|
return false;
|
|
|
|
}
|
2016-12-14 16:06:58 -07:00
|
|
|
vmf.pte = pte_offset_map(pmd, address);
|
|
|
|
for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
|
|
|
|
vmf.pte++, vmf.address += PAGE_SIZE) {
|
2016-12-14 16:07:16 -07:00
|
|
|
vmf.orig_pte = *vmf.pte;
|
|
|
|
if (!is_swap_pte(vmf.orig_pte))
|
2016-07-26 16:26:24 -06:00
|
|
|
continue;
|
|
|
|
swapped_in++;
|
2016-12-14 16:07:16 -07:00
|
|
|
ret = do_swap_page(&vmf);
|
2016-07-26 16:26:46 -06:00
|
|
|
|
2016-07-26 16:26:24 -06:00
|
|
|
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
|
|
|
|
if (ret & VM_FAULT_RETRY) {
|
|
|
|
down_read(&mm->mmap_sem);
|
2016-12-14 16:06:58 -07:00
|
|
|
if (hugepage_vma_revalidate(mm, address, &vmf.vma)) {
|
2016-07-26 16:26:43 -06:00
|
|
|
/* vma is no longer available, don't continue to swapin */
|
2016-07-26 16:26:46 -06:00
|
|
|
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
|
2016-07-26 16:26:24 -06:00
|
|
|
return false;
|
2016-07-26 16:26:43 -06:00
|
|
|
}
|
2016-07-26 16:26:24 -06:00
|
|
|
/* check if the pmd is still valid */
|
2017-05-12 16:46:38 -06:00
|
|
|
if (mm_find_pmd(mm, address) != pmd) {
|
|
|
|
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
|
2016-07-26 16:26:24 -06:00
|
|
|
return false;
|
2017-05-12 16:46:38 -06:00
|
|
|
}
|
2016-07-26 16:26:24 -06:00
|
|
|
}
|
|
|
|
if (ret & VM_FAULT_ERROR) {
|
2016-07-26 16:26:46 -06:00
|
|
|
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
|
2016-07-26 16:26:24 -06:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
/* pte is unmapped now, we need to map it */
|
2016-12-14 16:06:58 -07:00
|
|
|
vmf.pte = pte_offset_map(pmd, vmf.address);
|
2016-07-26 16:26:24 -06:00
|
|
|
}
|
2016-12-14 16:06:58 -07:00
|
|
|
vmf.pte--;
|
|
|
|
pte_unmap(vmf.pte);
|
2016-07-26 16:26:46 -06:00
|
|
|
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 1);
|
2016-07-26 16:26:24 -06:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void collapse_huge_page(struct mm_struct *mm,
|
|
|
|
unsigned long address,
|
|
|
|
struct page **hpage,
|
2016-07-26 16:26:46 -06:00
|
|
|
int node, int referenced)
|
2016-07-26 16:26:24 -06:00
|
|
|
{
|
|
|
|
pmd_t *pmd, _pmd;
|
|
|
|
pte_t *pte;
|
|
|
|
pgtable_t pgtable;
|
|
|
|
struct page *new_page;
|
|
|
|
spinlock_t *pmd_ptl, *pte_ptl;
|
|
|
|
int isolated = 0, result = 0;
|
|
|
|
struct mem_cgroup *memcg;
|
2016-09-19 15:44:01 -06:00
|
|
|
struct vm_area_struct *vma;
|
2018-12-28 01:38:09 -07:00
|
|
|
struct mmu_notifier_range range;
|
2016-07-26 16:26:24 -06:00
|
|
|
gfp_t gfp;
|
|
|
|
|
|
|
|
VM_BUG_ON(address & ~HPAGE_PMD_MASK);
|
|
|
|
|
|
|
|
/* Only allocate from the target node */
|
2017-01-10 17:57:42 -07:00
|
|
|
gfp = alloc_hugepage_khugepaged_gfpmask() | __GFP_THISNODE;
|
2016-07-26 16:26:24 -06:00
|
|
|
|
2016-07-26 16:26:26 -06:00
|
|
|
/*
|
|
|
|
* Before allocating the hugepage, release the mmap_sem read lock.
|
|
|
|
* The allocation can take potentially a long time if it involves
|
|
|
|
* sync compaction, and we do not need to hold the mmap_sem during
|
|
|
|
* that. We will recheck the vma after taking it again in write mode.
|
|
|
|
*/
|
|
|
|
up_read(&mm->mmap_sem);
|
|
|
|
new_page = khugepaged_alloc_page(hpage, gfp, node);
|
2016-07-26 16:26:24 -06:00
|
|
|
if (!new_page) {
|
|
|
|
result = SCAN_ALLOC_HUGE_PAGE_FAIL;
|
|
|
|
goto out_nolock;
|
|
|
|
}
|
|
|
|
|
2018-04-10 17:29:30 -06:00
|
|
|
if (unlikely(mem_cgroup_try_charge(new_page, mm, gfp, &memcg, true))) {
|
2016-07-26 16:26:24 -06:00
|
|
|
result = SCAN_CGROUP_CHARGE_FAIL;
|
|
|
|
goto out_nolock;
|
|
|
|
}
|
|
|
|
|
|
|
|
down_read(&mm->mmap_sem);
|
2016-09-19 15:44:01 -06:00
|
|
|
result = hugepage_vma_revalidate(mm, address, &vma);
|
2016-07-26 16:26:24 -06:00
|
|
|
if (result) {
|
|
|
|
mem_cgroup_cancel_charge(new_page, memcg, true);
|
|
|
|
up_read(&mm->mmap_sem);
|
|
|
|
goto out_nolock;
|
|
|
|
}
|
|
|
|
|
|
|
|
pmd = mm_find_pmd(mm, address);
|
|
|
|
if (!pmd) {
|
|
|
|
result = SCAN_PMD_NULL;
|
|
|
|
mem_cgroup_cancel_charge(new_page, memcg, true);
|
|
|
|
up_read(&mm->mmap_sem);
|
|
|
|
goto out_nolock;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* __collapse_huge_page_swapin always returns with mmap_sem locked.
|
2016-07-26 16:26:43 -06:00
|
|
|
* If it fails, we release mmap_sem and jump out_nolock.
|
2016-07-26 16:26:24 -06:00
|
|
|
* Continuing to collapse causes inconsistency.
|
|
|
|
*/
|
2016-07-26 16:26:46 -06:00
|
|
|
if (!__collapse_huge_page_swapin(mm, vma, address, pmd, referenced)) {
|
2016-07-26 16:26:24 -06:00
|
|
|
mem_cgroup_cancel_charge(new_page, memcg, true);
|
|
|
|
up_read(&mm->mmap_sem);
|
|
|
|
goto out_nolock;
|
|
|
|
}
|
|
|
|
|
|
|
|
up_read(&mm->mmap_sem);
|
|
|
|
/*
|
|
|
|
* Prevent all access to pagetables with the exception of
|
|
|
|
* gup_fast later handled by the ptep_clear_flush and the VM
|
|
|
|
* handled by the anon_vma lock + PG_lock.
|
|
|
|
*/
|
|
|
|
down_write(&mm->mmap_sem);
|
2019-06-13 16:56:11 -06:00
|
|
|
result = SCAN_ANY_PROCESS;
|
|
|
|
if (!mmget_still_valid(mm))
|
|
|
|
goto out;
|
2016-09-19 15:44:01 -06:00
|
|
|
result = hugepage_vma_revalidate(mm, address, &vma);
|
2016-07-26 16:26:24 -06:00
|
|
|
if (result)
|
|
|
|
goto out;
|
|
|
|
/* check if the pmd is still valid */
|
|
|
|
if (mm_find_pmd(mm, address) != pmd)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
anon_vma_lock_write(vma->anon_vma);
|
|
|
|
|
2019-05-13 18:20:53 -06:00
|
|
|
mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm,
|
mm/mmu_notifier: contextual information for event triggering invalidation
CPU page table update can happens for many reasons, not only as a result
of a syscall (munmap(), mprotect(), mremap(), madvise(), ...) but also as
a result of kernel activities (memory compression, reclaim, migration,
...).
Users of mmu notifier API track changes to the CPU page table and take
specific action for them. While current API only provide range of virtual
address affected by the change, not why the changes is happening.
This patchset do the initial mechanical convertion of all the places that
calls mmu_notifier_range_init to also provide the default MMU_NOTIFY_UNMAP
event as well as the vma if it is know (most invalidation happens against
a given vma). Passing down the vma allows the users of mmu notifier to
inspect the new vma page protection.
The MMU_NOTIFY_UNMAP is always the safe default as users of mmu notifier
should assume that every for the range is going away when that event
happens. A latter patch do convert mm call path to use a more appropriate
events for each call.
This is done as 2 patches so that no call site is forgotten especialy
as it uses this following coccinelle patch:
%<----------------------------------------------------------------------
@@
identifier I1, I2, I3, I4;
@@
static inline void mmu_notifier_range_init(struct mmu_notifier_range *I1,
+enum mmu_notifier_event event,
+unsigned flags,
+struct vm_area_struct *vma,
struct mm_struct *I2, unsigned long I3, unsigned long I4) { ... }
@@
@@
-#define mmu_notifier_range_init(range, mm, start, end)
+#define mmu_notifier_range_init(range, event, flags, vma, mm, start, end)
@@
expression E1, E3, E4;
identifier I1;
@@
<...
mmu_notifier_range_init(E1,
+MMU_NOTIFY_UNMAP, 0, I1,
I1->vm_mm, E3, E4)
...>
@@
expression E1, E2, E3, E4;
identifier FN, VMA;
@@
FN(..., struct vm_area_struct *VMA, ...) {
<...
mmu_notifier_range_init(E1,
+MMU_NOTIFY_UNMAP, 0, VMA,
E2, E3, E4)
...> }
@@
expression E1, E2, E3, E4;
identifier FN, VMA;
@@
FN(...) {
struct vm_area_struct *VMA;
<...
mmu_notifier_range_init(E1,
+MMU_NOTIFY_UNMAP, 0, VMA,
E2, E3, E4)
...> }
@@
expression E1, E2, E3, E4;
identifier FN;
@@
FN(...) {
<...
mmu_notifier_range_init(E1,
+MMU_NOTIFY_UNMAP, 0, NULL,
E2, E3, E4)
...> }
---------------------------------------------------------------------->%
Applied with:
spatch --all-includes --sp-file mmu-notifier.spatch fs/proc/task_mmu.c --in-place
spatch --sp-file mmu-notifier.spatch --dir kernel/events/ --in-place
spatch --sp-file mmu-notifier.spatch --dir mm --in-place
Link: http://lkml.kernel.org/r/20190326164747.24405-6-jglisse@redhat.com
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Reviewed-by: Ralph Campbell <rcampbell@nvidia.com>
Reviewed-by: Ira Weiny <ira.weiny@intel.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Joonas Lahtinen <joonas.lahtinen@linux.intel.com>
Cc: Jani Nikula <jani.nikula@linux.intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Felix Kuehling <Felix.Kuehling@amd.com>
Cc: Jason Gunthorpe <jgg@mellanox.com>
Cc: Ross Zwisler <zwisler@kernel.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Radim Krcmar <rkrcmar@redhat.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Christian Koenig <christian.koenig@amd.com>
Cc: John Hubbard <jhubbard@nvidia.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2019-05-13 18:20:49 -06:00
|
|
|
address, address + HPAGE_PMD_SIZE);
|
2018-12-28 01:38:09 -07:00
|
|
|
mmu_notifier_invalidate_range_start(&range);
|
2019-11-05 22:16:48 -07:00
|
|
|
|
|
|
|
pte = pte_offset_map(pmd, address);
|
|
|
|
pte_ptl = pte_lockptr(mm, pmd);
|
|
|
|
|
2016-07-26 16:26:24 -06:00
|
|
|
pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */
|
|
|
|
/*
|
|
|
|
* After this gup_fast can't run anymore. This also removes
|
|
|
|
* any huge TLB entry from the CPU so we won't allow
|
|
|
|
* huge and small TLB entries for the same virtual address
|
|
|
|
* to avoid the risk of CPU bugs in that area.
|
|
|
|
*/
|
|
|
|
_pmd = pmdp_collapse_flush(vma, address, pmd);
|
|
|
|
spin_unlock(pmd_ptl);
|
2018-12-28 01:38:09 -07:00
|
|
|
mmu_notifier_invalidate_range_end(&range);
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
spin_lock(pte_ptl);
|
|
|
|
isolated = __collapse_huge_page_isolate(vma, address, pte);
|
|
|
|
spin_unlock(pte_ptl);
|
|
|
|
|
|
|
|
if (unlikely(!isolated)) {
|
|
|
|
pte_unmap(pte);
|
|
|
|
spin_lock(pmd_ptl);
|
|
|
|
BUG_ON(!pmd_none(*pmd));
|
|
|
|
/*
|
|
|
|
* We can only use set_pmd_at when establishing
|
|
|
|
* hugepmds and never for establishing regular pmds that
|
|
|
|
* points to regular pagetables. Use pmd_populate for that
|
|
|
|
*/
|
|
|
|
pmd_populate(mm, pmd, pmd_pgtable(_pmd));
|
|
|
|
spin_unlock(pmd_ptl);
|
|
|
|
anon_vma_unlock_write(vma->anon_vma);
|
|
|
|
result = SCAN_FAIL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* All pages are isolated and locked so anon_vma rmap
|
|
|
|
* can't run anymore.
|
|
|
|
*/
|
|
|
|
anon_vma_unlock_write(vma->anon_vma);
|
|
|
|
|
|
|
|
__collapse_huge_page_copy(pte, new_page, vma, address, pte_ptl);
|
|
|
|
pte_unmap(pte);
|
|
|
|
__SetPageUptodate(new_page);
|
|
|
|
pgtable = pmd_pgtable(_pmd);
|
|
|
|
|
|
|
|
_pmd = mk_huge_pmd(new_page, vma->vm_page_prot);
|
2017-11-29 10:01:01 -07:00
|
|
|
_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_pmd), vma);
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* spin_lock() below is not the equivalent of smp_wmb(), so
|
|
|
|
* this is needed to avoid the copy_huge_page writes to become
|
|
|
|
* visible after the set_pmd_at() write.
|
|
|
|
*/
|
|
|
|
smp_wmb();
|
|
|
|
|
|
|
|
spin_lock(pmd_ptl);
|
|
|
|
BUG_ON(!pmd_none(*pmd));
|
|
|
|
page_add_new_anon_rmap(new_page, vma, address, true);
|
|
|
|
mem_cgroup_commit_charge(new_page, memcg, false, true);
|
2019-03-05 16:48:09 -07:00
|
|
|
count_memcg_events(memcg, THP_COLLAPSE_ALLOC, 1);
|
2016-07-26 16:26:24 -06:00
|
|
|
lru_cache_add_active_or_unevictable(new_page, vma);
|
|
|
|
pgtable_trans_huge_deposit(mm, pmd, pgtable);
|
|
|
|
set_pmd_at(mm, address, pmd, _pmd);
|
|
|
|
update_mmu_cache_pmd(vma, address, pmd);
|
|
|
|
spin_unlock(pmd_ptl);
|
|
|
|
|
|
|
|
*hpage = NULL;
|
|
|
|
|
|
|
|
khugepaged_pages_collapsed++;
|
|
|
|
result = SCAN_SUCCEED;
|
|
|
|
out_up_write:
|
|
|
|
up_write(&mm->mmap_sem);
|
|
|
|
out_nolock:
|
|
|
|
trace_mm_collapse_huge_page(mm, isolated, result);
|
|
|
|
return;
|
|
|
|
out:
|
|
|
|
mem_cgroup_cancel_charge(new_page, memcg, true);
|
|
|
|
goto out_up_write;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int khugepaged_scan_pmd(struct mm_struct *mm,
|
|
|
|
struct vm_area_struct *vma,
|
|
|
|
unsigned long address,
|
|
|
|
struct page **hpage)
|
|
|
|
{
|
|
|
|
pmd_t *pmd;
|
|
|
|
pte_t *pte, *_pte;
|
2016-07-26 16:26:46 -06:00
|
|
|
int ret = 0, none_or_zero = 0, result = 0, referenced = 0;
|
2016-07-26 16:26:24 -06:00
|
|
|
struct page *page = NULL;
|
|
|
|
unsigned long _address;
|
|
|
|
spinlock_t *ptl;
|
|
|
|
int node = NUMA_NO_NODE, unmapped = 0;
|
2016-07-26 16:26:46 -06:00
|
|
|
bool writable = false;
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
VM_BUG_ON(address & ~HPAGE_PMD_MASK);
|
|
|
|
|
|
|
|
pmd = mm_find_pmd(mm, address);
|
|
|
|
if (!pmd) {
|
|
|
|
result = SCAN_PMD_NULL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
memset(khugepaged_node_load, 0, sizeof(khugepaged_node_load));
|
|
|
|
pte = pte_offset_map_lock(mm, pmd, address, &ptl);
|
|
|
|
for (_address = address, _pte = pte; _pte < pte+HPAGE_PMD_NR;
|
|
|
|
_pte++, _address += PAGE_SIZE) {
|
|
|
|
pte_t pteval = *_pte;
|
|
|
|
if (is_swap_pte(pteval)) {
|
|
|
|
if (++unmapped <= khugepaged_max_ptes_swap) {
|
|
|
|
continue;
|
|
|
|
} else {
|
|
|
|
result = SCAN_EXCEED_SWAP_PTE;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
|
|
|
|
if (!userfaultfd_armed(vma) &&
|
|
|
|
++none_or_zero <= khugepaged_max_ptes_none) {
|
|
|
|
continue;
|
|
|
|
} else {
|
|
|
|
result = SCAN_EXCEED_NONE_PTE;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!pte_present(pteval)) {
|
|
|
|
result = SCAN_PTE_NON_PRESENT;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
if (pte_write(pteval))
|
|
|
|
writable = true;
|
|
|
|
|
|
|
|
page = vm_normal_page(vma, _address, pteval);
|
|
|
|
if (unlikely(!page)) {
|
|
|
|
result = SCAN_PAGE_NULL;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* TODO: teach khugepaged to collapse THP mapped with pte */
|
|
|
|
if (PageCompound(page)) {
|
|
|
|
result = SCAN_PAGE_COMPOUND;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Record which node the original page is from and save this
|
|
|
|
* information to khugepaged_node_load[].
|
|
|
|
* Khupaged will allocate hugepage from the node has the max
|
|
|
|
* hit record.
|
|
|
|
*/
|
|
|
|
node = page_to_nid(page);
|
|
|
|
if (khugepaged_scan_abort(node)) {
|
|
|
|
result = SCAN_SCAN_ABORT;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
khugepaged_node_load[node]++;
|
|
|
|
if (!PageLRU(page)) {
|
|
|
|
result = SCAN_PAGE_LRU;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
if (PageLocked(page)) {
|
|
|
|
result = SCAN_PAGE_LOCK;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
if (!PageAnon(page)) {
|
|
|
|
result = SCAN_PAGE_ANON;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* cannot use mapcount: can't collapse if there's a gup pin.
|
|
|
|
* The page must only be referenced by the scanned process
|
|
|
|
* and page swap cache.
|
|
|
|
*/
|
2017-05-03 15:53:35 -06:00
|
|
|
if (page_count(page) != 1 + PageSwapCache(page)) {
|
2016-07-26 16:26:24 -06:00
|
|
|
result = SCAN_PAGE_COUNT;
|
|
|
|
goto out_unmap;
|
|
|
|
}
|
|
|
|
if (pte_young(pteval) ||
|
|
|
|
page_is_young(page) || PageReferenced(page) ||
|
|
|
|
mmu_notifier_test_young(vma->vm_mm, address))
|
2016-07-26 16:26:46 -06:00
|
|
|
referenced++;
|
2016-07-26 16:26:24 -06:00
|
|
|
}
|
|
|
|
if (writable) {
|
|
|
|
if (referenced) {
|
|
|
|
result = SCAN_SUCCEED;
|
|
|
|
ret = 1;
|
|
|
|
} else {
|
2016-07-26 16:26:46 -06:00
|
|
|
result = SCAN_LACK_REFERENCED_PAGE;
|
2016-07-26 16:26:24 -06:00
|
|
|
}
|
|
|
|
} else {
|
|
|
|
result = SCAN_PAGE_RO;
|
|
|
|
}
|
|
|
|
out_unmap:
|
|
|
|
pte_unmap_unlock(pte, ptl);
|
|
|
|
if (ret) {
|
|
|
|
node = khugepaged_find_target_node();
|
|
|
|
/* collapse_huge_page will return with the mmap_sem released */
|
2016-09-19 15:44:01 -06:00
|
|
|
collapse_huge_page(mm, address, hpage, node, referenced);
|
2016-07-26 16:26:24 -06:00
|
|
|
}
|
|
|
|
out:
|
|
|
|
trace_mm_khugepaged_scan_pmd(mm, page, writable, referenced,
|
|
|
|
none_or_zero, result, unmapped);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void collect_mm_slot(struct mm_slot *mm_slot)
|
|
|
|
{
|
|
|
|
struct mm_struct *mm = mm_slot->mm;
|
|
|
|
|
2018-10-05 00:45:47 -06:00
|
|
|
lockdep_assert_held(&khugepaged_mm_lock);
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
if (khugepaged_test_exit(mm)) {
|
|
|
|
/* free mm_slot */
|
|
|
|
hash_del(&mm_slot->hash);
|
|
|
|
list_del(&mm_slot->mm_node);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Not strictly needed because the mm exited already.
|
|
|
|
*
|
|
|
|
* clear_bit(MMF_VM_HUGEPAGE, &mm->flags);
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* khugepaged_mm_lock actually not necessary for the below */
|
|
|
|
free_mm_slot(mm_slot);
|
|
|
|
mmdrop(mm);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-07-26 16:26:35 -06:00
|
|
|
#if defined(CONFIG_SHMEM) && defined(CONFIG_TRANSPARENT_HUGE_PAGECACHE)
|
2019-09-23 16:38:30 -06:00
|
|
|
/*
|
|
|
|
* Notify khugepaged that given addr of the mm is pte-mapped THP. Then
|
|
|
|
* khugepaged should try to collapse the page table.
|
|
|
|
*/
|
|
|
|
static int khugepaged_add_pte_mapped_thp(struct mm_struct *mm,
|
|
|
|
unsigned long addr)
|
|
|
|
{
|
|
|
|
struct mm_slot *mm_slot;
|
|
|
|
|
|
|
|
VM_BUG_ON(addr & ~HPAGE_PMD_MASK);
|
|
|
|
|
|
|
|
spin_lock(&khugepaged_mm_lock);
|
|
|
|
mm_slot = get_mm_slot(mm);
|
|
|
|
if (likely(mm_slot && mm_slot->nr_pte_mapped_thp < MAX_PTE_MAPPED_THP))
|
|
|
|
mm_slot->pte_mapped_thp[mm_slot->nr_pte_mapped_thp++] = addr;
|
|
|
|
spin_unlock(&khugepaged_mm_lock);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Try to collapse a pte-mapped THP for mm at address haddr.
|
|
|
|
*
|
|
|
|
* This function checks whether all the PTEs in the PMD are pointing to the
|
|
|
|
* right THP. If so, retract the page table so the THP can refault in with
|
|
|
|
* as pmd-mapped.
|
|
|
|
*/
|
|
|
|
void collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr)
|
|
|
|
{
|
|
|
|
unsigned long haddr = addr & HPAGE_PMD_MASK;
|
|
|
|
struct vm_area_struct *vma = find_vma(mm, haddr);
|
|
|
|
struct page *hpage = NULL;
|
|
|
|
pte_t *start_pte, *pte;
|
|
|
|
pmd_t *pmd, _pmd;
|
|
|
|
spinlock_t *ptl;
|
|
|
|
int count = 0;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
if (!vma || !vma->vm_file ||
|
|
|
|
vma->vm_start > haddr || vma->vm_end < haddr + HPAGE_PMD_SIZE)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This vm_flags may not have VM_HUGEPAGE if the page was not
|
|
|
|
* collapsed by this mm. But we can still collapse if the page is
|
|
|
|
* the valid THP. Add extra VM_HUGEPAGE so hugepage_vma_check()
|
|
|
|
* will not fail the vma for missing VM_HUGEPAGE
|
|
|
|
*/
|
|
|
|
if (!hugepage_vma_check(vma, vma->vm_flags | VM_HUGEPAGE))
|
|
|
|
return;
|
|
|
|
|
|
|
|
pmd = mm_find_pmd(mm, haddr);
|
|
|
|
if (!pmd)
|
|
|
|
return;
|
|
|
|
|
|
|
|
start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl);
|
|
|
|
|
|
|
|
/* step 1: check all mapped PTEs are to the right huge page */
|
|
|
|
for (i = 0, addr = haddr, pte = start_pte;
|
|
|
|
i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) {
|
|
|
|
struct page *page;
|
|
|
|
|
|
|
|
/* empty pte, skip */
|
|
|
|
if (pte_none(*pte))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/* page swapped out, abort */
|
|
|
|
if (!pte_present(*pte))
|
|
|
|
goto abort;
|
|
|
|
|
|
|
|
page = vm_normal_page(vma, addr, *pte);
|
|
|
|
|
|
|
|
if (!page || !PageCompound(page))
|
|
|
|
goto abort;
|
|
|
|
|
|
|
|
if (!hpage) {
|
|
|
|
hpage = compound_head(page);
|
|
|
|
/*
|
|
|
|
* The mapping of the THP should not change.
|
|
|
|
*
|
|
|
|
* Note that uprobe, debugger, or MAP_PRIVATE may
|
|
|
|
* change the page table, but the new page will
|
|
|
|
* not pass PageCompound() check.
|
|
|
|
*/
|
|
|
|
if (WARN_ON(hpage->mapping != vma->vm_file->f_mapping))
|
|
|
|
goto abort;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Confirm the page maps to the correct subpage.
|
|
|
|
*
|
|
|
|
* Note that uprobe, debugger, or MAP_PRIVATE may change
|
|
|
|
* the page table, but the new page will not pass
|
|
|
|
* PageCompound() check.
|
|
|
|
*/
|
|
|
|
if (WARN_ON(hpage + i != page))
|
|
|
|
goto abort;
|
|
|
|
count++;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* step 2: adjust rmap */
|
|
|
|
for (i = 0, addr = haddr, pte = start_pte;
|
|
|
|
i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE, pte++) {
|
|
|
|
struct page *page;
|
|
|
|
|
|
|
|
if (pte_none(*pte))
|
|
|
|
continue;
|
|
|
|
page = vm_normal_page(vma, addr, *pte);
|
|
|
|
page_remove_rmap(page, false);
|
|
|
|
}
|
|
|
|
|
|
|
|
pte_unmap_unlock(start_pte, ptl);
|
|
|
|
|
|
|
|
/* step 3: set proper refcount and mm_counters. */
|
|
|
|
if (hpage) {
|
|
|
|
page_ref_sub(hpage, count);
|
|
|
|
add_mm_counter(vma->vm_mm, mm_counter_file(hpage), -count);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* step 4: collapse pmd */
|
|
|
|
ptl = pmd_lock(vma->vm_mm, pmd);
|
|
|
|
_pmd = pmdp_collapse_flush(vma, addr, pmd);
|
|
|
|
spin_unlock(ptl);
|
|
|
|
mm_dec_nr_ptes(mm);
|
|
|
|
pte_free(mm, pmd_pgtable(_pmd));
|
|
|
|
return;
|
|
|
|
|
|
|
|
abort:
|
|
|
|
pte_unmap_unlock(start_pte, ptl);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int khugepaged_collapse_pte_mapped_thps(struct mm_slot *mm_slot)
|
|
|
|
{
|
|
|
|
struct mm_struct *mm = mm_slot->mm;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
if (likely(mm_slot->nr_pte_mapped_thp == 0))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
if (!down_write_trylock(&mm->mmap_sem))
|
|
|
|
return -EBUSY;
|
|
|
|
|
|
|
|
if (unlikely(khugepaged_test_exit(mm)))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
for (i = 0; i < mm_slot->nr_pte_mapped_thp; i++)
|
|
|
|
collapse_pte_mapped_thp(mm, mm_slot->pte_mapped_thp[i]);
|
|
|
|
|
|
|
|
out:
|
|
|
|
mm_slot->nr_pte_mapped_thp = 0;
|
|
|
|
up_write(&mm->mmap_sem);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-07-26 16:26:32 -06:00
|
|
|
static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
|
|
|
|
{
|
|
|
|
struct vm_area_struct *vma;
|
|
|
|
unsigned long addr;
|
|
|
|
pmd_t *pmd, _pmd;
|
|
|
|
|
|
|
|
i_mmap_lock_write(mapping);
|
|
|
|
vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff) {
|
2019-09-23 16:38:30 -06:00
|
|
|
/*
|
|
|
|
* Check vma->anon_vma to exclude MAP_PRIVATE mappings that
|
|
|
|
* got written to. These VMAs are likely not worth investing
|
|
|
|
* down_write(mmap_sem) as PMD-mapping is likely to be split
|
|
|
|
* later.
|
|
|
|
*
|
|
|
|
* Not that vma->anon_vma check is racy: it can be set up after
|
|
|
|
* the check but before we took mmap_sem by the fault path.
|
|
|
|
* But page lock would prevent establishing any new ptes of the
|
|
|
|
* page, so we are safe.
|
|
|
|
*
|
|
|
|
* An alternative would be drop the check, but check that page
|
|
|
|
* table is clear before calling pmdp_collapse_flush() under
|
|
|
|
* ptl. It has higher chance to recover THP for the VMA, but
|
|
|
|
* has higher cost too.
|
|
|
|
*/
|
2016-07-26 16:26:32 -06:00
|
|
|
if (vma->anon_vma)
|
|
|
|
continue;
|
|
|
|
addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
|
|
|
|
if (addr & ~HPAGE_PMD_MASK)
|
|
|
|
continue;
|
|
|
|
if (vma->vm_end < addr + HPAGE_PMD_SIZE)
|
|
|
|
continue;
|
|
|
|
pmd = mm_find_pmd(vma->vm_mm, addr);
|
|
|
|
if (!pmd)
|
|
|
|
continue;
|
|
|
|
/*
|
|
|
|
* We need exclusive mmap_sem to retract page table.
|
2019-09-23 16:38:30 -06:00
|
|
|
*
|
|
|
|
* We use trylock due to lock inversion: we need to acquire
|
|
|
|
* mmap_sem while holding page lock. Fault path does it in
|
|
|
|
* reverse order. Trylock is a way to avoid deadlock.
|
2016-07-26 16:26:32 -06:00
|
|
|
*/
|
|
|
|
if (down_write_trylock(&vma->vm_mm->mmap_sem)) {
|
|
|
|
spinlock_t *ptl = pmd_lock(vma->vm_mm, pmd);
|
|
|
|
/* assume page table is clear */
|
|
|
|
_pmd = pmdp_collapse_flush(vma, addr, pmd);
|
|
|
|
spin_unlock(ptl);
|
|
|
|
up_write(&vma->vm_mm->mmap_sem);
|
2017-11-15 18:35:37 -07:00
|
|
|
mm_dec_nr_ptes(vma->vm_mm);
|
2017-01-10 17:57:18 -07:00
|
|
|
pte_free(vma->vm_mm, pmd_pgtable(_pmd));
|
2019-09-23 16:38:30 -06:00
|
|
|
} else {
|
|
|
|
/* Try again later */
|
|
|
|
khugepaged_add_pte_mapped_thp(vma->vm_mm, addr);
|
2016-07-26 16:26:32 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
i_mmap_unlock_write(mapping);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
2019-09-23 16:38:00 -06:00
|
|
|
* collapse_file - collapse filemap/tmpfs/shmem pages into huge one.
|
2016-07-26 16:26:32 -06:00
|
|
|
*
|
|
|
|
* Basic scheme is simple, details are more complex:
|
mm/khugepaged: collapse_shmem() without freezing new_page
khugepaged's collapse_shmem() does almost all of its work, to assemble
the huge new_page from 512 scattered old pages, with the new_page's
refcount frozen to 0 (and refcounts of all old pages so far also frozen
to 0). Including shmem_getpage() to read in any which were out on swap,
memory reclaim if necessary to allocate their intermediate pages, and
copying over all the data from old to new.
Imagine the frozen refcount as a spinlock held, but without any lock
debugging to highlight the abuse: it's not good, and under serious load
heads into lockups - speculative getters of the page are not expecting
to spin while khugepaged is rescheduled.
One can get a little further under load by hacking around elsewhere; but
fortunately, freezing the new_page turns out to have been entirely
unnecessary, with no hacks needed elsewhere.
The huge new_page lock is already held throughout, and guards all its
subpages as they are brought one by one into the page cache tree; and
anything reading the data in that page, without the lock, before it has
been marked PageUptodate, would already be in the wrong. So simply
eliminate the freezing of the new_page.
Each of the old pages remains frozen with refcount 0 after it has been
replaced by a new_page subpage in the page cache tree, until they are
all unfrozen on success or failure: just as before. They could be
unfrozen sooner, but cause no problem once no longer visible to
find_get_entry(), filemap_map_pages() and other speculative lookups.
Link: http://lkml.kernel.org/r/alpine.LSU.2.11.1811261527570.2275@eggly.anvils
Fixes: f3f0e1d2150b2 ("khugepaged: add support of collapse for tmpfs/shmem pages")
Signed-off-by: Hugh Dickins <hughd@google.com>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: <stable@vger.kernel.org> [4.8+]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-11-30 15:10:43 -07:00
|
|
|
* - allocate and lock a new huge page;
|
2017-12-04 12:56:08 -07:00
|
|
|
* - scan page cache replacing old pages with the new one
|
2019-09-23 16:38:00 -06:00
|
|
|
* + swap/gup in pages if necessary;
|
2016-07-26 16:26:32 -06:00
|
|
|
* + fill in gaps;
|
2017-12-04 12:56:08 -07:00
|
|
|
* + keep old pages around in case rollback is required;
|
|
|
|
* - if replacing succeeds:
|
2016-07-26 16:26:32 -06:00
|
|
|
* + copy data over;
|
|
|
|
* + free old pages;
|
mm/khugepaged: collapse_shmem() without freezing new_page
khugepaged's collapse_shmem() does almost all of its work, to assemble
the huge new_page from 512 scattered old pages, with the new_page's
refcount frozen to 0 (and refcounts of all old pages so far also frozen
to 0). Including shmem_getpage() to read in any which were out on swap,
memory reclaim if necessary to allocate their intermediate pages, and
copying over all the data from old to new.
Imagine the frozen refcount as a spinlock held, but without any lock
debugging to highlight the abuse: it's not good, and under serious load
heads into lockups - speculative getters of the page are not expecting
to spin while khugepaged is rescheduled.
One can get a little further under load by hacking around elsewhere; but
fortunately, freezing the new_page turns out to have been entirely
unnecessary, with no hacks needed elsewhere.
The huge new_page lock is already held throughout, and guards all its
subpages as they are brought one by one into the page cache tree; and
anything reading the data in that page, without the lock, before it has
been marked PageUptodate, would already be in the wrong. So simply
eliminate the freezing of the new_page.
Each of the old pages remains frozen with refcount 0 after it has been
replaced by a new_page subpage in the page cache tree, until they are
all unfrozen on success or failure: just as before. They could be
unfrozen sooner, but cause no problem once no longer visible to
find_get_entry(), filemap_map_pages() and other speculative lookups.
Link: http://lkml.kernel.org/r/alpine.LSU.2.11.1811261527570.2275@eggly.anvils
Fixes: f3f0e1d2150b2 ("khugepaged: add support of collapse for tmpfs/shmem pages")
Signed-off-by: Hugh Dickins <hughd@google.com>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: <stable@vger.kernel.org> [4.8+]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-11-30 15:10:43 -07:00
|
|
|
* + unlock huge page;
|
2016-07-26 16:26:32 -06:00
|
|
|
* - if replacing failed;
|
|
|
|
* + put all pages back and unfreeze them;
|
2017-12-04 12:56:08 -07:00
|
|
|
* + restore gaps in the page cache;
|
mm/khugepaged: collapse_shmem() without freezing new_page
khugepaged's collapse_shmem() does almost all of its work, to assemble
the huge new_page from 512 scattered old pages, with the new_page's
refcount frozen to 0 (and refcounts of all old pages so far also frozen
to 0). Including shmem_getpage() to read in any which were out on swap,
memory reclaim if necessary to allocate their intermediate pages, and
copying over all the data from old to new.
Imagine the frozen refcount as a spinlock held, but without any lock
debugging to highlight the abuse: it's not good, and under serious load
heads into lockups - speculative getters of the page are not expecting
to spin while khugepaged is rescheduled.
One can get a little further under load by hacking around elsewhere; but
fortunately, freezing the new_page turns out to have been entirely
unnecessary, with no hacks needed elsewhere.
The huge new_page lock is already held throughout, and guards all its
subpages as they are brought one by one into the page cache tree; and
anything reading the data in that page, without the lock, before it has
been marked PageUptodate, would already be in the wrong. So simply
eliminate the freezing of the new_page.
Each of the old pages remains frozen with refcount 0 after it has been
replaced by a new_page subpage in the page cache tree, until they are
all unfrozen on success or failure: just as before. They could be
unfrozen sooner, but cause no problem once no longer visible to
find_get_entry(), filemap_map_pages() and other speculative lookups.
Link: http://lkml.kernel.org/r/alpine.LSU.2.11.1811261527570.2275@eggly.anvils
Fixes: f3f0e1d2150b2 ("khugepaged: add support of collapse for tmpfs/shmem pages")
Signed-off-by: Hugh Dickins <hughd@google.com>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: <stable@vger.kernel.org> [4.8+]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-11-30 15:10:43 -07:00
|
|
|
* + unlock and free huge page;
|
2016-07-26 16:26:32 -06:00
|
|
|
*/
|
2019-09-23 16:37:57 -06:00
|
|
|
static void collapse_file(struct mm_struct *mm,
|
|
|
|
struct file *file, pgoff_t start,
|
2016-07-26 16:26:32 -06:00
|
|
|
struct page **hpage, int node)
|
|
|
|
{
|
2019-09-23 16:37:57 -06:00
|
|
|
struct address_space *mapping = file->f_mapping;
|
2016-07-26 16:26:32 -06:00
|
|
|
gfp_t gfp;
|
2017-12-04 12:56:08 -07:00
|
|
|
struct page *new_page;
|
2016-07-26 16:26:32 -06:00
|
|
|
struct mem_cgroup *memcg;
|
|
|
|
pgoff_t index, end = start + HPAGE_PMD_NR;
|
|
|
|
LIST_HEAD(pagelist);
|
2017-12-04 12:56:08 -07:00
|
|
|
XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER);
|
2016-07-26 16:26:32 -06:00
|
|
|
int nr_none = 0, result = SCAN_SUCCEED;
|
2019-09-23 16:38:00 -06:00
|
|
|
bool is_shmem = shmem_file(file);
|
2016-07-26 16:26:32 -06:00
|
|
|
|
2019-09-23 16:38:00 -06:00
|
|
|
VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem);
|
2016-07-26 16:26:32 -06:00
|
|
|
VM_BUG_ON(start & (HPAGE_PMD_NR - 1));
|
|
|
|
|
|
|
|
/* Only allocate from the target node */
|
2017-01-10 17:57:42 -07:00
|
|
|
gfp = alloc_hugepage_khugepaged_gfpmask() | __GFP_THISNODE;
|
2016-07-26 16:26:32 -06:00
|
|
|
|
|
|
|
new_page = khugepaged_alloc_page(hpage, gfp, node);
|
|
|
|
if (!new_page) {
|
|
|
|
result = SCAN_ALLOC_HUGE_PAGE_FAIL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2018-04-10 17:29:30 -06:00
|
|
|
if (unlikely(mem_cgroup_try_charge(new_page, mm, gfp, &memcg, true))) {
|
2016-07-26 16:26:32 -06:00
|
|
|
result = SCAN_CGROUP_CHARGE_FAIL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2018-11-30 15:10:50 -07:00
|
|
|
/* This will be less messy when we use multi-index entries */
|
|
|
|
do {
|
|
|
|
xas_lock_irq(&xas);
|
|
|
|
xas_create_range(&xas);
|
|
|
|
if (!xas_error(&xas))
|
|
|
|
break;
|
|
|
|
xas_unlock_irq(&xas);
|
|
|
|
if (!xas_nomem(&xas, GFP_KERNEL)) {
|
|
|
|
mem_cgroup_cancel_charge(new_page, memcg, true);
|
|
|
|
result = SCAN_FAIL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
} while (1);
|
|
|
|
|
2018-11-30 15:10:39 -07:00
|
|
|
__SetPageLocked(new_page);
|
2019-09-23 16:38:00 -06:00
|
|
|
if (is_shmem)
|
|
|
|
__SetPageSwapBacked(new_page);
|
2016-07-26 16:26:32 -06:00
|
|
|
new_page->index = start;
|
|
|
|
new_page->mapping = mapping;
|
|
|
|
|
|
|
|
/*
|
mm/khugepaged: collapse_shmem() without freezing new_page
khugepaged's collapse_shmem() does almost all of its work, to assemble
the huge new_page from 512 scattered old pages, with the new_page's
refcount frozen to 0 (and refcounts of all old pages so far also frozen
to 0). Including shmem_getpage() to read in any which were out on swap,
memory reclaim if necessary to allocate their intermediate pages, and
copying over all the data from old to new.
Imagine the frozen refcount as a spinlock held, but without any lock
debugging to highlight the abuse: it's not good, and under serious load
heads into lockups - speculative getters of the page are not expecting
to spin while khugepaged is rescheduled.
One can get a little further under load by hacking around elsewhere; but
fortunately, freezing the new_page turns out to have been entirely
unnecessary, with no hacks needed elsewhere.
The huge new_page lock is already held throughout, and guards all its
subpages as they are brought one by one into the page cache tree; and
anything reading the data in that page, without the lock, before it has
been marked PageUptodate, would already be in the wrong. So simply
eliminate the freezing of the new_page.
Each of the old pages remains frozen with refcount 0 after it has been
replaced by a new_page subpage in the page cache tree, until they are
all unfrozen on success or failure: just as before. They could be
unfrozen sooner, but cause no problem once no longer visible to
find_get_entry(), filemap_map_pages() and other speculative lookups.
Link: http://lkml.kernel.org/r/alpine.LSU.2.11.1811261527570.2275@eggly.anvils
Fixes: f3f0e1d2150b2 ("khugepaged: add support of collapse for tmpfs/shmem pages")
Signed-off-by: Hugh Dickins <hughd@google.com>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: <stable@vger.kernel.org> [4.8+]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-11-30 15:10:43 -07:00
|
|
|
* At this point the new_page is locked and not up-to-date.
|
|
|
|
* It's safe to insert it into the page cache, because nobody would
|
|
|
|
* be able to map it or use it in another way until we unlock it.
|
2016-07-26 16:26:32 -06:00
|
|
|
*/
|
|
|
|
|
2017-12-04 12:56:08 -07:00
|
|
|
xas_set(&xas, start);
|
|
|
|
for (index = start; index < end; index++) {
|
|
|
|
struct page *page = xas_next(&xas);
|
|
|
|
|
|
|
|
VM_BUG_ON(index != xas.xa_index);
|
2019-09-23 16:38:00 -06:00
|
|
|
if (is_shmem) {
|
|
|
|
if (!page) {
|
|
|
|
/*
|
|
|
|
* Stop if extent has been truncated or
|
|
|
|
* hole-punched, and is now completely
|
|
|
|
* empty.
|
|
|
|
*/
|
|
|
|
if (index == start) {
|
|
|
|
if (!xas_next_entry(&xas, end - 1)) {
|
|
|
|
result = SCAN_TRUNCATED;
|
|
|
|
goto xa_locked;
|
|
|
|
}
|
|
|
|
xas_set(&xas, index);
|
|
|
|
}
|
|
|
|
if (!shmem_charge(mapping->host, 1)) {
|
|
|
|
result = SCAN_FAIL;
|
2018-11-30 15:10:39 -07:00
|
|
|
goto xa_locked;
|
2018-11-30 15:10:25 -07:00
|
|
|
}
|
2019-09-23 16:38:00 -06:00
|
|
|
xas_store(&xas, new_page);
|
|
|
|
nr_none++;
|
|
|
|
continue;
|
2018-11-30 15:10:25 -07:00
|
|
|
}
|
2019-09-23 16:38:00 -06:00
|
|
|
|
|
|
|
if (xa_is_value(page) || !PageUptodate(page)) {
|
|
|
|
xas_unlock_irq(&xas);
|
|
|
|
/* swap in or instantiate fallocated page */
|
|
|
|
if (shmem_getpage(mapping->host, index, &page,
|
|
|
|
SGP_NOHUGE)) {
|
|
|
|
result = SCAN_FAIL;
|
|
|
|
goto xa_unlocked;
|
|
|
|
}
|
|
|
|
} else if (trylock_page(page)) {
|
|
|
|
get_page(page);
|
|
|
|
xas_unlock_irq(&xas);
|
|
|
|
} else {
|
|
|
|
result = SCAN_PAGE_LOCK;
|
2018-11-30 15:10:39 -07:00
|
|
|
goto xa_locked;
|
2017-12-04 12:56:08 -07:00
|
|
|
}
|
2019-09-23 16:38:00 -06:00
|
|
|
} else { /* !is_shmem */
|
|
|
|
if (!page || xa_is_value(page)) {
|
|
|
|
xas_unlock_irq(&xas);
|
|
|
|
page_cache_sync_readahead(mapping, &file->f_ra,
|
|
|
|
file, index,
|
|
|
|
PAGE_SIZE);
|
|
|
|
/* drain pagevecs to help isolate_lru_page() */
|
|
|
|
lru_add_drain();
|
|
|
|
page = find_lock_page(mapping, index);
|
|
|
|
if (unlikely(page == NULL)) {
|
|
|
|
result = SCAN_FAIL;
|
|
|
|
goto xa_unlocked;
|
|
|
|
}
|
2019-11-30 18:57:19 -07:00
|
|
|
} else if (PageDirty(page)) {
|
|
|
|
/*
|
|
|
|
* khugepaged only works on read-only fd,
|
|
|
|
* so this page is dirty because it hasn't
|
|
|
|
* been flushed since first write. There
|
|
|
|
* won't be new dirty pages.
|
|
|
|
*
|
|
|
|
* Trigger async flush here and hope the
|
|
|
|
* writeback is done when khugepaged
|
|
|
|
* revisits this page.
|
|
|
|
*
|
|
|
|
* This is a one-off situation. We are not
|
|
|
|
* forcing writeback in loop.
|
|
|
|
*/
|
|
|
|
xas_unlock_irq(&xas);
|
|
|
|
filemap_flush(mapping);
|
|
|
|
result = SCAN_FAIL;
|
|
|
|
goto xa_unlocked;
|
2019-09-23 16:38:00 -06:00
|
|
|
} else if (trylock_page(page)) {
|
|
|
|
get_page(page);
|
|
|
|
xas_unlock_irq(&xas);
|
|
|
|
} else {
|
|
|
|
result = SCAN_PAGE_LOCK;
|
|
|
|
goto xa_locked;
|
2016-07-26 16:26:32 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2018-04-10 17:36:56 -06:00
|
|
|
* The page must be locked, so we can drop the i_pages lock
|
2016-07-26 16:26:32 -06:00
|
|
|
* without racing with truncate.
|
|
|
|
*/
|
|
|
|
VM_BUG_ON_PAGE(!PageLocked(page), page);
|
mm,thp: recheck each page before collapsing file THP
In collapse_file(), for !is_shmem case, current check cannot guarantee
the locked page is up-to-date. Specifically, xas_unlock_irq() should
not be called before lock_page() and get_page(); and it is necessary to
recheck PageUptodate() after locking the page.
With this bug and CONFIG_READ_ONLY_THP_FOR_FS=y, madvise(HUGE)'ed .text
may contain corrupted data. This is because khugepaged mistakenly
collapses some not up-to-date sub pages into a huge page, and assumes
the huge page is up-to-date. This will NOT corrupt data in the disk,
because the page is read-only and never written back. Fix this by
properly checking PageUptodate() after locking the page. This check
replaces "VM_BUG_ON_PAGE(!PageUptodate(page), page);".
Also, move PageDirty() check after locking the page. Current khugepaged
should not try to collapse dirty file THP, because it is limited to
read-only .text. The only case we hit a dirty page here is when the
page hasn't been written since write. Bail out and retry when this
happens.
syzbot reported bug on previous version of this patch.
Link: http://lkml.kernel.org/r/20191106060930.2571389-2-songliubraving@fb.com
Fixes: 99cb0dbd47a1 ("mm,thp: add read-only THP support for (non-shmem) FS")
Signed-off-by: Song Liu <songliubraving@fb.com>
Reported-by: syzbot+efb9e48b9fbdc49bb34a@syzkaller.appspotmail.com
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: William Kucharski <william.kucharski@oracle.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2019-11-15 18:34:53 -07:00
|
|
|
|
|
|
|
/* make sure the page is up to date */
|
|
|
|
if (unlikely(!PageUptodate(page))) {
|
|
|
|
result = SCAN_FAIL;
|
|
|
|
goto out_unlock;
|
|
|
|
}
|
2018-11-30 15:10:47 -07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If file was truncated then extended, or hole-punched, before
|
|
|
|
* we locked the first page, then a THP might be there already.
|
|
|
|
*/
|
|
|
|
if (PageTransCompound(page)) {
|
|
|
|
result = SCAN_PAGE_COMPOUND;
|
|
|
|
goto out_unlock;
|
|
|
|
}
|
2016-07-26 16:26:32 -06:00
|
|
|
|
|
|
|
if (page_mapping(page) != mapping) {
|
|
|
|
result = SCAN_TRUNCATED;
|
|
|
|
goto out_unlock;
|
|
|
|
}
|
|
|
|
|
mm,thp: recheck each page before collapsing file THP
In collapse_file(), for !is_shmem case, current check cannot guarantee
the locked page is up-to-date. Specifically, xas_unlock_irq() should
not be called before lock_page() and get_page(); and it is necessary to
recheck PageUptodate() after locking the page.
With this bug and CONFIG_READ_ONLY_THP_FOR_FS=y, madvise(HUGE)'ed .text
may contain corrupted data. This is because khugepaged mistakenly
collapses some not up-to-date sub pages into a huge page, and assumes
the huge page is up-to-date. This will NOT corrupt data in the disk,
because the page is read-only and never written back. Fix this by
properly checking PageUptodate() after locking the page. This check
replaces "VM_BUG_ON_PAGE(!PageUptodate(page), page);".
Also, move PageDirty() check after locking the page. Current khugepaged
should not try to collapse dirty file THP, because it is limited to
read-only .text. The only case we hit a dirty page here is when the
page hasn't been written since write. Bail out and retry when this
happens.
syzbot reported bug on previous version of this patch.
Link: http://lkml.kernel.org/r/20191106060930.2571389-2-songliubraving@fb.com
Fixes: 99cb0dbd47a1 ("mm,thp: add read-only THP support for (non-shmem) FS")
Signed-off-by: Song Liu <songliubraving@fb.com>
Reported-by: syzbot+efb9e48b9fbdc49bb34a@syzkaller.appspotmail.com
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: William Kucharski <william.kucharski@oracle.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2019-11-15 18:34:53 -07:00
|
|
|
if (!is_shmem && PageDirty(page)) {
|
|
|
|
/*
|
|
|
|
* khugepaged only works on read-only fd, so this
|
|
|
|
* page is dirty because it hasn't been flushed
|
|
|
|
* since first write.
|
|
|
|
*/
|
|
|
|
result = SCAN_FAIL;
|
|
|
|
goto out_unlock;
|
|
|
|
}
|
|
|
|
|
2016-07-26 16:26:32 -06:00
|
|
|
if (isolate_lru_page(page)) {
|
|
|
|
result = SCAN_DEL_PAGE_LRU;
|
2018-11-30 15:10:39 -07:00
|
|
|
goto out_unlock;
|
2016-07-26 16:26:32 -06:00
|
|
|
}
|
|
|
|
|
2019-09-23 16:38:00 -06:00
|
|
|
if (page_has_private(page) &&
|
|
|
|
!try_to_release_page(page, GFP_KERNEL)) {
|
|
|
|
result = SCAN_PAGE_HAS_PRIVATE;
|
|
|
|
goto out_unlock;
|
|
|
|
}
|
|
|
|
|
2016-07-26 16:26:32 -06:00
|
|
|
if (page_mapped(page))
|
2018-01-31 17:17:36 -07:00
|
|
|
unmap_mapping_pages(mapping, index, 1, false);
|
2016-07-26 16:26:32 -06:00
|
|
|
|
2017-12-04 12:56:08 -07:00
|
|
|
xas_lock_irq(&xas);
|
|
|
|
xas_set(&xas, index);
|
2016-07-26 16:26:32 -06:00
|
|
|
|
2017-12-04 12:56:08 -07:00
|
|
|
VM_BUG_ON_PAGE(page != xas_load(&xas), page);
|
2016-07-26 16:26:32 -06:00
|
|
|
VM_BUG_ON_PAGE(page_mapped(page), page);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The page is expected to have page_count() == 3:
|
|
|
|
* - we hold a pin on it;
|
2017-12-04 12:56:08 -07:00
|
|
|
* - one reference from page cache;
|
2016-07-26 16:26:32 -06:00
|
|
|
* - one from isolate_lru_page;
|
|
|
|
*/
|
|
|
|
if (!page_ref_freeze(page, 3)) {
|
|
|
|
result = SCAN_PAGE_COUNT;
|
2018-11-30 15:10:39 -07:00
|
|
|
xas_unlock_irq(&xas);
|
|
|
|
putback_lru_page(page);
|
|
|
|
goto out_unlock;
|
2016-07-26 16:26:32 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Add the page to the list to be able to undo the collapse if
|
|
|
|
* something go wrong.
|
|
|
|
*/
|
|
|
|
list_add_tail(&page->lru, &pagelist);
|
|
|
|
|
|
|
|
/* Finally, replace with the new page. */
|
2019-09-23 16:34:52 -06:00
|
|
|
xas_store(&xas, new_page);
|
2016-07-26 16:26:32 -06:00
|
|
|
continue;
|
|
|
|
out_unlock:
|
|
|
|
unlock_page(page);
|
|
|
|
put_page(page);
|
2018-11-30 15:10:39 -07:00
|
|
|
goto xa_unlocked;
|
2016-07-26 16:26:32 -06:00
|
|
|
}
|
|
|
|
|
2019-09-23 16:38:00 -06:00
|
|
|
if (is_shmem)
|
|
|
|
__inc_node_page_state(new_page, NR_SHMEM_THPS);
|
2019-09-23 16:38:03 -06:00
|
|
|
else {
|
2019-09-23 16:38:00 -06:00
|
|
|
__inc_node_page_state(new_page, NR_FILE_THPS);
|
2019-09-23 16:38:03 -06:00
|
|
|
filemap_nr_thps_inc(mapping);
|
|
|
|
}
|
2019-09-23 16:38:00 -06:00
|
|
|
|
2018-11-30 15:10:39 -07:00
|
|
|
if (nr_none) {
|
|
|
|
struct zone *zone = page_zone(new_page);
|
|
|
|
|
|
|
|
__mod_node_page_state(zone->zone_pgdat, NR_FILE_PAGES, nr_none);
|
2019-09-23 16:38:00 -06:00
|
|
|
if (is_shmem)
|
|
|
|
__mod_node_page_state(zone->zone_pgdat,
|
|
|
|
NR_SHMEM, nr_none);
|
2018-11-30 15:10:39 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
xa_locked:
|
|
|
|
xas_unlock_irq(&xas);
|
2017-12-04 12:56:08 -07:00
|
|
|
xa_unlocked:
|
2018-11-30 15:10:39 -07:00
|
|
|
|
2016-07-26 16:26:32 -06:00
|
|
|
if (result == SCAN_SUCCEED) {
|
2017-12-04 12:56:08 -07:00
|
|
|
struct page *page, *tmp;
|
2016-07-26 16:26:32 -06:00
|
|
|
|
|
|
|
/*
|
2017-12-04 12:56:08 -07:00
|
|
|
* Replacing old pages with new one has succeeded, now we
|
|
|
|
* need to copy the content and free the old pages.
|
2016-07-26 16:26:32 -06:00
|
|
|
*/
|
2018-11-30 15:10:35 -07:00
|
|
|
index = start;
|
2016-07-26 16:26:32 -06:00
|
|
|
list_for_each_entry_safe(page, tmp, &pagelist, lru) {
|
2018-11-30 15:10:35 -07:00
|
|
|
while (index < page->index) {
|
|
|
|
clear_highpage(new_page + (index % HPAGE_PMD_NR));
|
|
|
|
index++;
|
|
|
|
}
|
2016-07-26 16:26:32 -06:00
|
|
|
copy_highpage(new_page + (page->index % HPAGE_PMD_NR),
|
|
|
|
page);
|
|
|
|
list_del(&page->lru);
|
|
|
|
page->mapping = NULL;
|
2018-11-30 15:10:39 -07:00
|
|
|
page_ref_unfreeze(page, 1);
|
2016-07-26 16:26:32 -06:00
|
|
|
ClearPageActive(page);
|
|
|
|
ClearPageUnevictable(page);
|
2018-11-30 15:10:39 -07:00
|
|
|
unlock_page(page);
|
2016-07-26 16:26:32 -06:00
|
|
|
put_page(page);
|
2018-11-30 15:10:35 -07:00
|
|
|
index++;
|
|
|
|
}
|
|
|
|
while (index < end) {
|
|
|
|
clear_highpage(new_page + (index % HPAGE_PMD_NR));
|
|
|
|
index++;
|
2016-07-26 16:26:32 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
SetPageUptodate(new_page);
|
mm/khugepaged: collapse_shmem() without freezing new_page
khugepaged's collapse_shmem() does almost all of its work, to assemble
the huge new_page from 512 scattered old pages, with the new_page's
refcount frozen to 0 (and refcounts of all old pages so far also frozen
to 0). Including shmem_getpage() to read in any which were out on swap,
memory reclaim if necessary to allocate their intermediate pages, and
copying over all the data from old to new.
Imagine the frozen refcount as a spinlock held, but without any lock
debugging to highlight the abuse: it's not good, and under serious load
heads into lockups - speculative getters of the page are not expecting
to spin while khugepaged is rescheduled.
One can get a little further under load by hacking around elsewhere; but
fortunately, freezing the new_page turns out to have been entirely
unnecessary, with no hacks needed elsewhere.
The huge new_page lock is already held throughout, and guards all its
subpages as they are brought one by one into the page cache tree; and
anything reading the data in that page, without the lock, before it has
been marked PageUptodate, would already be in the wrong. So simply
eliminate the freezing of the new_page.
Each of the old pages remains frozen with refcount 0 after it has been
replaced by a new_page subpage in the page cache tree, until they are
all unfrozen on success or failure: just as before. They could be
unfrozen sooner, but cause no problem once no longer visible to
find_get_entry(), filemap_map_pages() and other speculative lookups.
Link: http://lkml.kernel.org/r/alpine.LSU.2.11.1811261527570.2275@eggly.anvils
Fixes: f3f0e1d2150b2 ("khugepaged: add support of collapse for tmpfs/shmem pages")
Signed-off-by: Hugh Dickins <hughd@google.com>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Jerome Glisse <jglisse@redhat.com>
Cc: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Cc: Matthew Wilcox <willy@infradead.org>
Cc: <stable@vger.kernel.org> [4.8+]
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-11-30 15:10:43 -07:00
|
|
|
page_ref_add(new_page, HPAGE_PMD_NR - 1);
|
2016-07-26 16:26:32 -06:00
|
|
|
mem_cgroup_commit_charge(new_page, memcg, false, true);
|
2019-09-23 16:38:00 -06:00
|
|
|
|
|
|
|
if (is_shmem) {
|
|
|
|
set_page_dirty(new_page);
|
|
|
|
lru_cache_add_anon(new_page);
|
|
|
|
} else {
|
|
|
|
lru_cache_add_file(new_page);
|
|
|
|
}
|
2019-03-05 16:48:09 -07:00
|
|
|
count_memcg_events(memcg, THP_COLLAPSE_ALLOC, 1);
|
2016-07-26 16:26:32 -06:00
|
|
|
|
2018-11-30 15:10:39 -07:00
|
|
|
/*
|
|
|
|
* Remove pte page tables, so we can re-fault the page as huge.
|
|
|
|
*/
|
|
|
|
retract_page_tables(mapping, start);
|
2016-07-26 16:26:32 -06:00
|
|
|
*hpage = NULL;
|
2018-08-17 16:45:29 -06:00
|
|
|
|
|
|
|
khugepaged_pages_collapsed++;
|
2016-07-26 16:26:32 -06:00
|
|
|
} else {
|
2017-12-04 12:56:08 -07:00
|
|
|
struct page *page;
|
2018-11-30 15:10:29 -07:00
|
|
|
|
2017-12-04 12:56:08 -07:00
|
|
|
/* Something went wrong: roll back page cache changes */
|
|
|
|
xas_lock_irq(&xas);
|
2018-11-30 15:10:29 -07:00
|
|
|
mapping->nrpages -= nr_none;
|
2019-09-23 16:38:00 -06:00
|
|
|
|
|
|
|
if (is_shmem)
|
|
|
|
shmem_uncharge(mapping->host, nr_none);
|
2018-11-30 15:10:29 -07:00
|
|
|
|
2017-12-04 12:56:08 -07:00
|
|
|
xas_set(&xas, start);
|
|
|
|
xas_for_each(&xas, page, end - 1) {
|
2016-07-26 16:26:32 -06:00
|
|
|
page = list_first_entry_or_null(&pagelist,
|
|
|
|
struct page, lru);
|
2017-12-04 12:56:08 -07:00
|
|
|
if (!page || xas.xa_index < page->index) {
|
2016-07-26 16:26:32 -06:00
|
|
|
if (!nr_none)
|
|
|
|
break;
|
|
|
|
nr_none--;
|
2016-12-12 17:43:35 -07:00
|
|
|
/* Put holes back where they were */
|
2017-12-04 12:56:08 -07:00
|
|
|
xas_store(&xas, NULL);
|
2016-07-26 16:26:32 -06:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2017-12-04 12:56:08 -07:00
|
|
|
VM_BUG_ON_PAGE(page->index != xas.xa_index, page);
|
2016-07-26 16:26:32 -06:00
|
|
|
|
|
|
|
/* Unfreeze the page. */
|
|
|
|
list_del(&page->lru);
|
|
|
|
page_ref_unfreeze(page, 2);
|
2017-12-04 12:56:08 -07:00
|
|
|
xas_store(&xas, page);
|
|
|
|
xas_pause(&xas);
|
|
|
|
xas_unlock_irq(&xas);
|
2016-07-26 16:26:32 -06:00
|
|
|
unlock_page(page);
|
2018-11-30 15:10:39 -07:00
|
|
|
putback_lru_page(page);
|
2017-12-04 12:56:08 -07:00
|
|
|
xas_lock_irq(&xas);
|
2016-07-26 16:26:32 -06:00
|
|
|
}
|
|
|
|
VM_BUG_ON(nr_none);
|
2017-12-04 12:56:08 -07:00
|
|
|
xas_unlock_irq(&xas);
|
2016-07-26 16:26:32 -06:00
|
|
|
|
|
|
|
mem_cgroup_cancel_charge(new_page, memcg, true);
|
|
|
|
new_page->mapping = NULL;
|
|
|
|
}
|
2018-11-30 15:10:39 -07:00
|
|
|
|
|
|
|
unlock_page(new_page);
|
2016-07-26 16:26:32 -06:00
|
|
|
out:
|
|
|
|
VM_BUG_ON(!list_empty(&pagelist));
|
|
|
|
/* TODO: tracepoints */
|
|
|
|
}
|
|
|
|
|
2019-09-23 16:37:57 -06:00
|
|
|
static void khugepaged_scan_file(struct mm_struct *mm,
|
|
|
|
struct file *file, pgoff_t start, struct page **hpage)
|
2016-07-26 16:26:32 -06:00
|
|
|
{
|
|
|
|
struct page *page = NULL;
|
2019-09-23 16:37:57 -06:00
|
|
|
struct address_space *mapping = file->f_mapping;
|
2017-12-04 13:06:23 -07:00
|
|
|
XA_STATE(xas, &mapping->i_pages, start);
|
2016-07-26 16:26:32 -06:00
|
|
|
int present, swap;
|
|
|
|
int node = NUMA_NO_NODE;
|
|
|
|
int result = SCAN_SUCCEED;
|
|
|
|
|
|
|
|
present = 0;
|
|
|
|
swap = 0;
|
|
|
|
memset(khugepaged_node_load, 0, sizeof(khugepaged_node_load));
|
|
|
|
rcu_read_lock();
|
2017-12-04 13:06:23 -07:00
|
|
|
xas_for_each(&xas, page, start + HPAGE_PMD_NR - 1) {
|
|
|
|
if (xas_retry(&xas, page))
|
2016-07-26 16:26:32 -06:00
|
|
|
continue;
|
|
|
|
|
2017-12-04 13:06:23 -07:00
|
|
|
if (xa_is_value(page)) {
|
2016-07-26 16:26:32 -06:00
|
|
|
if (++swap > khugepaged_max_ptes_swap) {
|
|
|
|
result = SCAN_EXCEED_SWAP_PTE;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (PageTransCompound(page)) {
|
|
|
|
result = SCAN_PAGE_COMPOUND;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
node = page_to_nid(page);
|
|
|
|
if (khugepaged_scan_abort(node)) {
|
|
|
|
result = SCAN_SCAN_ABORT;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
khugepaged_node_load[node]++;
|
|
|
|
|
|
|
|
if (!PageLRU(page)) {
|
|
|
|
result = SCAN_PAGE_LRU;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2019-09-23 16:38:00 -06:00
|
|
|
if (page_count(page) !=
|
|
|
|
1 + page_mapcount(page) + page_has_private(page)) {
|
2016-07-26 16:26:32 -06:00
|
|
|
result = SCAN_PAGE_COUNT;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We probably should check if the page is referenced here, but
|
|
|
|
* nobody would transfer pte_young() to PageReferenced() for us.
|
|
|
|
* And rmap walk here is just too costly...
|
|
|
|
*/
|
|
|
|
|
|
|
|
present++;
|
|
|
|
|
|
|
|
if (need_resched()) {
|
2017-12-04 13:06:23 -07:00
|
|
|
xas_pause(&xas);
|
2016-07-26 16:26:32 -06:00
|
|
|
cond_resched_rcu();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
rcu_read_unlock();
|
|
|
|
|
|
|
|
if (result == SCAN_SUCCEED) {
|
|
|
|
if (present < HPAGE_PMD_NR - khugepaged_max_ptes_none) {
|
|
|
|
result = SCAN_EXCEED_NONE_PTE;
|
|
|
|
} else {
|
|
|
|
node = khugepaged_find_target_node();
|
2019-09-23 16:37:57 -06:00
|
|
|
collapse_file(mm, file, start, hpage, node);
|
2016-07-26 16:26:32 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* TODO: tracepoints */
|
|
|
|
}
|
|
|
|
#else
|
2019-09-23 16:37:57 -06:00
|
|
|
static void khugepaged_scan_file(struct mm_struct *mm,
|
|
|
|
struct file *file, pgoff_t start, struct page **hpage)
|
2016-07-26 16:26:32 -06:00
|
|
|
{
|
|
|
|
BUILD_BUG();
|
|
|
|
}
|
2019-09-23 16:38:30 -06:00
|
|
|
|
|
|
|
static int khugepaged_collapse_pte_mapped_thps(struct mm_slot *mm_slot)
|
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
2016-07-26 16:26:32 -06:00
|
|
|
#endif
|
|
|
|
|
2016-07-26 16:26:24 -06:00
|
|
|
static unsigned int khugepaged_scan_mm_slot(unsigned int pages,
|
|
|
|
struct page **hpage)
|
|
|
|
__releases(&khugepaged_mm_lock)
|
|
|
|
__acquires(&khugepaged_mm_lock)
|
|
|
|
{
|
|
|
|
struct mm_slot *mm_slot;
|
|
|
|
struct mm_struct *mm;
|
|
|
|
struct vm_area_struct *vma;
|
|
|
|
int progress = 0;
|
|
|
|
|
|
|
|
VM_BUG_ON(!pages);
|
2018-10-05 00:45:47 -06:00
|
|
|
lockdep_assert_held(&khugepaged_mm_lock);
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
if (khugepaged_scan.mm_slot)
|
|
|
|
mm_slot = khugepaged_scan.mm_slot;
|
|
|
|
else {
|
|
|
|
mm_slot = list_entry(khugepaged_scan.mm_head.next,
|
|
|
|
struct mm_slot, mm_node);
|
|
|
|
khugepaged_scan.address = 0;
|
|
|
|
khugepaged_scan.mm_slot = mm_slot;
|
|
|
|
}
|
|
|
|
spin_unlock(&khugepaged_mm_lock);
|
2019-09-23 16:38:30 -06:00
|
|
|
khugepaged_collapse_pte_mapped_thps(mm_slot);
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
mm = mm_slot->mm;
|
2018-01-31 17:18:28 -07:00
|
|
|
/*
|
|
|
|
* Don't wait for semaphore (to avoid long wait times). Just move to
|
|
|
|
* the next mm on the list.
|
|
|
|
*/
|
|
|
|
vma = NULL;
|
|
|
|
if (unlikely(!down_read_trylock(&mm->mmap_sem)))
|
|
|
|
goto breakouterloop_mmap_sem;
|
|
|
|
if (likely(!khugepaged_test_exit(mm)))
|
2016-07-26 16:26:24 -06:00
|
|
|
vma = find_vma(mm, khugepaged_scan.address);
|
|
|
|
|
|
|
|
progress++;
|
|
|
|
for (; vma; vma = vma->vm_next) {
|
|
|
|
unsigned long hstart, hend;
|
|
|
|
|
|
|
|
cond_resched();
|
|
|
|
if (unlikely(khugepaged_test_exit(mm))) {
|
|
|
|
progress++;
|
|
|
|
break;
|
|
|
|
}
|
mm: thp: pass correct vm_flags to hugepage_vma_check()
khugepaged_enter_vma_merge() passes a stale vma->vm_flags to
hugepage_vma_check(). The argument vm_flags contains the latest value.
Therefore, it is necessary to pass this vm_flags into
hugepage_vma_check().
With this bug, madvise(MADV_HUGEPAGE) for mmap files in shmem fails to
put memory in huge pages. Here is an example of failed madvise():
/* mount /dev/shm with huge=advise:
* mount -o remount,huge=advise /dev/shm */
/* create file /dev/shm/huge */
#define HUGE_FILE "/dev/shm/huge"
fd = open(HUGE_FILE, O_RDONLY);
ptr = mmap(NULL, FILE_SIZE, PROT_READ, MAP_PRIVATE, fd, 0);
ret = madvise(ptr, FILE_SIZE, MADV_HUGEPAGE);
madvise() will return 0, but this memory region is never put in huge
page (check from /proc/meminfo: ShmemHugePages).
Link: http://lkml.kernel.org/r/20180629181752.792831-1-songliubraving@fb.com
Fixes: 02b75dc8160d ("mm: thp: register mm for khugepaged when merging vma for shmem")
Signed-off-by: Song Liu <songliubraving@fb.com>
Reviewed-by: Rik van Riel <riel@surriel.com>
Reviewed-by: Yang Shi <yang.shi@linux.alibaba.com>
Cc: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-08-17 16:47:00 -06:00
|
|
|
if (!hugepage_vma_check(vma, vma->vm_flags)) {
|
2016-07-26 16:26:24 -06:00
|
|
|
skip:
|
|
|
|
progress++;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
hstart = (vma->vm_start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK;
|
|
|
|
hend = vma->vm_end & HPAGE_PMD_MASK;
|
|
|
|
if (hstart >= hend)
|
|
|
|
goto skip;
|
|
|
|
if (khugepaged_scan.address > hend)
|
|
|
|
goto skip;
|
|
|
|
if (khugepaged_scan.address < hstart)
|
|
|
|
khugepaged_scan.address = hstart;
|
|
|
|
VM_BUG_ON(khugepaged_scan.address & ~HPAGE_PMD_MASK);
|
|
|
|
|
|
|
|
while (khugepaged_scan.address < hend) {
|
|
|
|
int ret;
|
|
|
|
cond_resched();
|
|
|
|
if (unlikely(khugepaged_test_exit(mm)))
|
|
|
|
goto breakouterloop;
|
|
|
|
|
|
|
|
VM_BUG_ON(khugepaged_scan.address < hstart ||
|
|
|
|
khugepaged_scan.address + HPAGE_PMD_SIZE >
|
|
|
|
hend);
|
2019-09-23 16:38:00 -06:00
|
|
|
if (IS_ENABLED(CONFIG_SHMEM) && vma->vm_file) {
|
2016-07-26 16:26:35 -06:00
|
|
|
struct file *file;
|
2016-07-26 16:26:32 -06:00
|
|
|
pgoff_t pgoff = linear_page_index(vma,
|
|
|
|
khugepaged_scan.address);
|
2019-09-23 16:38:00 -06:00
|
|
|
|
|
|
|
if (shmem_file(vma->vm_file)
|
|
|
|
&& !shmem_huge_enabled(vma))
|
2016-07-26 16:26:35 -06:00
|
|
|
goto skip;
|
|
|
|
file = get_file(vma->vm_file);
|
2016-07-26 16:26:32 -06:00
|
|
|
up_read(&mm->mmap_sem);
|
|
|
|
ret = 1;
|
2019-09-23 16:37:57 -06:00
|
|
|
khugepaged_scan_file(mm, file, pgoff, hpage);
|
2016-07-26 16:26:32 -06:00
|
|
|
fput(file);
|
|
|
|
} else {
|
|
|
|
ret = khugepaged_scan_pmd(mm, vma,
|
|
|
|
khugepaged_scan.address,
|
|
|
|
hpage);
|
|
|
|
}
|
2016-07-26 16:26:24 -06:00
|
|
|
/* move to next address */
|
|
|
|
khugepaged_scan.address += HPAGE_PMD_SIZE;
|
|
|
|
progress += HPAGE_PMD_NR;
|
|
|
|
if (ret)
|
|
|
|
/* we released mmap_sem so break loop */
|
|
|
|
goto breakouterloop_mmap_sem;
|
|
|
|
if (progress >= pages)
|
|
|
|
goto breakouterloop;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
breakouterloop:
|
|
|
|
up_read(&mm->mmap_sem); /* exit_mmap will destroy ptes after this */
|
|
|
|
breakouterloop_mmap_sem:
|
|
|
|
|
|
|
|
spin_lock(&khugepaged_mm_lock);
|
|
|
|
VM_BUG_ON(khugepaged_scan.mm_slot != mm_slot);
|
|
|
|
/*
|
|
|
|
* Release the current mm_slot if this mm is about to die, or
|
|
|
|
* if we scanned all vmas of this mm.
|
|
|
|
*/
|
|
|
|
if (khugepaged_test_exit(mm) || !vma) {
|
|
|
|
/*
|
|
|
|
* Make sure that if mm_users is reaching zero while
|
|
|
|
* khugepaged runs here, khugepaged_exit will find
|
|
|
|
* mm_slot not pointing to the exiting mm.
|
|
|
|
*/
|
|
|
|
if (mm_slot->mm_node.next != &khugepaged_scan.mm_head) {
|
|
|
|
khugepaged_scan.mm_slot = list_entry(
|
|
|
|
mm_slot->mm_node.next,
|
|
|
|
struct mm_slot, mm_node);
|
|
|
|
khugepaged_scan.address = 0;
|
|
|
|
} else {
|
|
|
|
khugepaged_scan.mm_slot = NULL;
|
|
|
|
khugepaged_full_scans++;
|
|
|
|
}
|
|
|
|
|
|
|
|
collect_mm_slot(mm_slot);
|
|
|
|
}
|
|
|
|
|
|
|
|
return progress;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int khugepaged_has_work(void)
|
|
|
|
{
|
|
|
|
return !list_empty(&khugepaged_scan.mm_head) &&
|
|
|
|
khugepaged_enabled();
|
|
|
|
}
|
|
|
|
|
|
|
|
static int khugepaged_wait_event(void)
|
|
|
|
{
|
|
|
|
return !list_empty(&khugepaged_scan.mm_head) ||
|
|
|
|
kthread_should_stop();
|
|
|
|
}
|
|
|
|
|
|
|
|
static void khugepaged_do_scan(void)
|
|
|
|
{
|
|
|
|
struct page *hpage = NULL;
|
|
|
|
unsigned int progress = 0, pass_through_head = 0;
|
|
|
|
unsigned int pages = khugepaged_pages_to_scan;
|
|
|
|
bool wait = true;
|
|
|
|
|
|
|
|
barrier(); /* write khugepaged_pages_to_scan to local stack */
|
|
|
|
|
|
|
|
while (progress < pages) {
|
|
|
|
if (!khugepaged_prealloc_page(&hpage, &wait))
|
|
|
|
break;
|
|
|
|
|
|
|
|
cond_resched();
|
|
|
|
|
|
|
|
if (unlikely(kthread_should_stop() || try_to_freeze()))
|
|
|
|
break;
|
|
|
|
|
|
|
|
spin_lock(&khugepaged_mm_lock);
|
|
|
|
if (!khugepaged_scan.mm_slot)
|
|
|
|
pass_through_head++;
|
|
|
|
if (khugepaged_has_work() &&
|
|
|
|
pass_through_head < 2)
|
|
|
|
progress += khugepaged_scan_mm_slot(pages - progress,
|
|
|
|
&hpage);
|
|
|
|
else
|
|
|
|
progress = pages;
|
|
|
|
spin_unlock(&khugepaged_mm_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!IS_ERR_OR_NULL(hpage))
|
|
|
|
put_page(hpage);
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool khugepaged_should_wakeup(void)
|
|
|
|
{
|
|
|
|
return kthread_should_stop() ||
|
|
|
|
time_after_eq(jiffies, khugepaged_sleep_expire);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void khugepaged_wait_work(void)
|
|
|
|
{
|
|
|
|
if (khugepaged_has_work()) {
|
|
|
|
const unsigned long scan_sleep_jiffies =
|
|
|
|
msecs_to_jiffies(khugepaged_scan_sleep_millisecs);
|
|
|
|
|
|
|
|
if (!scan_sleep_jiffies)
|
|
|
|
return;
|
|
|
|
|
|
|
|
khugepaged_sleep_expire = jiffies + scan_sleep_jiffies;
|
|
|
|
wait_event_freezable_timeout(khugepaged_wait,
|
|
|
|
khugepaged_should_wakeup(),
|
|
|
|
scan_sleep_jiffies);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (khugepaged_enabled())
|
|
|
|
wait_event_freezable(khugepaged_wait, khugepaged_wait_event());
|
|
|
|
}
|
|
|
|
|
|
|
|
static int khugepaged(void *none)
|
|
|
|
{
|
|
|
|
struct mm_slot *mm_slot;
|
|
|
|
|
|
|
|
set_freezable();
|
|
|
|
set_user_nice(current, MAX_NICE);
|
|
|
|
|
|
|
|
while (!kthread_should_stop()) {
|
|
|
|
khugepaged_do_scan();
|
|
|
|
khugepaged_wait_work();
|
|
|
|
}
|
|
|
|
|
|
|
|
spin_lock(&khugepaged_mm_lock);
|
|
|
|
mm_slot = khugepaged_scan.mm_slot;
|
|
|
|
khugepaged_scan.mm_slot = NULL;
|
|
|
|
if (mm_slot)
|
|
|
|
collect_mm_slot(mm_slot);
|
|
|
|
spin_unlock(&khugepaged_mm_lock);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void set_recommended_min_free_kbytes(void)
|
|
|
|
{
|
|
|
|
struct zone *zone;
|
|
|
|
int nr_zones = 0;
|
|
|
|
unsigned long recommended_min;
|
|
|
|
|
mm/thp: don't count ZONE_MOVABLE as the target for freepage reserving
There was a regression report for "mm/cma: manage the memory of the CMA
area by using the ZONE_MOVABLE" [1] and I think that it is related to
this problem. CMA patchset makes the system use one more zone
(ZONE_MOVABLE) and then increases min_free_kbytes. It reduces usable
memory and it could cause regression.
ZONE_MOVABLE only has movable pages so we don't need to keep enough
freepages to avoid or deal with fragmentation. So, don't count it.
This changes min_free_kbytes and thus min_watermark greatly if
ZONE_MOVABLE is used. It will make the user uses more memory.
System:
22GB ram, fakenuma, 2 nodes. 5 zones are used.
Before:
min_free_kbytes: 112640
zone_info (min_watermark):
Node 0, zone DMA
min 19
Node 0, zone DMA32
min 3778
Node 0, zone Normal
min 10191
Node 0, zone Movable
min 0
Node 0, zone Device
min 0
Node 1, zone DMA
min 0
Node 1, zone DMA32
min 0
Node 1, zone Normal
min 14043
Node 1, zone Movable
min 127
Node 1, zone Device
min 0
After:
min_free_kbytes: 90112
zone_info (min_watermark):
Node 0, zone DMA
min 15
Node 0, zone DMA32
min 3022
Node 0, zone Normal
min 8152
Node 0, zone Movable
min 0
Node 0, zone Device
min 0
Node 1, zone DMA
min 0
Node 1, zone DMA32
min 0
Node 1, zone Normal
min 11234
Node 1, zone Movable
min 102
Node 1, zone Device
min 0
[1] (lkml.kernel.org/r/20180102063528.GG30397%20()%20yexl-desktop)
Link: http://lkml.kernel.org/r/1522913236-15776-1-git-send-email-iamjoonsoo.kim@lge.com
Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-04-10 17:30:27 -06:00
|
|
|
for_each_populated_zone(zone) {
|
|
|
|
/*
|
|
|
|
* We don't need to worry about fragmentation of
|
|
|
|
* ZONE_MOVABLE since it only has movable pages.
|
|
|
|
*/
|
|
|
|
if (zone_idx(zone) > gfp_zone(GFP_USER))
|
|
|
|
continue;
|
|
|
|
|
2016-07-26 16:26:24 -06:00
|
|
|
nr_zones++;
|
mm/thp: don't count ZONE_MOVABLE as the target for freepage reserving
There was a regression report for "mm/cma: manage the memory of the CMA
area by using the ZONE_MOVABLE" [1] and I think that it is related to
this problem. CMA patchset makes the system use one more zone
(ZONE_MOVABLE) and then increases min_free_kbytes. It reduces usable
memory and it could cause regression.
ZONE_MOVABLE only has movable pages so we don't need to keep enough
freepages to avoid or deal with fragmentation. So, don't count it.
This changes min_free_kbytes and thus min_watermark greatly if
ZONE_MOVABLE is used. It will make the user uses more memory.
System:
22GB ram, fakenuma, 2 nodes. 5 zones are used.
Before:
min_free_kbytes: 112640
zone_info (min_watermark):
Node 0, zone DMA
min 19
Node 0, zone DMA32
min 3778
Node 0, zone Normal
min 10191
Node 0, zone Movable
min 0
Node 0, zone Device
min 0
Node 1, zone DMA
min 0
Node 1, zone DMA32
min 0
Node 1, zone Normal
min 14043
Node 1, zone Movable
min 127
Node 1, zone Device
min 0
After:
min_free_kbytes: 90112
zone_info (min_watermark):
Node 0, zone DMA
min 15
Node 0, zone DMA32
min 3022
Node 0, zone Normal
min 8152
Node 0, zone Movable
min 0
Node 0, zone Device
min 0
Node 1, zone DMA
min 0
Node 1, zone DMA32
min 0
Node 1, zone Normal
min 11234
Node 1, zone Movable
min 102
Node 1, zone Device
min 0
[1] (lkml.kernel.org/r/20180102063528.GG30397%20()%20yexl-desktop)
Link: http://lkml.kernel.org/r/1522913236-15776-1-git-send-email-iamjoonsoo.kim@lge.com
Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2018-04-10 17:30:27 -06:00
|
|
|
}
|
2016-07-26 16:26:24 -06:00
|
|
|
|
|
|
|
/* Ensure 2 pageblocks are free to assist fragmentation avoidance */
|
|
|
|
recommended_min = pageblock_nr_pages * nr_zones * 2;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Make sure that on average at least two pageblocks are almost free
|
|
|
|
* of another type, one for a migratetype to fall back to and a
|
|
|
|
* second to avoid subsequent fallbacks of other types There are 3
|
|
|
|
* MIGRATE_TYPES we care about.
|
|
|
|
*/
|
|
|
|
recommended_min += pageblock_nr_pages * nr_zones *
|
|
|
|
MIGRATE_PCPTYPES * MIGRATE_PCPTYPES;
|
|
|
|
|
|
|
|
/* don't ever allow to reserve more than 5% of the lowmem */
|
|
|
|
recommended_min = min(recommended_min,
|
|
|
|
(unsigned long) nr_free_buffer_pages() / 20);
|
|
|
|
recommended_min <<= (PAGE_SHIFT-10);
|
|
|
|
|
|
|
|
if (recommended_min > min_free_kbytes) {
|
|
|
|
if (user_min_free_kbytes >= 0)
|
|
|
|
pr_info("raising min_free_kbytes from %d to %lu to help transparent hugepage allocations\n",
|
|
|
|
min_free_kbytes, recommended_min);
|
|
|
|
|
|
|
|
min_free_kbytes = recommended_min;
|
|
|
|
}
|
|
|
|
setup_per_zone_wmarks();
|
|
|
|
}
|
|
|
|
|
|
|
|
int start_stop_khugepaged(void)
|
|
|
|
{
|
|
|
|
static struct task_struct *khugepaged_thread __read_mostly;
|
|
|
|
static DEFINE_MUTEX(khugepaged_mutex);
|
|
|
|
int err = 0;
|
|
|
|
|
|
|
|
mutex_lock(&khugepaged_mutex);
|
|
|
|
if (khugepaged_enabled()) {
|
|
|
|
if (!khugepaged_thread)
|
|
|
|
khugepaged_thread = kthread_run(khugepaged, NULL,
|
|
|
|
"khugepaged");
|
|
|
|
if (IS_ERR(khugepaged_thread)) {
|
|
|
|
pr_err("khugepaged: kthread_run(khugepaged) failed\n");
|
|
|
|
err = PTR_ERR(khugepaged_thread);
|
|
|
|
khugepaged_thread = NULL;
|
|
|
|
goto fail;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!list_empty(&khugepaged_scan.mm_head))
|
|
|
|
wake_up_interruptible(&khugepaged_wait);
|
|
|
|
|
|
|
|
set_recommended_min_free_kbytes();
|
|
|
|
} else if (khugepaged_thread) {
|
|
|
|
kthread_stop(khugepaged_thread);
|
|
|
|
khugepaged_thread = NULL;
|
|
|
|
}
|
|
|
|
fail:
|
|
|
|
mutex_unlock(&khugepaged_mutex);
|
|
|
|
return err;
|
|
|
|
}
|