![Andrea Arcangeli](/assets/img/avatar_default.png)
With KVM/GFP/XPMEM there isn't just the primary CPU MMU pointing to pages. There are secondary MMUs (with secondary sptes and secondary tlbs) too. sptes in the kvm case are shadow pagetables, but when I say spte in mmu-notifier context, I mean "secondary pte". In GRU case there's no actual secondary pte and there's only a secondary tlb because the GRU secondary MMU has no knowledge about sptes and every secondary tlb miss event in the MMU always generates a page fault that has to be resolved by the CPU (this is not the case of KVM where the a secondary tlb miss will walk sptes in hardware and it will refill the secondary tlb transparently to software if the corresponding spte is present). The same way zap_page_range has to invalidate the pte before freeing the page, the spte (and secondary tlb) must also be invalidated before any page is freed and reused. Currently we take a page_count pin on every page mapped by sptes, but that means the pages can't be swapped whenever they're mapped by any spte because they're part of the guest working set. Furthermore a spte unmap event can immediately lead to a page to be freed when the pin is released (so requiring the same complex and relatively slow tlb_gather smp safe logic we have in zap_page_range and that can be avoided completely if the spte unmap event doesn't require an unpin of the page previously mapped in the secondary MMU). The mmu notifiers allow kvm/GRU/XPMEM to attach to the tsk->mm and know when the VM is swapping or freeing or doing anything on the primary MMU so that the secondary MMU code can drop sptes before the pages are freed, avoiding all page pinning and allowing 100% reliable swapping of guest physical address space. Furthermore it avoids the code that teardown the mappings of the secondary MMU, to implement a logic like tlb_gather in zap_page_range that would require many IPI to flush other cpu tlbs, for each fixed number of spte unmapped. To make an example: if what happens on the primary MMU is a protection downgrade (from writeable to wrprotect) the secondary MMU mappings will be invalidated, and the next secondary-mmu-page-fault will call get_user_pages and trigger a do_wp_page through get_user_pages if it called get_user_pages with write=1, and it'll re-establishing an updated spte or secondary-tlb-mapping on the copied page. Or it will setup a readonly spte or readonly tlb mapping if it's a guest-read, if it calls get_user_pages with write=0. This is just an example. This allows to map any page pointed by any pte (and in turn visible in the primary CPU MMU), into a secondary MMU (be it a pure tlb like GRU, or an full MMU with both sptes and secondary-tlb like the shadow-pagetable layer with kvm), or a remote DMA in software like XPMEM (hence needing of schedule in XPMEM code to send the invalidate to the remote node, while no need to schedule in kvm/gru as it's an immediate event like invalidating primary-mmu pte). At least for KVM without this patch it's impossible to swap guests reliably. And having this feature and removing the page pin allows several other optimizations that simplify life considerably. Dependencies: 1) mm_take_all_locks() to register the mmu notifier when the whole VM isn't doing anything with "mm". This allows mmu notifier users to keep track if the VM is in the middle of the invalidate_range_begin/end critical section with an atomic counter incraese in range_begin and decreased in range_end. No secondary MMU page fault is allowed to map any spte or secondary tlb reference, while the VM is in the middle of range_begin/end as any page returned by get_user_pages in that critical section could later immediately be freed without any further ->invalidate_page notification (invalidate_range_begin/end works on ranges and ->invalidate_page isn't called immediately before freeing the page). To stop all page freeing and pagetable overwrites the mmap_sem must be taken in write mode and all other anon_vma/i_mmap locks must be taken too. 2) It'd be a waste to add branches in the VM if nobody could possibly run KVM/GRU/XPMEM on the kernel, so mmu notifiers will only enabled if CONFIG_KVM=m/y. In the current kernel kvm won't yet take advantage of mmu notifiers, but this already allows to compile a KVM external module against a kernel with mmu notifiers enabled and from the next pull from kvm.git we'll start using them. And GRU/XPMEM will also be able to continue the development by enabling KVM=m in their config, until they submit all GRU/XPMEM GPLv2 code to the mainline kernel. Then they can also enable MMU_NOTIFIERS in the same way KVM does it (even if KVM=n). This guarantees nobody selects MMU_NOTIFIER=y if KVM and GRU and XPMEM are all =n. The mmu_notifier_register call can fail because mm_take_all_locks may be interrupted by a signal and return -EINTR. Because mmu_notifier_reigster is used when a driver startup, a failure can be gracefully handled. Here an example of the change applied to kvm to register the mmu notifiers. Usually when a driver startups other allocations are required anyway and -ENOMEM failure paths exists already. struct kvm *kvm_arch_create_vm(void) { struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL); + int err; if (!kvm) return ERR_PTR(-ENOMEM); INIT_LIST_HEAD(&kvm->arch.active_mmu_pages); + kvm->arch.mmu_notifier.ops = &kvm_mmu_notifier_ops; + err = mmu_notifier_register(&kvm->arch.mmu_notifier, current->mm); + if (err) { + kfree(kvm); + return ERR_PTR(err); + } + return kvm; } mmu_notifier_unregister returns void and it's reliable. The patch also adds a few needed but missing includes that would prevent kernel to compile after these changes on non-x86 archs (x86 didn't need them by luck). [akpm@linux-foundation.org: coding-style fixes] [akpm@linux-foundation.org: fix mm/filemap_xip.c build] [akpm@linux-foundation.org: fix mm/mmu_notifier.c build] Signed-off-by: Andrea Arcangeli <andrea@qumranet.com> Signed-off-by: Nick Piggin <npiggin@suse.de> Signed-off-by: Christoph Lameter <cl@linux-foundation.org> Cc: Jack Steiner <steiner@sgi.com> Cc: Robin Holt <holt@sgi.com> Cc: Nick Piggin <npiggin@suse.de> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Kanoj Sarcar <kanojsarcar@yahoo.com> Cc: Roland Dreier <rdreier@cisco.com> Cc: Steve Wise <swise@opengridcomputing.com> Cc: Avi Kivity <avi@qumranet.com> Cc: Hugh Dickins <hugh@veritas.com> Cc: Rusty Russell <rusty@rustcorp.com.au> Cc: Anthony Liguori <aliguori@us.ibm.com> Cc: Chris Wright <chrisw@redhat.com> Cc: Marcelo Tosatti <marcelo@kvack.org> Cc: Eric Dumazet <dada1@cosmosbay.com> Cc: "Paul E. McKenney" <paulmck@us.ibm.com> Cc: Izik Eidus <izike@qumranet.com> Cc: Anthony Liguori <aliguori@us.ibm.com> Cc: Rik van Riel <riel@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
214 lines
6.3 KiB
Plaintext
214 lines
6.3 KiB
Plaintext
config SELECT_MEMORY_MODEL
|
|
def_bool y
|
|
depends on EXPERIMENTAL || ARCH_SELECT_MEMORY_MODEL
|
|
|
|
choice
|
|
prompt "Memory model"
|
|
depends on SELECT_MEMORY_MODEL
|
|
default DISCONTIGMEM_MANUAL if ARCH_DISCONTIGMEM_DEFAULT
|
|
default SPARSEMEM_MANUAL if ARCH_SPARSEMEM_DEFAULT
|
|
default FLATMEM_MANUAL
|
|
|
|
config FLATMEM_MANUAL
|
|
bool "Flat Memory"
|
|
depends on !(ARCH_DISCONTIGMEM_ENABLE || ARCH_SPARSEMEM_ENABLE) || ARCH_FLATMEM_ENABLE
|
|
help
|
|
This option allows you to change some of the ways that
|
|
Linux manages its memory internally. Most users will
|
|
only have one option here: FLATMEM. This is normal
|
|
and a correct option.
|
|
|
|
Some users of more advanced features like NUMA and
|
|
memory hotplug may have different options here.
|
|
DISCONTIGMEM is an more mature, better tested system,
|
|
but is incompatible with memory hotplug and may suffer
|
|
decreased performance over SPARSEMEM. If unsure between
|
|
"Sparse Memory" and "Discontiguous Memory", choose
|
|
"Discontiguous Memory".
|
|
|
|
If unsure, choose this option (Flat Memory) over any other.
|
|
|
|
config DISCONTIGMEM_MANUAL
|
|
bool "Discontiguous Memory"
|
|
depends on ARCH_DISCONTIGMEM_ENABLE
|
|
help
|
|
This option provides enhanced support for discontiguous
|
|
memory systems, over FLATMEM. These systems have holes
|
|
in their physical address spaces, and this option provides
|
|
more efficient handling of these holes. However, the vast
|
|
majority of hardware has quite flat address spaces, and
|
|
can have degraded performance from the extra overhead that
|
|
this option imposes.
|
|
|
|
Many NUMA configurations will have this as the only option.
|
|
|
|
If unsure, choose "Flat Memory" over this option.
|
|
|
|
config SPARSEMEM_MANUAL
|
|
bool "Sparse Memory"
|
|
depends on ARCH_SPARSEMEM_ENABLE
|
|
help
|
|
This will be the only option for some systems, including
|
|
memory hotplug systems. This is normal.
|
|
|
|
For many other systems, this will be an alternative to
|
|
"Discontiguous Memory". This option provides some potential
|
|
performance benefits, along with decreased code complexity,
|
|
but it is newer, and more experimental.
|
|
|
|
If unsure, choose "Discontiguous Memory" or "Flat Memory"
|
|
over this option.
|
|
|
|
endchoice
|
|
|
|
config DISCONTIGMEM
|
|
def_bool y
|
|
depends on (!SELECT_MEMORY_MODEL && ARCH_DISCONTIGMEM_ENABLE) || DISCONTIGMEM_MANUAL
|
|
|
|
config SPARSEMEM
|
|
def_bool y
|
|
depends on SPARSEMEM_MANUAL
|
|
|
|
config FLATMEM
|
|
def_bool y
|
|
depends on (!DISCONTIGMEM && !SPARSEMEM) || FLATMEM_MANUAL
|
|
|
|
config FLAT_NODE_MEM_MAP
|
|
def_bool y
|
|
depends on !SPARSEMEM
|
|
|
|
config HAVE_GET_USER_PAGES_FAST
|
|
bool
|
|
|
|
#
|
|
# Both the NUMA code and DISCONTIGMEM use arrays of pg_data_t's
|
|
# to represent different areas of memory. This variable allows
|
|
# those dependencies to exist individually.
|
|
#
|
|
config NEED_MULTIPLE_NODES
|
|
def_bool y
|
|
depends on DISCONTIGMEM || NUMA
|
|
|
|
config HAVE_MEMORY_PRESENT
|
|
def_bool y
|
|
depends on ARCH_HAVE_MEMORY_PRESENT || SPARSEMEM
|
|
|
|
#
|
|
# SPARSEMEM_EXTREME (which is the default) does some bootmem
|
|
# allocations when memory_present() is called. If this cannot
|
|
# be done on your architecture, select this option. However,
|
|
# statically allocating the mem_section[] array can potentially
|
|
# consume vast quantities of .bss, so be careful.
|
|
#
|
|
# This option will also potentially produce smaller runtime code
|
|
# with gcc 3.4 and later.
|
|
#
|
|
config SPARSEMEM_STATIC
|
|
def_bool n
|
|
|
|
#
|
|
# Architecture platforms which require a two level mem_section in SPARSEMEM
|
|
# must select this option. This is usually for architecture platforms with
|
|
# an extremely sparse physical address space.
|
|
#
|
|
config SPARSEMEM_EXTREME
|
|
def_bool y
|
|
depends on SPARSEMEM && !SPARSEMEM_STATIC
|
|
|
|
config SPARSEMEM_VMEMMAP_ENABLE
|
|
def_bool n
|
|
|
|
config SPARSEMEM_VMEMMAP
|
|
bool "Sparse Memory virtual memmap"
|
|
depends on SPARSEMEM && SPARSEMEM_VMEMMAP_ENABLE
|
|
default y
|
|
help
|
|
SPARSEMEM_VMEMMAP uses a virtually mapped memmap to optimise
|
|
pfn_to_page and page_to_pfn operations. This is the most
|
|
efficient option when sufficient kernel resources are available.
|
|
|
|
# eventually, we can have this option just 'select SPARSEMEM'
|
|
config MEMORY_HOTPLUG
|
|
bool "Allow for memory hot-add"
|
|
depends on SPARSEMEM || X86_64_ACPI_NUMA
|
|
depends on HOTPLUG && !HIBERNATION && ARCH_ENABLE_MEMORY_HOTPLUG
|
|
depends on (IA64 || X86 || PPC64 || SUPERH || S390)
|
|
|
|
comment "Memory hotplug is currently incompatible with Software Suspend"
|
|
depends on SPARSEMEM && HOTPLUG && HIBERNATION
|
|
|
|
config MEMORY_HOTPLUG_SPARSE
|
|
def_bool y
|
|
depends on SPARSEMEM && MEMORY_HOTPLUG
|
|
|
|
config MEMORY_HOTREMOVE
|
|
bool "Allow for memory hot remove"
|
|
depends on MEMORY_HOTPLUG && ARCH_ENABLE_MEMORY_HOTREMOVE
|
|
depends on MIGRATION
|
|
|
|
#
|
|
# If we have space for more page flags then we can enable additional
|
|
# optimizations and functionality.
|
|
#
|
|
# Regular Sparsemem takes page flag bits for the sectionid if it does not
|
|
# use a virtual memmap. Disable extended page flags for 32 bit platforms
|
|
# that require the use of a sectionid in the page flags.
|
|
#
|
|
config PAGEFLAGS_EXTENDED
|
|
def_bool y
|
|
depends on 64BIT || SPARSEMEM_VMEMMAP || !NUMA || !SPARSEMEM
|
|
|
|
# Heavily threaded applications may benefit from splitting the mm-wide
|
|
# page_table_lock, so that faults on different parts of the user address
|
|
# space can be handled with less contention: split it at this NR_CPUS.
|
|
# Default to 4 for wider testing, though 8 might be more appropriate.
|
|
# ARM's adjust_pte (unused if VIPT) depends on mm-wide page_table_lock.
|
|
# PA-RISC 7xxx's spinlock_t would enlarge struct page from 32 to 44 bytes.
|
|
#
|
|
config SPLIT_PTLOCK_CPUS
|
|
int
|
|
default "4096" if ARM && !CPU_CACHE_VIPT
|
|
default "4096" if PARISC && !PA20
|
|
default "4"
|
|
|
|
#
|
|
# support for page migration
|
|
#
|
|
config MIGRATION
|
|
bool "Page migration"
|
|
def_bool y
|
|
depends on NUMA || ARCH_ENABLE_MEMORY_HOTREMOVE
|
|
help
|
|
Allows the migration of the physical location of pages of processes
|
|
while the virtual addresses are not changed. This is useful for
|
|
example on NUMA systems to put pages nearer to the processors accessing
|
|
the page.
|
|
|
|
config RESOURCES_64BIT
|
|
bool "64 bit Memory and IO resources (EXPERIMENTAL)" if (!64BIT && EXPERIMENTAL)
|
|
default 64BIT
|
|
help
|
|
This option allows memory and IO resources to be 64 bit.
|
|
|
|
config ZONE_DMA_FLAG
|
|
int
|
|
default "0" if !ZONE_DMA
|
|
default "1"
|
|
|
|
config BOUNCE
|
|
def_bool y
|
|
depends on BLOCK && MMU && (ZONE_DMA || HIGHMEM)
|
|
|
|
config NR_QUICK
|
|
int
|
|
depends on QUICKLIST
|
|
default "2" if SUPERH || AVR32
|
|
default "1"
|
|
|
|
config VIRT_TO_BUS
|
|
def_bool y
|
|
depends on !ARCH_NO_VIRT_TO_BUS
|
|
|
|
config MMU_NOTIFIER
|
|
bool
|