![Vlastimil Babka](/assets/img/avatar_default.png)
Patch series "try to reduce fragmenting fallbacks", v3. Last year, Johannes Weiner has reported a regression in page mobility grouping [1] and while the exact cause was not found, I've come up with some ways to improve it by reducing the number of allocations falling back to different migratetype and causing permanent fragmentation. The series was tested with mmtests stress-highalloc modified to do GFP_KERNEL order-4 allocations, on 4.9 with "mm, vmscan: fix zone balance check in prepare_kswapd_sleep" (without that, kcompactd indeed wasn't woken up) on UMA machine with 4GB memory. There were 5 repeats of each run, as the extfrag stats are quite volatile (note the stats below are sums, not averages, as it was less perl hacking for me). Success rate are the same, already high due to the low allocation order used, so I'm not including them. Compaction stats: (the patches are stacked, and I haven't measured the non-functional-changes patches separately) patch 1 patch 2 patch 3 patch 4 patch 7 patch 8 Compaction stalls 22449 24680 24846 19765 22059 17480 Compaction success 12971 14836 14608 10475 11632 8757 Compaction failures 9477 9843 10238 9290 10426 8722 Page migrate success 3109022 3370438 3312164 1695105 1608435 2111379 Page migrate failure 911588 1149065 1028264 1112675 1077251 1026367 Compaction pages isolated 7242983 8015530 7782467 4629063 4402787 5377665 Compaction migrate scanned 980838938 987367943 957690188 917647238 947155598 1018922197 Compaction free scanned 557926893 598946443 602236894 594024490 541169699 763651731 Compaction cost 10243 10578 10304 8286 8398 9440 Compaction stats are mostly within noise until patch 4, which decreases the number of compactions, and migrations. Part of that could be due to more pageblocks marked as unmovable, and async compaction skipping those. This changes a bit with patch 7, but not so much. Patch 8 increases free scanner stats and migrations, which comes from the changed termination criteria. Interestingly number of compactions decreases - probably the fully compacted pageblock satisfies multiple subsequent allocations, so it amortizes. Next comes the extfrag tracepoint, where "fragmenting" means that an allocation had to fallback to a pageblock of another migratetype which wasn't fully free (which is almost all of the fallbacks). I have locally added another tracepoint for "Page steal" into steal_suitable_fallback() which triggers in situations where we are allowed to do move_freepages_block(). If we decide to also do set_pageblock_migratetype(), it's "Pages steal with pageblock" with break down for which allocation migratetype we are stealing and from which fallback migratetype. The last part "due to counting" comes from patch 4 and counts the events where the counting of movable pages allowed us to change pageblock's migratetype, while the number of free pages alone wouldn't be enough to cross the threshold. patch 1 patch 2 patch 3 patch 4 patch 7 patch 8 Page alloc extfrag event 10155066 8522968 10164959 15622080 13727068 13140319 Extfrag fragmenting 10149231 8517025 10159040 15616925 13721391 13134792 Extfrag fragmenting for unmovable 159504 168500 184177 97835 70625 56948 Extfrag fragmenting unmovable placed with movable 153613 163549 172693 91740 64099 50917 Extfrag fragmenting unmovable placed with reclaim. 5891 4951 11484 6095 6526 6031 Extfrag fragmenting for reclaimable 4738 4829 6345 4822 5640 5378 Extfrag fragmenting reclaimable placed with movable 1836 1902 1851 1579 1739 1760 Extfrag fragmenting reclaimable placed with unmov. 2902 2927 4494 3243 3901 3618 Extfrag fragmenting for movable 9984989 8343696 9968518 15514268 13645126 13072466 Pages steal 179954 192291 210880 123254 94545 81486 Pages steal with pageblock 22153 18943 20154 33562 29969 33444 Pages steal with pageblock for unmovable 14350 12858 13256 20660 19003 20852 Pages steal with pageblock for unmovable from mov. 12812 11402 11683 19072 17467 19298 Pages steal with pageblock for unmovable from recl. 1538 1456 1573 1588 1536 1554 Pages steal with pageblock for movable 7114 5489 5965 11787 10012 11493 Pages steal with pageblock for movable from unmov. 6885 5291 5541 11179 9525 10885 Pages steal with pageblock for movable from recl. 229 198 424 608 487 608 Pages steal with pageblock for reclaimable 689 596 933 1115 954 1099 Pages steal with pageblock for reclaimable from unmov. 273 219 537 658 547 667 Pages steal with pageblock for reclaimable from mov. 416 377 396 457 407 432 Pages steal with pageblock due to counting 11834 10075 7530 ... for unmovable 8993 7381 4616 ... for movable 2792 2653 2851 ... for reclaimable 49 41 63 What we can see is that "Extfrag fragmenting for unmovable" and "... placed with movable" drops with almost each patch, which is good as we are polluting less movable pageblocks with unmovable pages. The most significant change is patch 4 with movable page counting. On the other hand it increases "Extfrag fragmenting for movable" by 50%. "Pages steal" drops though, so these movable allocation fallbacks find only small free pages and are not allowed to steal whole pageblocks back. "Pages steal with pageblock" raises, because the patch increases the chances of pageblock migratetype changes to happen. This affects all migratetypes. The summary is that patch 4 is not a clear win wrt these stats, but I believe that the tradeoff it makes is a good one. There's less pollution of movable pageblocks by unmovable allocations. There's less stealing between pageblock, and those that remain have higher chance of changing migratetype also the pageblock itself, so it should more faithfully reflect the migratetype of the pages within the pageblock. The increase of movable allocations falling back to unmovable pageblock might look dramatic, but those allocations can be migrated by compaction when needed, and other patches in the series (7-9) improve that aspect. Patches 7 and 8 continue the trend of reduced unmovable fallbacks and also reduce the impact on movable fallbacks from patch 4. [1] https://www.spinics.net/lists/linux-mm/msg114237.html This patch (of 8): While currently there are (mostly by accident) no holes in struct compact_control (on x86_64), but we are going to add more bool flags, so place them all together to the end of the structure. While at it, just order all fields from largest to smallest. Link: http://lkml.kernel.org/r/20170307131545.28577-2-vbabka@suse.cz Signed-off-by: Vlastimil Babka <vbabka@suse.cz> Acked-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com> Cc: David Rientjes <rientjes@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
524 lines
16 KiB
C
524 lines
16 KiB
C
/* internal.h: mm/ internal definitions
|
|
*
|
|
* Copyright (C) 2004 Red Hat, Inc. All Rights Reserved.
|
|
* Written by David Howells (dhowells@redhat.com)
|
|
*
|
|
* This program is free software; you can redistribute it and/or
|
|
* modify it under the terms of the GNU General Public License
|
|
* as published by the Free Software Foundation; either version
|
|
* 2 of the License, or (at your option) any later version.
|
|
*/
|
|
#ifndef __MM_INTERNAL_H
|
|
#define __MM_INTERNAL_H
|
|
|
|
#include <linux/fs.h>
|
|
#include <linux/mm.h>
|
|
#include <linux/pagemap.h>
|
|
#include <linux/tracepoint-defs.h>
|
|
|
|
/*
|
|
* The set of flags that only affect watermark checking and reclaim
|
|
* behaviour. This is used by the MM to obey the caller constraints
|
|
* about IO, FS and watermark checking while ignoring placement
|
|
* hints such as HIGHMEM usage.
|
|
*/
|
|
#define GFP_RECLAIM_MASK (__GFP_RECLAIM|__GFP_HIGH|__GFP_IO|__GFP_FS|\
|
|
__GFP_NOWARN|__GFP_REPEAT|__GFP_NOFAIL|\
|
|
__GFP_NORETRY|__GFP_MEMALLOC|__GFP_NOMEMALLOC|\
|
|
__GFP_ATOMIC)
|
|
|
|
/* The GFP flags allowed during early boot */
|
|
#define GFP_BOOT_MASK (__GFP_BITS_MASK & ~(__GFP_RECLAIM|__GFP_IO|__GFP_FS))
|
|
|
|
/* Control allocation cpuset and node placement constraints */
|
|
#define GFP_CONSTRAINT_MASK (__GFP_HARDWALL|__GFP_THISNODE)
|
|
|
|
/* Do not use these with a slab allocator */
|
|
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
|
|
|
|
void page_writeback_init(void);
|
|
|
|
int do_swap_page(struct vm_fault *vmf);
|
|
|
|
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
|
|
unsigned long floor, unsigned long ceiling);
|
|
|
|
static inline bool can_madv_dontneed_vma(struct vm_area_struct *vma)
|
|
{
|
|
return !(vma->vm_flags & (VM_LOCKED|VM_HUGETLB|VM_PFNMAP));
|
|
}
|
|
|
|
void unmap_page_range(struct mmu_gather *tlb,
|
|
struct vm_area_struct *vma,
|
|
unsigned long addr, unsigned long end,
|
|
struct zap_details *details);
|
|
|
|
extern int __do_page_cache_readahead(struct address_space *mapping,
|
|
struct file *filp, pgoff_t offset, unsigned long nr_to_read,
|
|
unsigned long lookahead_size);
|
|
|
|
/*
|
|
* Submit IO for the read-ahead request in file_ra_state.
|
|
*/
|
|
static inline unsigned long ra_submit(struct file_ra_state *ra,
|
|
struct address_space *mapping, struct file *filp)
|
|
{
|
|
return __do_page_cache_readahead(mapping, filp,
|
|
ra->start, ra->size, ra->async_size);
|
|
}
|
|
|
|
/*
|
|
* Turn a non-refcounted page (->_refcount == 0) into refcounted with
|
|
* a count of one.
|
|
*/
|
|
static inline void set_page_refcounted(struct page *page)
|
|
{
|
|
VM_BUG_ON_PAGE(PageTail(page), page);
|
|
VM_BUG_ON_PAGE(page_ref_count(page), page);
|
|
set_page_count(page, 1);
|
|
}
|
|
|
|
extern unsigned long highest_memmap_pfn;
|
|
|
|
/*
|
|
* Maximum number of reclaim retries without progress before the OOM
|
|
* killer is consider the only way forward.
|
|
*/
|
|
#define MAX_RECLAIM_RETRIES 16
|
|
|
|
/*
|
|
* in mm/vmscan.c:
|
|
*/
|
|
extern int isolate_lru_page(struct page *page);
|
|
extern void putback_lru_page(struct page *page);
|
|
|
|
/*
|
|
* in mm/rmap.c:
|
|
*/
|
|
extern pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);
|
|
|
|
/*
|
|
* in mm/page_alloc.c
|
|
*/
|
|
|
|
/*
|
|
* Structure for holding the mostly immutable allocation parameters passed
|
|
* between functions involved in allocations, including the alloc_pages*
|
|
* family of functions.
|
|
*
|
|
* nodemask, migratetype and high_zoneidx are initialized only once in
|
|
* __alloc_pages_nodemask() and then never change.
|
|
*
|
|
* zonelist, preferred_zone and classzone_idx are set first in
|
|
* __alloc_pages_nodemask() for the fast path, and might be later changed
|
|
* in __alloc_pages_slowpath(). All other functions pass the whole strucure
|
|
* by a const pointer.
|
|
*/
|
|
struct alloc_context {
|
|
struct zonelist *zonelist;
|
|
nodemask_t *nodemask;
|
|
struct zoneref *preferred_zoneref;
|
|
int migratetype;
|
|
enum zone_type high_zoneidx;
|
|
bool spread_dirty_pages;
|
|
};
|
|
|
|
#define ac_classzone_idx(ac) zonelist_zone_idx(ac->preferred_zoneref)
|
|
|
|
/*
|
|
* Locate the struct page for both the matching buddy in our
|
|
* pair (buddy1) and the combined O(n+1) page they form (page).
|
|
*
|
|
* 1) Any buddy B1 will have an order O twin B2 which satisfies
|
|
* the following equation:
|
|
* B2 = B1 ^ (1 << O)
|
|
* For example, if the starting buddy (buddy2) is #8 its order
|
|
* 1 buddy is #10:
|
|
* B2 = 8 ^ (1 << 1) = 8 ^ 2 = 10
|
|
*
|
|
* 2) Any buddy B will have an order O+1 parent P which
|
|
* satisfies the following equation:
|
|
* P = B & ~(1 << O)
|
|
*
|
|
* Assumption: *_mem_map is contiguous at least up to MAX_ORDER
|
|
*/
|
|
static inline unsigned long
|
|
__find_buddy_pfn(unsigned long page_pfn, unsigned int order)
|
|
{
|
|
return page_pfn ^ (1 << order);
|
|
}
|
|
|
|
extern struct page *__pageblock_pfn_to_page(unsigned long start_pfn,
|
|
unsigned long end_pfn, struct zone *zone);
|
|
|
|
static inline struct page *pageblock_pfn_to_page(unsigned long start_pfn,
|
|
unsigned long end_pfn, struct zone *zone)
|
|
{
|
|
if (zone->contiguous)
|
|
return pfn_to_page(start_pfn);
|
|
|
|
return __pageblock_pfn_to_page(start_pfn, end_pfn, zone);
|
|
}
|
|
|
|
extern int __isolate_free_page(struct page *page, unsigned int order);
|
|
extern void __free_pages_bootmem(struct page *page, unsigned long pfn,
|
|
unsigned int order);
|
|
extern void prep_compound_page(struct page *page, unsigned int order);
|
|
extern void post_alloc_hook(struct page *page, unsigned int order,
|
|
gfp_t gfp_flags);
|
|
extern int user_min_free_kbytes;
|
|
|
|
#if defined CONFIG_COMPACTION || defined CONFIG_CMA
|
|
|
|
/*
|
|
* in mm/compaction.c
|
|
*/
|
|
/*
|
|
* compact_control is used to track pages being migrated and the free pages
|
|
* they are being migrated to during memory compaction. The free_pfn starts
|
|
* at the end of a zone and migrate_pfn begins at the start. Movable pages
|
|
* are moved to the end of a zone during a compaction run and the run
|
|
* completes when free_pfn <= migrate_pfn
|
|
*/
|
|
struct compact_control {
|
|
struct list_head freepages; /* List of free pages to migrate to */
|
|
struct list_head migratepages; /* List of pages being migrated */
|
|
struct zone *zone;
|
|
unsigned long nr_freepages; /* Number of isolated free pages */
|
|
unsigned long nr_migratepages; /* Number of pages to migrate */
|
|
unsigned long total_migrate_scanned;
|
|
unsigned long total_free_scanned;
|
|
unsigned long free_pfn; /* isolate_freepages search base */
|
|
unsigned long migrate_pfn; /* isolate_migratepages search base */
|
|
unsigned long last_migrated_pfn;/* Not yet flushed page being freed */
|
|
const gfp_t gfp_mask; /* gfp mask of a direct compactor */
|
|
int order; /* order a direct compactor needs */
|
|
const unsigned int alloc_flags; /* alloc flags of a direct compactor */
|
|
const int classzone_idx; /* zone index of a direct compactor */
|
|
enum migrate_mode mode; /* Async or sync migration mode */
|
|
bool ignore_skip_hint; /* Scan blocks even if marked skip */
|
|
bool ignore_block_suitable; /* Scan blocks considered unsuitable */
|
|
bool direct_compaction; /* False from kcompactd or /proc/... */
|
|
bool whole_zone; /* Whole zone should/has been scanned */
|
|
bool contended; /* Signal lock or sched contention */
|
|
};
|
|
|
|
unsigned long
|
|
isolate_freepages_range(struct compact_control *cc,
|
|
unsigned long start_pfn, unsigned long end_pfn);
|
|
unsigned long
|
|
isolate_migratepages_range(struct compact_control *cc,
|
|
unsigned long low_pfn, unsigned long end_pfn);
|
|
int find_suitable_fallback(struct free_area *area, unsigned int order,
|
|
int migratetype, bool only_stealable, bool *can_steal);
|
|
|
|
#endif
|
|
|
|
/*
|
|
* This function returns the order of a free page in the buddy system. In
|
|
* general, page_zone(page)->lock must be held by the caller to prevent the
|
|
* page from being allocated in parallel and returning garbage as the order.
|
|
* If a caller does not hold page_zone(page)->lock, it must guarantee that the
|
|
* page cannot be allocated or merged in parallel. Alternatively, it must
|
|
* handle invalid values gracefully, and use page_order_unsafe() below.
|
|
*/
|
|
static inline unsigned int page_order(struct page *page)
|
|
{
|
|
/* PageBuddy() must be checked by the caller */
|
|
return page_private(page);
|
|
}
|
|
|
|
/*
|
|
* Like page_order(), but for callers who cannot afford to hold the zone lock.
|
|
* PageBuddy() should be checked first by the caller to minimize race window,
|
|
* and invalid values must be handled gracefully.
|
|
*
|
|
* READ_ONCE is used so that if the caller assigns the result into a local
|
|
* variable and e.g. tests it for valid range before using, the compiler cannot
|
|
* decide to remove the variable and inline the page_private(page) multiple
|
|
* times, potentially observing different values in the tests and the actual
|
|
* use of the result.
|
|
*/
|
|
#define page_order_unsafe(page) READ_ONCE(page_private(page))
|
|
|
|
static inline bool is_cow_mapping(vm_flags_t flags)
|
|
{
|
|
return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE;
|
|
}
|
|
|
|
/*
|
|
* These three helpers classifies VMAs for virtual memory accounting.
|
|
*/
|
|
|
|
/*
|
|
* Executable code area - executable, not writable, not stack
|
|
*/
|
|
static inline bool is_exec_mapping(vm_flags_t flags)
|
|
{
|
|
return (flags & (VM_EXEC | VM_WRITE | VM_STACK)) == VM_EXEC;
|
|
}
|
|
|
|
/*
|
|
* Stack area - atomatically grows in one direction
|
|
*
|
|
* VM_GROWSUP / VM_GROWSDOWN VMAs are always private anonymous:
|
|
* do_mmap() forbids all other combinations.
|
|
*/
|
|
static inline bool is_stack_mapping(vm_flags_t flags)
|
|
{
|
|
return (flags & VM_STACK) == VM_STACK;
|
|
}
|
|
|
|
/*
|
|
* Data area - private, writable, not stack
|
|
*/
|
|
static inline bool is_data_mapping(vm_flags_t flags)
|
|
{
|
|
return (flags & (VM_WRITE | VM_SHARED | VM_STACK)) == VM_WRITE;
|
|
}
|
|
|
|
/* mm/util.c */
|
|
void __vma_link_list(struct mm_struct *mm, struct vm_area_struct *vma,
|
|
struct vm_area_struct *prev, struct rb_node *rb_parent);
|
|
|
|
#ifdef CONFIG_MMU
|
|
extern long populate_vma_page_range(struct vm_area_struct *vma,
|
|
unsigned long start, unsigned long end, int *nonblocking);
|
|
extern void munlock_vma_pages_range(struct vm_area_struct *vma,
|
|
unsigned long start, unsigned long end);
|
|
static inline void munlock_vma_pages_all(struct vm_area_struct *vma)
|
|
{
|
|
munlock_vma_pages_range(vma, vma->vm_start, vma->vm_end);
|
|
}
|
|
|
|
/*
|
|
* must be called with vma's mmap_sem held for read or write, and page locked.
|
|
*/
|
|
extern void mlock_vma_page(struct page *page);
|
|
extern unsigned int munlock_vma_page(struct page *page);
|
|
|
|
/*
|
|
* Clear the page's PageMlocked(). This can be useful in a situation where
|
|
* we want to unconditionally remove a page from the pagecache -- e.g.,
|
|
* on truncation or freeing.
|
|
*
|
|
* It is legal to call this function for any page, mlocked or not.
|
|
* If called for a page that is still mapped by mlocked vmas, all we do
|
|
* is revert to lazy LRU behaviour -- semantics are not broken.
|
|
*/
|
|
extern void clear_page_mlock(struct page *page);
|
|
|
|
/*
|
|
* mlock_migrate_page - called only from migrate_misplaced_transhuge_page()
|
|
* (because that does not go through the full procedure of migration ptes):
|
|
* to migrate the Mlocked page flag; update statistics.
|
|
*/
|
|
static inline void mlock_migrate_page(struct page *newpage, struct page *page)
|
|
{
|
|
if (TestClearPageMlocked(page)) {
|
|
int nr_pages = hpage_nr_pages(page);
|
|
|
|
/* Holding pmd lock, no change in irq context: __mod is safe */
|
|
__mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages);
|
|
SetPageMlocked(newpage);
|
|
__mod_zone_page_state(page_zone(newpage), NR_MLOCK, nr_pages);
|
|
}
|
|
}
|
|
|
|
extern pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma);
|
|
|
|
/*
|
|
* At what user virtual address is page expected in @vma?
|
|
*/
|
|
static inline unsigned long
|
|
__vma_address(struct page *page, struct vm_area_struct *vma)
|
|
{
|
|
pgoff_t pgoff = page_to_pgoff(page);
|
|
return vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
|
|
}
|
|
|
|
static inline unsigned long
|
|
vma_address(struct page *page, struct vm_area_struct *vma)
|
|
{
|
|
unsigned long start, end;
|
|
|
|
start = __vma_address(page, vma);
|
|
end = start + PAGE_SIZE * (hpage_nr_pages(page) - 1);
|
|
|
|
/* page should be within @vma mapping range */
|
|
VM_BUG_ON_VMA(end < vma->vm_start || start >= vma->vm_end, vma);
|
|
|
|
return max(start, vma->vm_start);
|
|
}
|
|
|
|
#else /* !CONFIG_MMU */
|
|
static inline void clear_page_mlock(struct page *page) { }
|
|
static inline void mlock_vma_page(struct page *page) { }
|
|
static inline void mlock_migrate_page(struct page *new, struct page *old) { }
|
|
|
|
#endif /* !CONFIG_MMU */
|
|
|
|
/*
|
|
* Return the mem_map entry representing the 'offset' subpage within
|
|
* the maximally aligned gigantic page 'base'. Handle any discontiguity
|
|
* in the mem_map at MAX_ORDER_NR_PAGES boundaries.
|
|
*/
|
|
static inline struct page *mem_map_offset(struct page *base, int offset)
|
|
{
|
|
if (unlikely(offset >= MAX_ORDER_NR_PAGES))
|
|
return nth_page(base, offset);
|
|
return base + offset;
|
|
}
|
|
|
|
/*
|
|
* Iterator over all subpages within the maximally aligned gigantic
|
|
* page 'base'. Handle any discontiguity in the mem_map.
|
|
*/
|
|
static inline struct page *mem_map_next(struct page *iter,
|
|
struct page *base, int offset)
|
|
{
|
|
if (unlikely((offset & (MAX_ORDER_NR_PAGES - 1)) == 0)) {
|
|
unsigned long pfn = page_to_pfn(base) + offset;
|
|
if (!pfn_valid(pfn))
|
|
return NULL;
|
|
return pfn_to_page(pfn);
|
|
}
|
|
return iter + 1;
|
|
}
|
|
|
|
/*
|
|
* FLATMEM and DISCONTIGMEM configurations use alloc_bootmem_node,
|
|
* so all functions starting at paging_init should be marked __init
|
|
* in those cases. SPARSEMEM, however, allows for memory hotplug,
|
|
* and alloc_bootmem_node is not used.
|
|
*/
|
|
#ifdef CONFIG_SPARSEMEM
|
|
#define __paginginit __meminit
|
|
#else
|
|
#define __paginginit __init
|
|
#endif
|
|
|
|
/* Memory initialisation debug and verification */
|
|
enum mminit_level {
|
|
MMINIT_WARNING,
|
|
MMINIT_VERIFY,
|
|
MMINIT_TRACE
|
|
};
|
|
|
|
#ifdef CONFIG_DEBUG_MEMORY_INIT
|
|
|
|
extern int mminit_loglevel;
|
|
|
|
#define mminit_dprintk(level, prefix, fmt, arg...) \
|
|
do { \
|
|
if (level < mminit_loglevel) { \
|
|
if (level <= MMINIT_WARNING) \
|
|
pr_warn("mminit::" prefix " " fmt, ##arg); \
|
|
else \
|
|
printk(KERN_DEBUG "mminit::" prefix " " fmt, ##arg); \
|
|
} \
|
|
} while (0)
|
|
|
|
extern void mminit_verify_pageflags_layout(void);
|
|
extern void mminit_verify_zonelist(void);
|
|
#else
|
|
|
|
static inline void mminit_dprintk(enum mminit_level level,
|
|
const char *prefix, const char *fmt, ...)
|
|
{
|
|
}
|
|
|
|
static inline void mminit_verify_pageflags_layout(void)
|
|
{
|
|
}
|
|
|
|
static inline void mminit_verify_zonelist(void)
|
|
{
|
|
}
|
|
#endif /* CONFIG_DEBUG_MEMORY_INIT */
|
|
|
|
/* mminit_validate_memmodel_limits is independent of CONFIG_DEBUG_MEMORY_INIT */
|
|
#if defined(CONFIG_SPARSEMEM)
|
|
extern void mminit_validate_memmodel_limits(unsigned long *start_pfn,
|
|
unsigned long *end_pfn);
|
|
#else
|
|
static inline void mminit_validate_memmodel_limits(unsigned long *start_pfn,
|
|
unsigned long *end_pfn)
|
|
{
|
|
}
|
|
#endif /* CONFIG_SPARSEMEM */
|
|
|
|
#define NODE_RECLAIM_NOSCAN -2
|
|
#define NODE_RECLAIM_FULL -1
|
|
#define NODE_RECLAIM_SOME 0
|
|
#define NODE_RECLAIM_SUCCESS 1
|
|
|
|
extern int hwpoison_filter(struct page *p);
|
|
|
|
extern u32 hwpoison_filter_dev_major;
|
|
extern u32 hwpoison_filter_dev_minor;
|
|
extern u64 hwpoison_filter_flags_mask;
|
|
extern u64 hwpoison_filter_flags_value;
|
|
extern u64 hwpoison_filter_memcg;
|
|
extern u32 hwpoison_filter_enable;
|
|
|
|
extern unsigned long __must_check vm_mmap_pgoff(struct file *, unsigned long,
|
|
unsigned long, unsigned long,
|
|
unsigned long, unsigned long);
|
|
|
|
extern void set_pageblock_order(void);
|
|
unsigned long reclaim_clean_pages_from_list(struct zone *zone,
|
|
struct list_head *page_list);
|
|
/* The ALLOC_WMARK bits are used as an index to zone->watermark */
|
|
#define ALLOC_WMARK_MIN WMARK_MIN
|
|
#define ALLOC_WMARK_LOW WMARK_LOW
|
|
#define ALLOC_WMARK_HIGH WMARK_HIGH
|
|
#define ALLOC_NO_WATERMARKS 0x04 /* don't check watermarks at all */
|
|
|
|
/* Mask to get the watermark bits */
|
|
#define ALLOC_WMARK_MASK (ALLOC_NO_WATERMARKS-1)
|
|
|
|
#define ALLOC_HARDER 0x10 /* try to alloc harder */
|
|
#define ALLOC_HIGH 0x20 /* __GFP_HIGH set */
|
|
#define ALLOC_CPUSET 0x40 /* check for correct cpuset */
|
|
#define ALLOC_CMA 0x80 /* allow allocations from CMA areas */
|
|
|
|
enum ttu_flags;
|
|
struct tlbflush_unmap_batch;
|
|
|
|
|
|
/*
|
|
* only for MM internal work items which do not depend on
|
|
* any allocations or locks which might depend on allocations
|
|
*/
|
|
extern struct workqueue_struct *mm_percpu_wq;
|
|
|
|
#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH
|
|
void try_to_unmap_flush(void);
|
|
void try_to_unmap_flush_dirty(void);
|
|
#else
|
|
static inline void try_to_unmap_flush(void)
|
|
{
|
|
}
|
|
static inline void try_to_unmap_flush_dirty(void)
|
|
{
|
|
}
|
|
|
|
#endif /* CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH */
|
|
|
|
extern const struct trace_print_flags pageflag_names[];
|
|
extern const struct trace_print_flags vmaflag_names[];
|
|
extern const struct trace_print_flags gfpflag_names[];
|
|
|
|
static inline bool is_migrate_highatomic(enum migratetype migratetype)
|
|
{
|
|
return migratetype == MIGRATE_HIGHATOMIC;
|
|
}
|
|
|
|
static inline bool is_migrate_highatomic_page(struct page *page)
|
|
{
|
|
return get_pageblock_migratetype(page) == MIGRATE_HIGHATOMIC;
|
|
}
|
|
|
|
#endif /* __MM_INTERNAL_H */
|