2009-02-08 23:52:14 -07:00
|
|
|
#ifndef _ASM_X86_PAGE_64_DEFS_H
|
|
|
|
|
#define _ASM_X86_PAGE_64_DEFS_H
|
|
|
|
|
|
x86_64: expand kernel stack to 16K
While I play inhouse patches with much memory pressure on qemu-kvm,
3.14 kernel was randomly crashed. The reason was kernel stack overflow.
When I investigated the problem, the callstack was a little bit deeper
by involve with reclaim functions but not direct reclaim path.
I tried to diet stack size of some functions related with alloc/reclaim
so did a hundred of byte but overflow was't disappeard so that I encounter
overflow by another deeper callstack on reclaim/allocator path.
Of course, we might sweep every sites we have found for reducing
stack usage but I'm not sure how long it saves the world(surely,
lots of developer start to add nice features which will use stack
agains) and if we consider another more complex feature in I/O layer
and/or reclaim path, it might be better to increase stack size(
meanwhile, stack usage on 64bit machine was doubled compared to 32bit
while it have sticked to 8K. Hmm, it's not a fair to me and arm64
already expaned to 16K. )
So, my stupid idea is just let's expand stack size and keep an eye
toward stack consumption on each kernel functions via stacktrace of ftrace.
For example, we can have a bar like that each funcion shouldn't exceed 200K
and emit the warning when some function consumes more in runtime.
Of course, it could make false positive but at least, it could make a
chance to think over it.
I guess this topic was discussed several time so there might be
strong reason not to increase kernel stack size on x86_64, for me not
knowing so Ccing x86_64 maintainers, other MM guys and virtio
maintainers.
Here's an example call trace using up the kernel stack:
Depth Size Location (51 entries)
----- ---- --------
0) 7696 16 lookup_address
1) 7680 16 _lookup_address_cpa.isra.3
2) 7664 24 __change_page_attr_set_clr
3) 7640 392 kernel_map_pages
4) 7248 256 get_page_from_freelist
5) 6992 352 __alloc_pages_nodemask
6) 6640 8 alloc_pages_current
7) 6632 168 new_slab
8) 6464 8 __slab_alloc
9) 6456 80 __kmalloc
10) 6376 376 vring_add_indirect
11) 6000 144 virtqueue_add_sgs
12) 5856 288 __virtblk_add_req
13) 5568 96 virtio_queue_rq
14) 5472 128 __blk_mq_run_hw_queue
15) 5344 16 blk_mq_run_hw_queue
16) 5328 96 blk_mq_insert_requests
17) 5232 112 blk_mq_flush_plug_list
18) 5120 112 blk_flush_plug_list
19) 5008 64 io_schedule_timeout
20) 4944 128 mempool_alloc
21) 4816 96 bio_alloc_bioset
22) 4720 48 get_swap_bio
23) 4672 160 __swap_writepage
24) 4512 32 swap_writepage
25) 4480 320 shrink_page_list
26) 4160 208 shrink_inactive_list
27) 3952 304 shrink_lruvec
28) 3648 80 shrink_zone
29) 3568 128 do_try_to_free_pages
30) 3440 208 try_to_free_pages
31) 3232 352 __alloc_pages_nodemask
32) 2880 8 alloc_pages_current
33) 2872 200 __page_cache_alloc
34) 2672 80 find_or_create_page
35) 2592 80 ext4_mb_load_buddy
36) 2512 176 ext4_mb_regular_allocator
37) 2336 128 ext4_mb_new_blocks
38) 2208 256 ext4_ext_map_blocks
39) 1952 160 ext4_map_blocks
40) 1792 384 ext4_writepages
41) 1408 16 do_writepages
42) 1392 96 __writeback_single_inode
43) 1296 176 writeback_sb_inodes
44) 1120 80 __writeback_inodes_wb
45) 1040 160 wb_writeback
46) 880 208 bdi_writeback_workfn
47) 672 144 process_one_work
48) 528 112 worker_thread
49) 416 240 kthread
50) 176 176 ret_from_fork
[ Note: the problem is exacerbated by certain gcc versions that seem to
generate much bigger stack frames due to apparently bad coalescing of
temporaries and generating too many spills. Rusty saw gcc-4.6.4 using
35% more stack on the virtio path than 4.8.2 does, for example.
Minchan not only uses such a bad gcc version (4.6.3 in his case), but
some of the stack use is due to debugging (CONFIG_DEBUG_PAGEALLOC is
what causes that kernel_map_pages() frame, for example). But we're
clearly getting too close.
The VM code also seems to have excessive stack frames partly for the
same compiler reason, triggered by excessive inlining and lots of
function arguments.
We need to improve on our stack use, but in the meantime let's do this
simple stack increase too. Unlike most earlier reports, there is
nothing simple that stands out as being really horribly wrong here,
apart from the fact that the stack frames are just bigger than they
should need to be. - Linus ]
Signed-off-by: Minchan Kim <minchan@kernel.org>
Cc: Peter Anvin <hpa@zytor.com>
Cc: Dave Chinner <david@fromorbit.com>
Cc: Dave Jones <davej@redhat.com>
Cc: Jens Axboe <axboe@kernel.dk>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Rik van Riel <riel@redhat.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Michael S Tsirkin <mst@redhat.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: PJ Waskiewicz <pjwaskiewicz@gmail.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-05-28 00:53:59 -06:00
|
|
|
#define THREAD_SIZE_ORDER 2
|
2012-05-05 09:05:42 -06:00
|
|
|
#define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER)
|
2009-02-08 23:52:14 -07:00
|
|
|
#define CURRENT_MASK (~(THREAD_SIZE - 1))
|
|
|
|
|
|
|
|
|
|
#define EXCEPTION_STACK_ORDER 0
|
|
|
|
|
#define EXCEPTION_STKSZ (PAGE_SIZE << EXCEPTION_STACK_ORDER)
|
|
|
|
|
|
|
|
|
|
#define DEBUG_STACK_ORDER (EXCEPTION_STACK_ORDER + 1)
|
|
|
|
|
#define DEBUG_STKSZ (PAGE_SIZE << DEBUG_STACK_ORDER)
|
|
|
|
|
|
|
|
|
|
#define IRQ_STACK_ORDER 2
|
|
|
|
|
#define IRQ_STACK_SIZE (PAGE_SIZE << IRQ_STACK_ORDER)
|
|
|
|
|
|
|
|
|
|
#define STACKFAULT_STACK 1
|
|
|
|
|
#define DOUBLEFAULT_STACK 2
|
|
|
|
|
#define NMI_STACK 3
|
|
|
|
|
#define DEBUG_STACK 4
|
|
|
|
|
#define MCE_STACK 5
|
|
|
|
|
#define N_EXCEPTION_STACKS 5 /* hw limit: 7 */
|
|
|
|
|
|
|
|
|
|
#define PUD_PAGE_SIZE (_AC(1, UL) << PUD_SHIFT)
|
|
|
|
|
#define PUD_PAGE_MASK (~(PUD_PAGE_SIZE-1))
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Set __PAGE_OFFSET to the most negative possible address +
|
|
|
|
|
* PGDIR_SIZE*16 (pgd slot 272). The gap is to allow a space for a
|
|
|
|
|
* hypervisor to fit. Choosing 16 slots here is arbitrary, but it's
|
|
|
|
|
* what Xen requires.
|
|
|
|
|
*/
|
|
|
|
|
#define __PAGE_OFFSET _AC(0xffff880000000000, UL)
|
|
|
|
|
|
|
|
|
|
#define __START_KERNEL_map _AC(0xffffffff80000000, UL)
|
|
|
|
|
|
2009-05-05 15:28:56 -06:00
|
|
|
/* See Documentation/x86/x86_64/mm.txt for a description of the memory map. */
|
2009-02-08 23:52:14 -07:00
|
|
|
#define __PHYSICAL_MASK_SHIFT 46
|
x86, 64-bit: Clean up user address masking
The discussion about using "access_ok()" in get_user_pages_fast() (see
commit 7f8189068726492950bf1a2dcfd9b51314560abf: "x86: don't use
'access_ok()' as a range check in get_user_pages_fast()" for details and
end result), made us notice that x86-64 was really being very sloppy
about virtual address checking.
So be way more careful and straightforward about masking x86-64 virtual
addresses:
- All the VIRTUAL_MASK* variants now cover half of the address
space, it's not like we can use the full mask on a signed
integer, and the larger mask just invites mistakes when
applying it to either half of the 48-bit address space.
- /proc/kcore's kc_offset_to_vaddr() becomes a lot more
obvious when it transforms a file offset into a
(kernel-half) virtual address.
- Unify/simplify the 32-bit and 64-bit USER_DS definition to
be based on TASK_SIZE_MAX.
This cleanup and more careful/obvious user virtual address checking also
uncovered a buglet in the x86-64 implementation of strnlen_user(): it
would do an "access_ok()" check on the whole potential area, even if the
string itself was much shorter, and thus return an error even for valid
strings. Our sloppy checking had hidden this.
So this fixes 'strnlen_user()' to do this properly, the same way we
already handled user strings in 'strncpy_from_user()'. Namely by just
checking the first byte, and then relying on fault handling for the
rest. That always works, since we impose a guard page that cannot be
mapped at the end of the user space address space (and even if we
didn't, we'd have the address space hole).
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Nick Piggin <npiggin@suse.de>
Cc: Hugh Dickins <hugh.dickins@tiscali.co.uk>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Alan Cox <alan@lxorguk.ukuu.org.uk>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2009-06-20 16:40:00 -06:00
|
|
|
#define __VIRTUAL_MASK_SHIFT 47
|
2009-02-08 23:52:14 -07:00
|
|
|
|
|
|
|
|
/*
|
2013-10-10 18:18:18 -06:00
|
|
|
* Kernel image size is limited to 1GiB due to the fixmap living in the
|
|
|
|
|
* next 1GiB (see level2_kernel_pgt in arch/x86/kernel/head_64.S). Use
|
|
|
|
|
* 512MiB by default, leaving 1.5GiB for modules once the page tables
|
|
|
|
|
* are fully set up. If kernel ASLR is configured, it can extend the
|
|
|
|
|
* kernel page table mapping, reducing the size of the modules area.
|
2009-02-08 23:52:14 -07:00
|
|
|
*/
|
2013-10-10 18:18:18 -06:00
|
|
|
#define KERNEL_IMAGE_SIZE_DEFAULT (512 * 1024 * 1024)
|
|
|
|
|
#if defined(CONFIG_RANDOMIZE_BASE) && \
|
|
|
|
|
CONFIG_RANDOMIZE_BASE_MAX_OFFSET > KERNEL_IMAGE_SIZE_DEFAULT
|
|
|
|
|
#define KERNEL_IMAGE_SIZE CONFIG_RANDOMIZE_BASE_MAX_OFFSET
|
|
|
|
|
#else
|
|
|
|
|
#define KERNEL_IMAGE_SIZE KERNEL_IMAGE_SIZE_DEFAULT
|
|
|
|
|
#endif
|
2009-02-08 23:52:14 -07:00
|
|
|
|
|
|
|
|
#endif /* _ASM_X86_PAGE_64_DEFS_H */
|