2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* kernel/sched.c
|
|
|
|
*
|
|
|
|
* Kernel scheduler and related syscalls
|
|
|
|
*
|
|
|
|
* Copyright (C) 1991-2002 Linus Torvalds
|
|
|
|
*
|
|
|
|
* 1996-12-23 Modified by Dave Grothe to fix bugs in semaphores and
|
|
|
|
* make semaphores SMP safe
|
|
|
|
* 1998-11-19 Implemented schedule_timeout() and related stuff
|
|
|
|
* by Andrea Arcangeli
|
|
|
|
* 2002-01-04 New ultra-scalable O(1) scheduler by Ingo Molnar:
|
|
|
|
* hybrid priority-list and round-robin design with
|
|
|
|
* an array-switch method of distributing timeslices
|
|
|
|
* and per-CPU runqueues. Cleanups and useful suggestions
|
|
|
|
* by Davide Libenzi, preemptible kernel bits by Robert Love.
|
|
|
|
* 2003-09-03 Interactivity tuning by Con Kolivas.
|
|
|
|
* 2004-04-02 Scheduler domains code by Nick Piggin
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/mm.h>
|
|
|
|
#include <linux/module.h>
|
|
|
|
#include <linux/nmi.h>
|
|
|
|
#include <linux/init.h>
|
|
|
|
#include <asm/uaccess.h>
|
|
|
|
#include <linux/highmem.h>
|
|
|
|
#include <linux/smp_lock.h>
|
|
|
|
#include <asm/mmu_context.h>
|
|
|
|
#include <linux/interrupt.h>
|
2006-01-11 13:17:46 -07:00
|
|
|
#include <linux/capability.h>
|
2005-04-16 16:20:36 -06:00
|
|
|
#include <linux/completion.h>
|
|
|
|
#include <linux/kernel_stat.h>
|
2006-07-03 01:24:33 -06:00
|
|
|
#include <linux/debug_locks.h>
|
2005-04-16 16:20:36 -06:00
|
|
|
#include <linux/security.h>
|
|
|
|
#include <linux/notifier.h>
|
|
|
|
#include <linux/profile.h>
|
|
|
|
#include <linux/suspend.h>
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
#include <linux/vmalloc.h>
|
2005-04-16 16:20:36 -06:00
|
|
|
#include <linux/blkdev.h>
|
|
|
|
#include <linux/delay.h>
|
|
|
|
#include <linux/smp.h>
|
|
|
|
#include <linux/threads.h>
|
|
|
|
#include <linux/timer.h>
|
|
|
|
#include <linux/rcupdate.h>
|
|
|
|
#include <linux/cpu.h>
|
|
|
|
#include <linux/cpuset.h>
|
|
|
|
#include <linux/percpu.h>
|
|
|
|
#include <linux/kthread.h>
|
|
|
|
#include <linux/seq_file.h>
|
|
|
|
#include <linux/syscalls.h>
|
|
|
|
#include <linux/times.h>
|
|
|
|
#include <linux/acct.h>
|
2006-03-26 02:38:20 -07:00
|
|
|
#include <linux/kprobes.h>
|
2006-07-14 01:24:37 -06:00
|
|
|
#include <linux/delayacct.h>
|
2005-04-16 16:20:36 -06:00
|
|
|
#include <asm/tlb.h>
|
|
|
|
|
|
|
|
#include <asm/unistd.h>
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Convert user-nice values [ -20 ... 0 ... 19 ]
|
|
|
|
* to static priority [ MAX_RT_PRIO..MAX_PRIO-1 ],
|
|
|
|
* and back.
|
|
|
|
*/
|
|
|
|
#define NICE_TO_PRIO(nice) (MAX_RT_PRIO + (nice) + 20)
|
|
|
|
#define PRIO_TO_NICE(prio) ((prio) - MAX_RT_PRIO - 20)
|
|
|
|
#define TASK_NICE(p) PRIO_TO_NICE((p)->static_prio)
|
|
|
|
|
|
|
|
/*
|
|
|
|
* 'User priority' is the nice value converted to something we
|
|
|
|
* can work with better when scaling various scheduler parameters,
|
|
|
|
* it's a [ 0 ... 39 ] range.
|
|
|
|
*/
|
|
|
|
#define USER_PRIO(p) ((p)-MAX_RT_PRIO)
|
|
|
|
#define TASK_USER_PRIO(p) USER_PRIO((p)->static_prio)
|
|
|
|
#define MAX_USER_PRIO (USER_PRIO(MAX_PRIO))
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Some helpers for converting nanosecond timing to jiffy resolution
|
|
|
|
*/
|
|
|
|
#define NS_TO_JIFFIES(TIME) ((TIME) / (1000000000 / HZ))
|
|
|
|
#define JIFFIES_TO_NS(TIME) ((TIME) * (1000000000 / HZ))
|
|
|
|
|
|
|
|
/*
|
|
|
|
* These are the 'tuning knobs' of the scheduler:
|
|
|
|
*
|
|
|
|
* Minimum timeslice is 5 msecs (or 1 jiffy, whichever is larger),
|
|
|
|
* default timeslice is 100 msecs, maximum timeslice is 800 msecs.
|
|
|
|
* Timeslices get refilled after they expire.
|
|
|
|
*/
|
|
|
|
#define MIN_TIMESLICE max(5 * HZ / 1000, 1)
|
|
|
|
#define DEF_TIMESLICE (100 * HZ / 1000)
|
|
|
|
#define ON_RUNQUEUE_WEIGHT 30
|
|
|
|
#define CHILD_PENALTY 95
|
|
|
|
#define PARENT_PENALTY 100
|
|
|
|
#define EXIT_WEIGHT 3
|
|
|
|
#define PRIO_BONUS_RATIO 25
|
|
|
|
#define MAX_BONUS (MAX_USER_PRIO * PRIO_BONUS_RATIO / 100)
|
|
|
|
#define INTERACTIVE_DELTA 2
|
|
|
|
#define MAX_SLEEP_AVG (DEF_TIMESLICE * MAX_BONUS)
|
|
|
|
#define STARVATION_LIMIT (MAX_SLEEP_AVG)
|
|
|
|
#define NS_MAX_SLEEP_AVG (JIFFIES_TO_NS(MAX_SLEEP_AVG))
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If a task is 'interactive' then we reinsert it in the active
|
|
|
|
* array after it has expired its current timeslice. (it will not
|
|
|
|
* continue to run immediately, it will still roundrobin with
|
|
|
|
* other interactive tasks.)
|
|
|
|
*
|
|
|
|
* This part scales the interactivity limit depending on niceness.
|
|
|
|
*
|
|
|
|
* We scale it linearly, offset by the INTERACTIVE_DELTA delta.
|
|
|
|
* Here are a few examples of different nice levels:
|
|
|
|
*
|
|
|
|
* TASK_INTERACTIVE(-20): [1,1,1,1,1,1,1,1,1,0,0]
|
|
|
|
* TASK_INTERACTIVE(-10): [1,1,1,1,1,1,1,0,0,0,0]
|
|
|
|
* TASK_INTERACTIVE( 0): [1,1,1,1,0,0,0,0,0,0,0]
|
|
|
|
* TASK_INTERACTIVE( 10): [1,1,0,0,0,0,0,0,0,0,0]
|
|
|
|
* TASK_INTERACTIVE( 19): [0,0,0,0,0,0,0,0,0,0,0]
|
|
|
|
*
|
|
|
|
* (the X axis represents the possible -5 ... 0 ... +5 dynamic
|
|
|
|
* priority range a task can explore, a value of '1' means the
|
|
|
|
* task is rated interactive.)
|
|
|
|
*
|
|
|
|
* Ie. nice +19 tasks can never get 'interactive' enough to be
|
|
|
|
* reinserted into the active array. And only heavily CPU-hog nice -20
|
|
|
|
* tasks will be expired. Default nice 0 tasks are somewhere between,
|
|
|
|
* it takes some effort for them to get interactive, but it's not
|
|
|
|
* too hard.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#define CURRENT_BONUS(p) \
|
|
|
|
(NS_TO_JIFFIES((p)->sleep_avg) * MAX_BONUS / \
|
|
|
|
MAX_SLEEP_AVG)
|
|
|
|
|
|
|
|
#define GRANULARITY (10 * HZ / 1000 ? : 1)
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
#define TIMESLICE_GRANULARITY(p) (GRANULARITY * \
|
|
|
|
(1 << (((MAX_BONUS - CURRENT_BONUS(p)) ? : 1) - 1)) * \
|
|
|
|
num_online_cpus())
|
|
|
|
#else
|
|
|
|
#define TIMESLICE_GRANULARITY(p) (GRANULARITY * \
|
|
|
|
(1 << (((MAX_BONUS - CURRENT_BONUS(p)) ? : 1) - 1)))
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#define SCALE(v1,v1_max,v2_max) \
|
|
|
|
(v1) * (v2_max) / (v1_max)
|
|
|
|
|
|
|
|
#define DELTA(p) \
|
[PATCH] sched: fix task interactivity calculation
Is a truncation error in kernel/sched.c triggered when the nice value is
negative. The affected code is used in the TASK_INTERACTIVE macro.
The code is:
#define SCALE(v1,v1_max,v2_max) \
(v1) * (v2_max) / (v1_max)
which is used in this way:
SCALE(TASK_NICE(p), 40, MAX_BONUS)
Comments in the code says:
* This part scales the interactivity limit depending on niceness.
*
* We scale it linearly, offset by the INTERACTIVE_DELTA delta.
* Here are a few examples of different nice levels:
*
* TASK_INTERACTIVE(-20): [1,1,1,1,1,1,1,1,1,0,0]
* TASK_INTERACTIVE(-10): [1,1,1,1,1,1,1,0,0,0,0]
* TASK_INTERACTIVE( 0): [1,1,1,1,0,0,0,0,0,0,0]
* TASK_INTERACTIVE( 10): [1,1,0,0,0,0,0,0,0,0,0]
* TASK_INTERACTIVE( 19): [0,0,0,0,0,0,0,0,0,0,0]
*
* (the X axis represents the possible -5 ... 0 ... +5 dynamic
* priority range a task can explore, a value of '1' means the
* task is rated interactive.)
However, the current code does not scale it linearly and the result differs
from the given examples. If the mathematical function "floor" is used when
the nice value is negative instead of the truncation one gets when using
integer division, the result conforms to the documentation.
Output of TASK_INTERACTIVE when using the kernel code:
nice dynamic priorities
-20 1 1 1 1 1 1 1 1 1 0 0
-19 1 1 1 1 1 1 1 1 0 0 0
-18 1 1 1 1 1 1 1 1 0 0 0
-17 1 1 1 1 1 1 1 1 0 0 0
-16 1 1 1 1 1 1 1 1 0 0 0
-15 1 1 1 1 1 1 1 0 0 0 0
-14 1 1 1 1 1 1 1 0 0 0 0
-13 1 1 1 1 1 1 1 0 0 0 0
-12 1 1 1 1 1 1 1 0 0 0 0
-11 1 1 1 1 1 1 0 0 0 0 0
-10 1 1 1 1 1 1 0 0 0 0 0
-9 1 1 1 1 1 1 0 0 0 0 0
-8 1 1 1 1 1 1 0 0 0 0 0
-7 1 1 1 1 1 0 0 0 0 0 0
-6 1 1 1 1 1 0 0 0 0 0 0
-5 1 1 1 1 1 0 0 0 0 0 0
-4 1 1 1 1 1 0 0 0 0 0 0
-3 1 1 1 1 0 0 0 0 0 0 0
-2 1 1 1 1 0 0 0 0 0 0 0
-1 1 1 1 1 0 0 0 0 0 0 0
0 1 1 1 1 0 0 0 0 0 0 0
1 1 1 1 1 0 0 0 0 0 0 0
2 1 1 1 1 0 0 0 0 0 0 0
3 1 1 1 1 0 0 0 0 0 0 0
4 1 1 1 0 0 0 0 0 0 0 0
5 1 1 1 0 0 0 0 0 0 0 0
6 1 1 1 0 0 0 0 0 0 0 0
7 1 1 1 0 0 0 0 0 0 0 0
8 1 1 0 0 0 0 0 0 0 0 0
9 1 1 0 0 0 0 0 0 0 0 0
10 1 1 0 0 0 0 0 0 0 0 0
11 1 1 0 0 0 0 0 0 0 0 0
12 1 0 0 0 0 0 0 0 0 0 0
13 1 0 0 0 0 0 0 0 0 0 0
14 1 0 0 0 0 0 0 0 0 0 0
15 1 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0
Output of TASK_INTERACTIVE when using "floor"
nice dynamic priorities
-20 1 1 1 1 1 1 1 1 1 0 0
-19 1 1 1 1 1 1 1 1 1 0 0
-18 1 1 1 1 1 1 1 1 1 0 0
-17 1 1 1 1 1 1 1 1 1 0 0
-16 1 1 1 1 1 1 1 1 0 0 0
-15 1 1 1 1 1 1 1 1 0 0 0
-14 1 1 1 1 1 1 1 1 0 0 0
-13 1 1 1 1 1 1 1 1 0 0 0
-12 1 1 1 1 1 1 1 0 0 0 0
-11 1 1 1 1 1 1 1 0 0 0 0
-10 1 1 1 1 1 1 1 0 0 0 0
-9 1 1 1 1 1 1 1 0 0 0 0
-8 1 1 1 1 1 1 0 0 0 0 0
-7 1 1 1 1 1 1 0 0 0 0 0
-6 1 1 1 1 1 1 0 0 0 0 0
-5 1 1 1 1 1 1 0 0 0 0 0
-4 1 1 1 1 1 0 0 0 0 0 0
-3 1 1 1 1 1 0 0 0 0 0 0
-2 1 1 1 1 1 0 0 0 0 0 0
-1 1 1 1 1 1 0 0 0 0 0 0
0 1 1 1 1 0 0 0 0 0 0 0
1 1 1 1 1 0 0 0 0 0 0 0
2 1 1 1 1 0 0 0 0 0 0 0
3 1 1 1 1 0 0 0 0 0 0 0
4 1 1 1 0 0 0 0 0 0 0 0
5 1 1 1 0 0 0 0 0 0 0 0
6 1 1 1 0 0 0 0 0 0 0 0
7 1 1 1 0 0 0 0 0 0 0 0
8 1 1 0 0 0 0 0 0 0 0 0
9 1 1 0 0 0 0 0 0 0 0 0
10 1 1 0 0 0 0 0 0 0 0 0
11 1 1 0 0 0 0 0 0 0 0 0
12 1 0 0 0 0 0 0 0 0 0 0
13 1 0 0 0 0 0 0 0 0 0 0
14 1 0 0 0 0 0 0 0 0 0 0
15 1 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0
Signed-off-by: Martin Andersson <martin.andersson@control.lth.se>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Williams <pwil3058@bigpond.net.au>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-03-27 02:15:18 -07:00
|
|
|
(SCALE(TASK_NICE(p) + 20, 40, MAX_BONUS) - 20 * MAX_BONUS / 40 + \
|
|
|
|
INTERACTIVE_DELTA)
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
#define TASK_INTERACTIVE(p) \
|
|
|
|
((p)->prio <= (p)->static_prio - DELTA(p))
|
|
|
|
|
|
|
|
#define INTERACTIVE_SLEEP(p) \
|
|
|
|
(JIFFIES_TO_NS(MAX_SLEEP_AVG * \
|
|
|
|
(MAX_BONUS / 2 + DELTA((p)) + 1) / MAX_BONUS - 1))
|
|
|
|
|
|
|
|
#define TASK_PREEMPTS_CURR(p, rq) \
|
|
|
|
((p)->prio < (rq)->curr->prio)
|
|
|
|
|
|
|
|
/*
|
|
|
|
* task_timeslice() scales user-nice values [ -20 ... 0 ... 19 ]
|
|
|
|
* to time slice values: [800ms ... 100ms ... 5ms]
|
|
|
|
*
|
|
|
|
* The higher a thread's priority, the bigger timeslices
|
|
|
|
* it gets during one round of execution. But even the lowest
|
|
|
|
* priority thread gets MIN_TIMESLICE worth of execution time.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#define SCALE_PRIO(x, prio) \
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
max(x * (MAX_PRIO - prio) / (MAX_USER_PRIO / 2), MIN_TIMESLICE)
|
2005-04-16 16:20:36 -06:00
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
static unsigned int static_prio_timeslice(int static_prio)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
if (static_prio < NICE_TO_PRIO(0))
|
|
|
|
return SCALE_PRIO(DEF_TIMESLICE * 4, static_prio);
|
2005-04-16 16:20:36 -06:00
|
|
|
else
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
return SCALE_PRIO(DEF_TIMESLICE, static_prio);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline unsigned int task_timeslice(struct task_struct *p)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
{
|
|
|
|
return static_prio_timeslice(p->static_prio);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* These are the runqueue data structures:
|
|
|
|
*/
|
|
|
|
|
|
|
|
struct prio_array {
|
|
|
|
unsigned int nr_active;
|
2006-06-27 03:54:29 -06:00
|
|
|
DECLARE_BITMAP(bitmap, MAX_PRIO+1); /* include 1 bit for delimiter */
|
2005-04-16 16:20:36 -06:00
|
|
|
struct list_head queue[MAX_PRIO];
|
|
|
|
};
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This is the main, per-CPU runqueue data structure.
|
|
|
|
*
|
|
|
|
* Locking rule: those places that want to lock multiple runqueues
|
|
|
|
* (such as the load balancing or the thread migration code), lock
|
|
|
|
* acquire operations must be ordered by ascending &runqueue.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq {
|
2005-04-16 16:20:36 -06:00
|
|
|
spinlock_t lock;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* nr_running and cpu_load should be in the same cacheline because
|
|
|
|
* remote CPUs use both these fields when doing load calculation.
|
|
|
|
*/
|
|
|
|
unsigned long nr_running;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
unsigned long raw_weighted_load;
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_SMP
|
2005-06-25 15:57:13 -06:00
|
|
|
unsigned long cpu_load[3];
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
|
|
|
unsigned long long nr_switches;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This is part of a global counter where only the total sum
|
|
|
|
* over all CPUs matters. A task can increase this counter on
|
|
|
|
* one CPU and if it got migrated afterwards it may decrease
|
|
|
|
* it on another CPU. Always updated under the runqueue lock:
|
|
|
|
*/
|
|
|
|
unsigned long nr_uninterruptible;
|
|
|
|
|
|
|
|
unsigned long expired_timestamp;
|
|
|
|
unsigned long long timestamp_last_tick;
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *curr, *idle;
|
2005-04-16 16:20:36 -06:00
|
|
|
struct mm_struct *prev_mm;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct prio_array *active, *expired, arrays[2];
|
2005-04-16 16:20:36 -06:00
|
|
|
int best_expired_prio;
|
|
|
|
atomic_t nr_iowait;
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
struct sched_domain *sd;
|
|
|
|
|
|
|
|
/* For active balancing */
|
|
|
|
int active_balance;
|
|
|
|
int push_cpu;
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *migration_thread;
|
2005-04-16 16:20:36 -06:00
|
|
|
struct list_head migration_queue;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#ifdef CONFIG_SCHEDSTATS
|
|
|
|
/* latency stats */
|
|
|
|
struct sched_info rq_sched_info;
|
|
|
|
|
|
|
|
/* sys_sched_yield() stats */
|
|
|
|
unsigned long yld_exp_empty;
|
|
|
|
unsigned long yld_act_empty;
|
|
|
|
unsigned long yld_both_empty;
|
|
|
|
unsigned long yld_cnt;
|
|
|
|
|
|
|
|
/* schedule() stats */
|
|
|
|
unsigned long sched_switch;
|
|
|
|
unsigned long sched_cnt;
|
|
|
|
unsigned long sched_goidle;
|
|
|
|
|
|
|
|
/* try_to_wake_up() stats */
|
|
|
|
unsigned long ttwu_cnt;
|
|
|
|
unsigned long ttwu_local;
|
|
|
|
#endif
|
2006-07-03 01:25:10 -06:00
|
|
|
struct lock_class_key rq_lock_key;
|
2005-04-16 16:20:36 -06:00
|
|
|
};
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static DEFINE_PER_CPU(struct rq, runqueues);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-06-25 15:57:27 -06:00
|
|
|
/*
|
|
|
|
* The domain tree (rq->sd) is protected by RCU's quiescent state transition.
|
2005-06-25 15:57:33 -06:00
|
|
|
* See detach_destroy_domains: synchronize_sched for details.
|
2005-06-25 15:57:27 -06:00
|
|
|
*
|
|
|
|
* The domain tree of any CPU may only be accessed from within
|
|
|
|
* preempt-disabled sections.
|
|
|
|
*/
|
2006-07-03 01:25:40 -06:00
|
|
|
#define for_each_domain(cpu, __sd) \
|
|
|
|
for (__sd = rcu_dereference(cpu_rq(cpu)->sd); __sd; __sd = __sd->parent)
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
#define cpu_rq(cpu) (&per_cpu(runqueues, (cpu)))
|
|
|
|
#define this_rq() (&__get_cpu_var(runqueues))
|
|
|
|
#define task_rq(p) cpu_rq(task_cpu(p))
|
|
|
|
#define cpu_curr(cpu) (cpu_rq(cpu)->curr)
|
|
|
|
|
|
|
|
#ifndef prepare_arch_switch
|
2005-06-25 15:57:23 -06:00
|
|
|
# define prepare_arch_switch(next) do { } while (0)
|
|
|
|
#endif
|
|
|
|
#ifndef finish_arch_switch
|
|
|
|
# define finish_arch_switch(prev) do { } while (0)
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#ifndef __ARCH_WANT_UNLOCKED_CTXSW
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline int task_running(struct rq *rq, struct task_struct *p)
|
2005-06-25 15:57:23 -06:00
|
|
|
{
|
|
|
|
return rq->curr == p;
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void prepare_lock_switch(struct rq *rq, struct task_struct *next)
|
2005-06-25 15:57:23 -06:00
|
|
|
{
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void finish_lock_switch(struct rq *rq, struct task_struct *prev)
|
2005-06-25 15:57:23 -06:00
|
|
|
{
|
2005-09-13 03:17:59 -06:00
|
|
|
#ifdef CONFIG_DEBUG_SPINLOCK
|
|
|
|
/* this is a valid case when another task releases the spinlock */
|
|
|
|
rq->lock.owner = current;
|
|
|
|
#endif
|
2006-07-03 01:24:54 -06:00
|
|
|
/*
|
|
|
|
* If we are tracking spinlock dependencies then we have to
|
|
|
|
* fix up the runqueue lock - which gets 'carried over' from
|
|
|
|
* prev into current:
|
|
|
|
*/
|
|
|
|
spin_acquire(&rq->lock.dep_map, 0, 0, _THIS_IP_);
|
|
|
|
|
2005-06-25 15:57:23 -06:00
|
|
|
spin_unlock_irq(&rq->lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
#else /* __ARCH_WANT_UNLOCKED_CTXSW */
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline int task_running(struct rq *rq, struct task_struct *p)
|
2005-06-25 15:57:23 -06:00
|
|
|
{
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
return p->oncpu;
|
|
|
|
#else
|
|
|
|
return rq->curr == p;
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void prepare_lock_switch(struct rq *rq, struct task_struct *next)
|
2005-06-25 15:57:23 -06:00
|
|
|
{
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
/*
|
|
|
|
* We can optimise this out completely for !SMP, because the
|
|
|
|
* SMP rebalancing from interrupt is the only thing that cares
|
|
|
|
* here.
|
|
|
|
*/
|
|
|
|
next->oncpu = 1;
|
|
|
|
#endif
|
|
|
|
#ifdef __ARCH_WANT_INTERRUPTS_ON_CTXSW
|
|
|
|
spin_unlock_irq(&rq->lock);
|
|
|
|
#else
|
|
|
|
spin_unlock(&rq->lock);
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void finish_lock_switch(struct rq *rq, struct task_struct *prev)
|
2005-06-25 15:57:23 -06:00
|
|
|
{
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
/*
|
|
|
|
* After ->oncpu is cleared, the task can be moved to a different CPU.
|
|
|
|
* We must ensure this doesn't happen until the switch is completely
|
|
|
|
* finished.
|
|
|
|
*/
|
|
|
|
smp_wmb();
|
|
|
|
prev->oncpu = 0;
|
|
|
|
#endif
|
|
|
|
#ifndef __ARCH_WANT_INTERRUPTS_ON_CTXSW
|
|
|
|
local_irq_enable();
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
2005-06-25 15:57:23 -06:00
|
|
|
}
|
|
|
|
#endif /* __ARCH_WANT_UNLOCKED_CTXSW */
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:51 -06:00
|
|
|
/*
|
|
|
|
* __task_rq_lock - lock the runqueue a given task resides on.
|
|
|
|
* Must be called interrupts disabled.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline struct rq *__task_rq_lock(struct task_struct *p)
|
2006-06-27 03:54:51 -06:00
|
|
|
__acquires(rq->lock)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2006-06-27 03:54:51 -06:00
|
|
|
|
|
|
|
repeat_lock_task:
|
|
|
|
rq = task_rq(p);
|
|
|
|
spin_lock(&rq->lock);
|
|
|
|
if (unlikely(rq != task_rq(p))) {
|
|
|
|
spin_unlock(&rq->lock);
|
|
|
|
goto repeat_lock_task;
|
|
|
|
}
|
|
|
|
return rq;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* task_rq_lock - lock the runqueue a given task resides on and disable
|
|
|
|
* interrupts. Note the ordering: we can safely lookup the task_rq without
|
|
|
|
* explicitly disabling preemption.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static struct rq *task_rq_lock(struct task_struct *p, unsigned long *flags)
|
2005-04-16 16:20:36 -06:00
|
|
|
__acquires(rq->lock)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
repeat_lock_task:
|
|
|
|
local_irq_save(*flags);
|
|
|
|
rq = task_rq(p);
|
|
|
|
spin_lock(&rq->lock);
|
|
|
|
if (unlikely(rq != task_rq(p))) {
|
|
|
|
spin_unlock_irqrestore(&rq->lock, *flags);
|
|
|
|
goto repeat_lock_task;
|
|
|
|
}
|
|
|
|
return rq;
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void __task_rq_unlock(struct rq *rq)
|
2006-06-27 03:54:51 -06:00
|
|
|
__releases(rq->lock)
|
|
|
|
{
|
|
|
|
spin_unlock(&rq->lock);
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void task_rq_unlock(struct rq *rq, unsigned long *flags)
|
2005-04-16 16:20:36 -06:00
|
|
|
__releases(rq->lock)
|
|
|
|
{
|
|
|
|
spin_unlock_irqrestore(&rq->lock, *flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_SCHEDSTATS
|
|
|
|
/*
|
|
|
|
* bump this up when changing the output format or the meaning of an existing
|
|
|
|
* format, so that tools can adapt (or abort)
|
|
|
|
*/
|
2005-06-25 15:57:20 -06:00
|
|
|
#define SCHEDSTAT_VERSION 12
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
static int show_schedstat(struct seq_file *seq, void *v)
|
|
|
|
{
|
|
|
|
int cpu;
|
|
|
|
|
|
|
|
seq_printf(seq, "version %d\n", SCHEDSTAT_VERSION);
|
|
|
|
seq_printf(seq, "timestamp %lu\n", jiffies);
|
|
|
|
for_each_online_cpu(cpu) {
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
struct sched_domain *sd;
|
|
|
|
int dcnt = 0;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/* runqueue-specific stats */
|
|
|
|
seq_printf(seq,
|
|
|
|
"cpu%d %lu %lu %lu %lu %lu %lu %lu %lu %lu %lu %lu %lu",
|
|
|
|
cpu, rq->yld_both_empty,
|
|
|
|
rq->yld_act_empty, rq->yld_exp_empty, rq->yld_cnt,
|
|
|
|
rq->sched_switch, rq->sched_cnt, rq->sched_goidle,
|
|
|
|
rq->ttwu_cnt, rq->ttwu_local,
|
|
|
|
rq->rq_sched_info.cpu_time,
|
|
|
|
rq->rq_sched_info.run_delay, rq->rq_sched_info.pcnt);
|
|
|
|
|
|
|
|
seq_printf(seq, "\n");
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
/* domain-specific stats */
|
2005-06-25 15:57:27 -06:00
|
|
|
preempt_disable();
|
2005-04-16 16:20:36 -06:00
|
|
|
for_each_domain(cpu, sd) {
|
|
|
|
enum idle_type itype;
|
|
|
|
char mask_str[NR_CPUS];
|
|
|
|
|
|
|
|
cpumask_scnprintf(mask_str, NR_CPUS, sd->span);
|
|
|
|
seq_printf(seq, "domain%d %s", dcnt++, mask_str);
|
|
|
|
for (itype = SCHED_IDLE; itype < MAX_IDLE_TYPES;
|
|
|
|
itype++) {
|
|
|
|
seq_printf(seq, " %lu %lu %lu %lu %lu %lu %lu %lu",
|
|
|
|
sd->lb_cnt[itype],
|
|
|
|
sd->lb_balanced[itype],
|
|
|
|
sd->lb_failed[itype],
|
|
|
|
sd->lb_imbalance[itype],
|
|
|
|
sd->lb_gained[itype],
|
|
|
|
sd->lb_hot_gained[itype],
|
|
|
|
sd->lb_nobusyq[itype],
|
|
|
|
sd->lb_nobusyg[itype]);
|
|
|
|
}
|
2005-06-25 15:57:20 -06:00
|
|
|
seq_printf(seq, " %lu %lu %lu %lu %lu %lu %lu %lu %lu %lu %lu %lu\n",
|
2005-04-16 16:20:36 -06:00
|
|
|
sd->alb_cnt, sd->alb_failed, sd->alb_pushed,
|
2005-06-25 15:57:20 -06:00
|
|
|
sd->sbe_cnt, sd->sbe_balanced, sd->sbe_pushed,
|
|
|
|
sd->sbf_cnt, sd->sbf_balanced, sd->sbf_pushed,
|
2005-04-16 16:20:36 -06:00
|
|
|
sd->ttwu_wake_remote, sd->ttwu_move_affine, sd->ttwu_move_balance);
|
|
|
|
}
|
2005-06-25 15:57:27 -06:00
|
|
|
preempt_enable();
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int schedstat_open(struct inode *inode, struct file *file)
|
|
|
|
{
|
|
|
|
unsigned int size = PAGE_SIZE * (1 + num_online_cpus() / 32);
|
|
|
|
char *buf = kmalloc(size, GFP_KERNEL);
|
|
|
|
struct seq_file *m;
|
|
|
|
int res;
|
|
|
|
|
|
|
|
if (!buf)
|
|
|
|
return -ENOMEM;
|
|
|
|
res = single_open(file, show_schedstat, NULL);
|
|
|
|
if (!res) {
|
|
|
|
m = file->private_data;
|
|
|
|
m->buf = buf;
|
|
|
|
m->size = size;
|
|
|
|
} else
|
|
|
|
kfree(buf);
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
|
|
|
struct file_operations proc_schedstat_operations = {
|
|
|
|
.open = schedstat_open,
|
|
|
|
.read = seq_read,
|
|
|
|
.llseek = seq_lseek,
|
|
|
|
.release = single_release,
|
|
|
|
};
|
|
|
|
|
2006-07-14 01:24:38 -06:00
|
|
|
/*
|
|
|
|
* Expects runqueue lock to be held for atomicity of update
|
|
|
|
*/
|
|
|
|
static inline void
|
|
|
|
rq_sched_info_arrive(struct rq *rq, unsigned long delta_jiffies)
|
|
|
|
{
|
|
|
|
if (rq) {
|
|
|
|
rq->rq_sched_info.run_delay += delta_jiffies;
|
|
|
|
rq->rq_sched_info.pcnt++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Expects runqueue lock to be held for atomicity of update
|
|
|
|
*/
|
|
|
|
static inline void
|
|
|
|
rq_sched_info_depart(struct rq *rq, unsigned long delta_jiffies)
|
|
|
|
{
|
|
|
|
if (rq)
|
|
|
|
rq->rq_sched_info.cpu_time += delta_jiffies;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
# define schedstat_inc(rq, field) do { (rq)->field++; } while (0)
|
|
|
|
# define schedstat_add(rq, field, amt) do { (rq)->field += (amt); } while (0)
|
|
|
|
#else /* !CONFIG_SCHEDSTATS */
|
2006-07-14 01:24:38 -06:00
|
|
|
static inline void
|
|
|
|
rq_sched_info_arrive(struct rq *rq, unsigned long delta_jiffies)
|
|
|
|
{}
|
|
|
|
static inline void
|
|
|
|
rq_sched_info_depart(struct rq *rq, unsigned long delta_jiffies)
|
|
|
|
{}
|
2005-04-16 16:20:36 -06:00
|
|
|
# define schedstat_inc(rq, field) do { } while (0)
|
|
|
|
# define schedstat_add(rq, field, amt) do { } while (0)
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/*
|
|
|
|
* rq_lock - lock a given runqueue and disable interrupts.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline struct rq *this_rq_lock(void)
|
2005-04-16 16:20:36 -06:00
|
|
|
__acquires(rq->lock)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
local_irq_disable();
|
|
|
|
rq = this_rq();
|
|
|
|
spin_lock(&rq->lock);
|
|
|
|
|
|
|
|
return rq;
|
|
|
|
}
|
|
|
|
|
2006-07-14 01:24:38 -06:00
|
|
|
#if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* Called when a process is dequeued from the active array and given
|
|
|
|
* the cpu. We should note that with the exception of interactive
|
|
|
|
* tasks, the expired queue will become the active queue after the active
|
|
|
|
* queue is empty, without explicitly dequeuing and requeuing tasks in the
|
|
|
|
* expired queue. (Interactive tasks may be requeued directly to the
|
|
|
|
* active queue, thus delaying tasks in the expired queue from running;
|
|
|
|
* see scheduler_tick()).
|
|
|
|
*
|
|
|
|
* This function is only called from sched_info_arrive(), rather than
|
|
|
|
* dequeue_task(). Even though a task may be queued and dequeued multiple
|
|
|
|
* times as it is shuffled about, we're really interested in knowing how
|
|
|
|
* long it was from the *first* time it was queued to the time that it
|
|
|
|
* finally hit a cpu.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline void sched_info_dequeued(struct task_struct *t)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
t->sched_info.last_queued = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called when a task finally hits the cpu. We can now calculate how
|
|
|
|
* long it was waiting to run. We also note when it began so that we
|
|
|
|
* can keep stats on how long its timeslice is.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static void sched_info_arrive(struct task_struct *t)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-14 01:24:38 -06:00
|
|
|
unsigned long now = jiffies, delta_jiffies = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (t->sched_info.last_queued)
|
2006-07-14 01:24:38 -06:00
|
|
|
delta_jiffies = now - t->sched_info.last_queued;
|
2005-04-16 16:20:36 -06:00
|
|
|
sched_info_dequeued(t);
|
2006-07-14 01:24:38 -06:00
|
|
|
t->sched_info.run_delay += delta_jiffies;
|
2005-04-16 16:20:36 -06:00
|
|
|
t->sched_info.last_arrival = now;
|
|
|
|
t->sched_info.pcnt++;
|
|
|
|
|
2006-07-14 01:24:38 -06:00
|
|
|
rq_sched_info_arrive(task_rq(t), delta_jiffies);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called when a process is queued into either the active or expired
|
|
|
|
* array. The time is noted and later used to determine how long we
|
|
|
|
* had to wait for us to reach the cpu. Since the expired queue will
|
|
|
|
* become the active queue after active queue is empty, without dequeuing
|
|
|
|
* and requeuing any tasks, we are interested in queuing to either. It
|
|
|
|
* is unusual but not impossible for tasks to be dequeued and immediately
|
|
|
|
* requeued in the same or another array: this can happen in sched_yield(),
|
|
|
|
* set_user_nice(), and even load_balance() as it moves tasks from runqueue
|
|
|
|
* to runqueue.
|
|
|
|
*
|
|
|
|
* This function is only called from enqueue_task(), but also only updates
|
|
|
|
* the timestamp if it is already not set. It's assumed that
|
|
|
|
* sched_info_dequeued() will clear that stamp when appropriate.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline void sched_info_queued(struct task_struct *t)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-14 01:24:38 -06:00
|
|
|
if (unlikely(sched_info_on()))
|
|
|
|
if (!t->sched_info.last_queued)
|
|
|
|
t->sched_info.last_queued = jiffies;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called when a process ceases being the active-running process, either
|
|
|
|
* voluntarily or involuntarily. Now we can calculate how long we ran.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline void sched_info_depart(struct task_struct *t)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-14 01:24:38 -06:00
|
|
|
unsigned long delta_jiffies = jiffies - t->sched_info.last_arrival;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-07-14 01:24:38 -06:00
|
|
|
t->sched_info.cpu_time += delta_jiffies;
|
|
|
|
rq_sched_info_depart(task_rq(t), delta_jiffies);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called when tasks are switched involuntarily due, typically, to expiring
|
|
|
|
* their time slice. (This may also be called when switching to or from
|
|
|
|
* the idle task.) We are only called when prev != next.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline void
|
2006-07-14 01:24:38 -06:00
|
|
|
__sched_info_switch(struct task_struct *prev, struct task_struct *next)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = task_rq(prev);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* prev now departs the cpu. It's not interesting to record
|
|
|
|
* stats about how efficient we were at scheduling the idle
|
|
|
|
* process, however.
|
|
|
|
*/
|
|
|
|
if (prev != rq->idle)
|
|
|
|
sched_info_depart(prev);
|
|
|
|
|
|
|
|
if (next != rq->idle)
|
|
|
|
sched_info_arrive(next);
|
|
|
|
}
|
2006-07-14 01:24:38 -06:00
|
|
|
static inline void
|
|
|
|
sched_info_switch(struct task_struct *prev, struct task_struct *next)
|
|
|
|
{
|
|
|
|
if (unlikely(sched_info_on()))
|
|
|
|
__sched_info_switch(prev, next);
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
#else
|
|
|
|
#define sched_info_queued(t) do { } while (0)
|
|
|
|
#define sched_info_switch(t, next) do { } while (0)
|
2006-07-14 01:24:38 -06:00
|
|
|
#endif /* CONFIG_SCHEDSTATS || CONFIG_TASK_DELAY_ACCT */
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Adding/removing a task to/from a priority array:
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void dequeue_task(struct task_struct *p, struct prio_array *array)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
array->nr_active--;
|
|
|
|
list_del(&p->run_list);
|
|
|
|
if (list_empty(array->queue + p->prio))
|
|
|
|
__clear_bit(p->prio, array->bitmap);
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static void enqueue_task(struct task_struct *p, struct prio_array *array)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
sched_info_queued(p);
|
|
|
|
list_add_tail(&p->run_list, array->queue + p->prio);
|
|
|
|
__set_bit(p->prio, array->bitmap);
|
|
|
|
array->nr_active++;
|
|
|
|
p->array = array;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Put task to the end of the run list without the overhead of dequeue
|
|
|
|
* followed by enqueue.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void requeue_task(struct task_struct *p, struct prio_array *array)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
list_move_tail(&p->run_list, array->queue + p->prio);
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void
|
|
|
|
enqueue_task_head(struct task_struct *p, struct prio_array *array)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
list_add(&p->run_list, array->queue + p->prio);
|
|
|
|
__set_bit(p->prio, array->bitmap);
|
|
|
|
array->nr_active++;
|
|
|
|
p->array = array;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2006-06-27 03:54:51 -06:00
|
|
|
* __normal_prio - return the priority that is based on the static
|
2005-04-16 16:20:36 -06:00
|
|
|
* priority but is modified by bonuses/penalties.
|
|
|
|
*
|
|
|
|
* We scale the actual sleep average [0 .... MAX_SLEEP_AVG]
|
|
|
|
* into the -5 ... 0 ... +5 bonus/penalty range.
|
|
|
|
*
|
|
|
|
* We use 25% of the full 0...39 priority range so that:
|
|
|
|
*
|
|
|
|
* 1) nice +19 interactive tasks do not preempt nice 0 CPU hogs.
|
|
|
|
* 2) nice -20 CPU hogs do not get preempted by nice 0 tasks.
|
|
|
|
*
|
|
|
|
* Both properties are important to certain workloads.
|
|
|
|
*/
|
2006-06-27 03:54:51 -06:00
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline int __normal_prio(struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
int bonus, prio;
|
|
|
|
|
|
|
|
bonus = CURRENT_BONUS(p) - MAX_BONUS / 2;
|
|
|
|
|
|
|
|
prio = p->static_prio - bonus;
|
|
|
|
if (prio < MAX_RT_PRIO)
|
|
|
|
prio = MAX_RT_PRIO;
|
|
|
|
if (prio > MAX_PRIO-1)
|
|
|
|
prio = MAX_PRIO-1;
|
|
|
|
return prio;
|
|
|
|
}
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
/*
|
|
|
|
* To aid in avoiding the subversion of "niceness" due to uneven distribution
|
|
|
|
* of tasks with abnormal "nice" values across CPUs the contribution that
|
|
|
|
* each task makes to its run queue's load is weighted according to its
|
|
|
|
* scheduling class and "nice" value. For SCHED_NORMAL tasks this is just a
|
|
|
|
* scaled version of the new time slice allocation that they receive on time
|
|
|
|
* slice expiry etc.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Assume: static_prio_timeslice(NICE_TO_PRIO(0)) == DEF_TIMESLICE
|
|
|
|
* If static_prio_timeslice() is ever changed to break this assumption then
|
|
|
|
* this code will need modification
|
|
|
|
*/
|
|
|
|
#define TIME_SLICE_NICE_ZERO DEF_TIMESLICE
|
|
|
|
#define LOAD_WEIGHT(lp) \
|
|
|
|
(((lp) * SCHED_LOAD_SCALE) / TIME_SLICE_NICE_ZERO)
|
|
|
|
#define PRIO_TO_LOAD_WEIGHT(prio) \
|
|
|
|
LOAD_WEIGHT(static_prio_timeslice(prio))
|
|
|
|
#define RTPRIO_TO_LOAD_WEIGHT(rp) \
|
|
|
|
(PRIO_TO_LOAD_WEIGHT(MAX_RT_PRIO) + LOAD_WEIGHT(rp))
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
static void set_load_weight(struct task_struct *p)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
{
|
2006-06-27 03:54:51 -06:00
|
|
|
if (has_rt_policy(p)) {
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
if (p == task_rq(p)->migration_thread)
|
|
|
|
/*
|
|
|
|
* The migration thread does the actual balancing.
|
|
|
|
* Giving its load any weight will skew balancing
|
|
|
|
* adversely.
|
|
|
|
*/
|
|
|
|
p->load_weight = 0;
|
|
|
|
else
|
|
|
|
#endif
|
|
|
|
p->load_weight = RTPRIO_TO_LOAD_WEIGHT(p->rt_priority);
|
|
|
|
} else
|
|
|
|
p->load_weight = PRIO_TO_LOAD_WEIGHT(p->static_prio);
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline void
|
2006-07-03 01:25:42 -06:00
|
|
|
inc_raw_weighted_load(struct rq *rq, const struct task_struct *p)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
{
|
|
|
|
rq->raw_weighted_load += p->load_weight;
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline void
|
2006-07-03 01:25:42 -06:00
|
|
|
dec_raw_weighted_load(struct rq *rq, const struct task_struct *p)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
{
|
|
|
|
rq->raw_weighted_load -= p->load_weight;
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void inc_nr_running(struct task_struct *p, struct rq *rq)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
{
|
|
|
|
rq->nr_running++;
|
|
|
|
inc_raw_weighted_load(rq, p);
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void dec_nr_running(struct task_struct *p, struct rq *rq)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
{
|
|
|
|
rq->nr_running--;
|
|
|
|
dec_raw_weighted_load(rq, p);
|
|
|
|
}
|
|
|
|
|
2006-06-27 03:54:51 -06:00
|
|
|
/*
|
|
|
|
* Calculate the expected normal priority: i.e. priority
|
|
|
|
* without taking RT-inheritance into account. Might be
|
|
|
|
* boosted by interactivity modifiers. Changes upon fork,
|
|
|
|
* setprio syscalls, and whenever the interactivity
|
|
|
|
* estimator recalculates.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline int normal_prio(struct task_struct *p)
|
2006-06-27 03:54:51 -06:00
|
|
|
{
|
|
|
|
int prio;
|
|
|
|
|
|
|
|
if (has_rt_policy(p))
|
|
|
|
prio = MAX_RT_PRIO-1 - p->rt_priority;
|
|
|
|
else
|
|
|
|
prio = __normal_prio(p);
|
|
|
|
return prio;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Calculate the current priority, i.e. the priority
|
|
|
|
* taken into account by the scheduler. This value might
|
|
|
|
* be boosted by RT tasks, or might be boosted by
|
|
|
|
* interactivity modifiers. Will be RT if the task got
|
|
|
|
* RT-boosted. If not then it returns p->normal_prio.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static int effective_prio(struct task_struct *p)
|
2006-06-27 03:54:51 -06:00
|
|
|
{
|
|
|
|
p->normal_prio = normal_prio(p);
|
|
|
|
/*
|
|
|
|
* If we are RT tasks or we were boosted to RT priority,
|
|
|
|
* keep the priority unchanged. Otherwise, update priority
|
|
|
|
* to the normal priority:
|
|
|
|
*/
|
|
|
|
if (!rt_prio(p->prio))
|
|
|
|
return p->normal_prio;
|
|
|
|
return p->prio;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* __activate_task - move a task to the runqueue.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void __activate_task(struct task_struct *p, struct rq *rq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct prio_array *target = rq->active;
|
2006-03-31 03:31:29 -07:00
|
|
|
|
2006-05-21 19:54:09 -06:00
|
|
|
if (batch_task(p))
|
2006-03-31 03:31:29 -07:00
|
|
|
target = rq->expired;
|
|
|
|
enqueue_task(p, target);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
inc_nr_running(p, rq);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* __activate_idle_task - move idle task to the _front_ of runqueue.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void __activate_idle_task(struct task_struct *p, struct rq *rq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
enqueue_task_head(p, rq->active);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
inc_nr_running(p, rq);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2006-06-27 03:54:51 -06:00
|
|
|
/*
|
|
|
|
* Recalculate p->normal_prio and p->prio after having slept,
|
|
|
|
* updating the sleep-average too:
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static int recalc_task_prio(struct task_struct *p, unsigned long long now)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
/* Caller must always ensure 'now >= p->timestamp' */
|
2006-06-27 03:54:30 -06:00
|
|
|
unsigned long sleep_time = now - p->timestamp;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-03-31 03:31:29 -07:00
|
|
|
if (batch_task(p))
|
2006-01-14 14:20:41 -07:00
|
|
|
sleep_time = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (likely(sleep_time > 0)) {
|
|
|
|
/*
|
2006-06-27 03:54:30 -06:00
|
|
|
* This ceiling is set to the lowest priority that would allow
|
|
|
|
* a task to be reinserted into the active array on timeslice
|
|
|
|
* completion.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2006-06-27 03:54:30 -06:00
|
|
|
unsigned long ceiling = INTERACTIVE_SLEEP(p);
|
2006-03-31 03:31:26 -07:00
|
|
|
|
2006-06-27 03:54:30 -06:00
|
|
|
if (p->mm && sleep_time > ceiling && p->sleep_avg < ceiling) {
|
|
|
|
/*
|
|
|
|
* Prevents user tasks from achieving best priority
|
|
|
|
* with one single large enough sleep.
|
|
|
|
*/
|
|
|
|
p->sleep_avg = ceiling;
|
|
|
|
/*
|
|
|
|
* Using INTERACTIVE_SLEEP() as a ceiling places a
|
|
|
|
* nice(0) task 1ms sleep away from promotion, and
|
|
|
|
* gives it 700ms to round-robin with no chance of
|
|
|
|
* being demoted. This is more than generous, so
|
|
|
|
* mark this sleep as non-interactive to prevent the
|
|
|
|
* on-runqueue bonus logic from intervening should
|
|
|
|
* this task not receive cpu immediately.
|
|
|
|
*/
|
|
|
|
p->sleep_type = SLEEP_NONINTERACTIVE;
|
2005-04-16 16:20:36 -06:00
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Tasks waking from uninterruptible sleep are
|
|
|
|
* limited in their sleep_avg rise as they
|
|
|
|
* are likely to be waiting on I/O
|
|
|
|
*/
|
2006-03-31 03:31:23 -07:00
|
|
|
if (p->sleep_type == SLEEP_NONINTERACTIVE && p->mm) {
|
2006-06-27 03:54:30 -06:00
|
|
|
if (p->sleep_avg >= ceiling)
|
2005-04-16 16:20:36 -06:00
|
|
|
sleep_time = 0;
|
|
|
|
else if (p->sleep_avg + sleep_time >=
|
2006-06-27 03:54:30 -06:00
|
|
|
ceiling) {
|
|
|
|
p->sleep_avg = ceiling;
|
|
|
|
sleep_time = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This code gives a bonus to interactive tasks.
|
|
|
|
*
|
|
|
|
* The boost works by updating the 'average sleep time'
|
|
|
|
* value here, based on ->timestamp. The more time a
|
|
|
|
* task spends sleeping, the higher the average gets -
|
|
|
|
* and the higher the priority boost gets as well.
|
|
|
|
*/
|
|
|
|
p->sleep_avg += sleep_time;
|
|
|
|
|
|
|
|
}
|
2006-06-27 03:54:30 -06:00
|
|
|
if (p->sleep_avg > NS_MAX_SLEEP_AVG)
|
|
|
|
p->sleep_avg = NS_MAX_SLEEP_AVG;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2005-06-25 15:57:31 -06:00
|
|
|
return effective_prio(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* activate_task - move a task to the runqueue and do priority recalculation
|
|
|
|
*
|
|
|
|
* Update all the scheduling statistics stuff. (sleep average
|
|
|
|
* calculation, priority modifiers, etc.)
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void activate_task(struct task_struct *p, struct rq *rq, int local)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
unsigned long long now;
|
|
|
|
|
|
|
|
now = sched_clock();
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
if (!local) {
|
|
|
|
/* Compensate for drifting sched_clock */
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *this_rq = this_rq();
|
2005-04-16 16:20:36 -06:00
|
|
|
now = (now - this_rq->timestamp_last_tick)
|
|
|
|
+ rq->timestamp_last_tick;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2005-11-09 16:45:29 -07:00
|
|
|
if (!rt_task(p))
|
|
|
|
p->prio = recalc_task_prio(p, now);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* This checks to make sure it's not an uninterruptible task
|
|
|
|
* that is now waking up.
|
|
|
|
*/
|
2006-03-31 03:31:23 -07:00
|
|
|
if (p->sleep_type == SLEEP_NORMAL) {
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* Tasks which were woken up by interrupts (ie. hw events)
|
|
|
|
* are most likely of interactive nature. So we give them
|
|
|
|
* the credit of extending their sleep time to the period
|
|
|
|
* of time they spend on the runqueue, waiting for execution
|
|
|
|
* on a CPU, first time around:
|
|
|
|
*/
|
|
|
|
if (in_interrupt())
|
2006-03-31 03:31:23 -07:00
|
|
|
p->sleep_type = SLEEP_INTERRUPTED;
|
2005-04-16 16:20:36 -06:00
|
|
|
else {
|
|
|
|
/*
|
|
|
|
* Normal first-time wakeups get a credit too for
|
|
|
|
* on-runqueue time, but it will be weighted down:
|
|
|
|
*/
|
2006-03-31 03:31:23 -07:00
|
|
|
p->sleep_type = SLEEP_INTERACTIVE;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
p->timestamp = now;
|
|
|
|
|
|
|
|
__activate_task(p, rq);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* deactivate_task - remove a task from the runqueue.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void deactivate_task(struct task_struct *p, struct rq *rq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
dec_nr_running(p, rq);
|
2005-04-16 16:20:36 -06:00
|
|
|
dequeue_task(p, p->array);
|
|
|
|
p->array = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* resched_task - mark a task 'to be rescheduled now'.
|
|
|
|
*
|
|
|
|
* On UP this means the setting of the need_resched flag, on SMP it
|
|
|
|
* might also involve a cross-CPU call to trigger the scheduler on
|
|
|
|
* the target CPU.
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_SMP
|
2006-06-26 05:59:11 -06:00
|
|
|
|
|
|
|
#ifndef tsk_is_polling
|
|
|
|
#define tsk_is_polling(t) test_tsk_thread_flag(t, TIF_POLLING_NRFLAG)
|
|
|
|
#endif
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
static void resched_task(struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-08 22:39:04 -07:00
|
|
|
int cpu;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
assert_spin_locked(&task_rq(p)->lock);
|
|
|
|
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-08 22:39:04 -07:00
|
|
|
if (unlikely(test_tsk_thread_flag(p, TIF_NEED_RESCHED)))
|
|
|
|
return;
|
|
|
|
|
|
|
|
set_tsk_thread_flag(p, TIF_NEED_RESCHED);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-08 22:39:04 -07:00
|
|
|
cpu = task_cpu(p);
|
|
|
|
if (cpu == smp_processor_id())
|
|
|
|
return;
|
|
|
|
|
2006-06-26 05:59:11 -06:00
|
|
|
/* NEED_RESCHED must be visible before we test polling */
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-08 22:39:04 -07:00
|
|
|
smp_mb();
|
2006-06-26 05:59:11 -06:00
|
|
|
if (!tsk_is_polling(p))
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-08 22:39:04 -07:00
|
|
|
smp_send_reschedule(cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
#else
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline void resched_task(struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
[PATCH] sched: resched and cpu_idle rework
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Improves efficiency of
resched_task and some cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of
POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Signed-off-by: Nick Piggin <npiggin@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Con Kolivas <kernel@kolivas.org>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-11-08 22:39:04 -07:00
|
|
|
assert_spin_locked(&task_rq(p)->lock);
|
2005-04-16 16:20:36 -06:00
|
|
|
set_tsk_need_resched(p);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/**
|
|
|
|
* task_curr - is this task currently executing on a CPU?
|
|
|
|
* @p: the task in question.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
inline int task_curr(const struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return cpu_curr(task_cpu(p)) == p;
|
|
|
|
}
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
/* Used instead of source_load when we know the type == 0 */
|
|
|
|
unsigned long weighted_cpuload(const int cpu)
|
|
|
|
{
|
|
|
|
return cpu_rq(cpu)->raw_weighted_load;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_SMP
|
2006-07-03 01:25:42 -06:00
|
|
|
struct migration_req {
|
2005-04-16 16:20:36 -06:00
|
|
|
struct list_head list;
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *task;
|
2005-04-16 16:20:36 -06:00
|
|
|
int dest_cpu;
|
|
|
|
|
|
|
|
struct completion done;
|
2006-07-03 01:25:42 -06:00
|
|
|
};
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The task's runqueue lock must be held.
|
|
|
|
* Returns true if you have to wait for migration thread.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static int
|
2006-07-03 01:25:42 -06:00
|
|
|
migrate_task(struct task_struct *p, int dest_cpu, struct migration_req *req)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = task_rq(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the task is not on a runqueue (and not running), then
|
|
|
|
* it is sufficient to simply update the task's cpu field.
|
|
|
|
*/
|
|
|
|
if (!p->array && !task_running(rq, p)) {
|
|
|
|
set_task_cpu(p, dest_cpu);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
init_completion(&req->done);
|
|
|
|
req->task = p;
|
|
|
|
req->dest_cpu = dest_cpu;
|
|
|
|
list_add(&req->list, &rq->migration_queue);
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* wait_task_inactive - wait for a thread to unschedule.
|
|
|
|
*
|
|
|
|
* The caller must ensure that the task *will* unschedule sometime soon,
|
|
|
|
* else this function might spin for a *long* time. This function can't
|
|
|
|
* be called with interrupts off, or it may introduce deadlock with
|
|
|
|
* smp_call_function() if an IPI is sent by the same process we are
|
|
|
|
* waiting to become inactive.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
void wait_task_inactive(struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
unsigned long flags;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
int preempted;
|
|
|
|
|
|
|
|
repeat:
|
|
|
|
rq = task_rq_lock(p, &flags);
|
|
|
|
/* Must be off runqueue entirely, not preempted. */
|
|
|
|
if (unlikely(p->array || task_running(rq, p))) {
|
|
|
|
/* If it's preempted, we yield. It could be a while. */
|
|
|
|
preempted = !task_running(rq, p);
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
cpu_relax();
|
|
|
|
if (preempted)
|
|
|
|
yield();
|
|
|
|
goto repeat;
|
|
|
|
}
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
/***
|
|
|
|
* kick_process - kick a running thread to enter/exit the kernel
|
|
|
|
* @p: the to-be-kicked thread
|
|
|
|
*
|
|
|
|
* Cause a process which is running on another CPU to enter
|
|
|
|
* kernel-mode, without any delay. (to get signals handled.)
|
|
|
|
*
|
|
|
|
* NOTE: this function doesnt have to take the runqueue lock,
|
|
|
|
* because all it wants to ensure is that the remote task enters
|
|
|
|
* the kernel. If the IPI races and the task has been migrated
|
|
|
|
* to another CPU then no harm is done and the purpose has been
|
|
|
|
* achieved as well.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
void kick_process(struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
int cpu;
|
|
|
|
|
|
|
|
preempt_disable();
|
|
|
|
cpu = task_cpu(p);
|
|
|
|
if ((cpu != smp_processor_id()) && task_curr(p))
|
|
|
|
smp_send_reschedule(cpu);
|
|
|
|
preempt_enable();
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
* Return a low guess at the load of a migration-source cpu weighted
|
|
|
|
* according to the scheduling class and "nice" value.
|
2005-04-16 16:20:36 -06:00
|
|
|
*
|
|
|
|
* We want to under-estimate the load of migration sources, to
|
|
|
|
* balance conservatively.
|
|
|
|
*/
|
[PATCH] sched: remove smpnice
I don't think the code is quite ready, which is why I asked for Peter's
additions to also be merged before I acked it (although it turned out that
it still isn't quite ready with his additions either).
Basically I have had similar observations to Suresh in that it does not
play nicely with the rest of the balancing infrastructure (and raised
similar concerns in my review).
The samples (group of 4) I got for "maximum recorded imbalance" on a 2x2
SMP+HT Xeon are as follows:
| Following boot | hackbench 20 | hackbench 40
-----------+----------------+---------------------+---------------------
2.6.16-rc2 | 30,37,100,112 | 5600,5530,6020,6090 | 6390,7090,8760,8470
+nosmpnice | 3, 2, 4, 2 | 28, 150, 294, 132 | 348, 348, 294, 347
Hackbench raw performance is down around 15% with smpnice (but that in
itself isn't a huge deal because it is just a benchmark). However, the
samples show that the imbalance passed into move_tasks is increased by
about a factor of 10-30. I think this would also go some way to explaining
latency blips turning up in the balancing code (though I haven't actually
measured that).
We'll probably have to revert this in the SUSE kernel.
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Peter Williams <pwil3058@bigpond.net.au>
Cc: "Martin J. Bligh" <mbligh@aracnet.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-02-10 02:51:02 -07:00
|
|
|
static inline unsigned long source_load(int cpu, int type)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(cpu);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
|
2005-11-08 22:38:58 -07:00
|
|
|
if (type == 0)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
return rq->raw_weighted_load;
|
2005-11-08 22:38:55 -07:00
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
return min(rq->cpu_load[type-1], rq->raw_weighted_load);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
* Return a high guess at the load of a migration-target cpu weighted
|
|
|
|
* according to the scheduling class and "nice" value.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
[PATCH] sched: remove smpnice
I don't think the code is quite ready, which is why I asked for Peter's
additions to also be merged before I acked it (although it turned out that
it still isn't quite ready with his additions either).
Basically I have had similar observations to Suresh in that it does not
play nicely with the rest of the balancing infrastructure (and raised
similar concerns in my review).
The samples (group of 4) I got for "maximum recorded imbalance" on a 2x2
SMP+HT Xeon are as follows:
| Following boot | hackbench 20 | hackbench 40
-----------+----------------+---------------------+---------------------
2.6.16-rc2 | 30,37,100,112 | 5600,5530,6020,6090 | 6390,7090,8760,8470
+nosmpnice | 3, 2, 4, 2 | 28, 150, 294, 132 | 348, 348, 294, 347
Hackbench raw performance is down around 15% with smpnice (but that in
itself isn't a huge deal because it is just a benchmark). However, the
samples show that the imbalance passed into move_tasks is increased by
about a factor of 10-30. I think this would also go some way to explaining
latency blips turning up in the balancing code (though I haven't actually
measured that).
We'll probably have to revert this in the SUSE kernel.
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Peter Williams <pwil3058@bigpond.net.au>
Cc: "Martin J. Bligh" <mbligh@aracnet.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-02-10 02:51:02 -07:00
|
|
|
static inline unsigned long target_load(int cpu, int type)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(cpu);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
|
2005-06-25 15:57:13 -06:00
|
|
|
if (type == 0)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
return rq->raw_weighted_load;
|
2005-11-08 22:38:58 -07:00
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
return max(rq->cpu_load[type-1], rq->raw_weighted_load);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Return the average load per task on the cpu's run queue
|
|
|
|
*/
|
|
|
|
static inline unsigned long cpu_avg_load_per_task(int cpu)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(cpu);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
unsigned long n = rq->nr_running;
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
return n ? rq->raw_weighted_load / n : SCHED_LOAD_SCALE;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2005-06-25 15:57:19 -06:00
|
|
|
/*
|
|
|
|
* find_idlest_group finds and returns the least busy CPU group within the
|
|
|
|
* domain.
|
|
|
|
*/
|
|
|
|
static struct sched_group *
|
|
|
|
find_idlest_group(struct sched_domain *sd, struct task_struct *p, int this_cpu)
|
|
|
|
{
|
|
|
|
struct sched_group *idlest = NULL, *this = NULL, *group = sd->groups;
|
|
|
|
unsigned long min_load = ULONG_MAX, this_load = 0;
|
|
|
|
int load_idx = sd->forkexec_idx;
|
|
|
|
int imbalance = 100 + (sd->imbalance_pct-100)/2;
|
|
|
|
|
|
|
|
do {
|
|
|
|
unsigned long load, avg_load;
|
|
|
|
int local_group;
|
|
|
|
int i;
|
|
|
|
|
2005-09-10 01:26:09 -06:00
|
|
|
/* Skip over this group if it has no CPUs allowed */
|
|
|
|
if (!cpus_intersects(group->cpumask, p->cpus_allowed))
|
|
|
|
goto nextgroup;
|
|
|
|
|
2005-06-25 15:57:19 -06:00
|
|
|
local_group = cpu_isset(this_cpu, group->cpumask);
|
|
|
|
|
|
|
|
/* Tally up the load of all CPUs in the group */
|
|
|
|
avg_load = 0;
|
|
|
|
|
|
|
|
for_each_cpu_mask(i, group->cpumask) {
|
|
|
|
/* Bias balancing toward cpus of our domain */
|
|
|
|
if (local_group)
|
|
|
|
load = source_load(i, load_idx);
|
|
|
|
else
|
|
|
|
load = target_load(i, load_idx);
|
|
|
|
|
|
|
|
avg_load += load;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Adjust by relative CPU power of the group */
|
|
|
|
avg_load = (avg_load * SCHED_LOAD_SCALE) / group->cpu_power;
|
|
|
|
|
|
|
|
if (local_group) {
|
|
|
|
this_load = avg_load;
|
|
|
|
this = group;
|
|
|
|
} else if (avg_load < min_load) {
|
|
|
|
min_load = avg_load;
|
|
|
|
idlest = group;
|
|
|
|
}
|
2005-09-10 01:26:09 -06:00
|
|
|
nextgroup:
|
2005-06-25 15:57:19 -06:00
|
|
|
group = group->next;
|
|
|
|
} while (group != sd->groups);
|
|
|
|
|
|
|
|
if (!idlest || 100*this_load < imbalance*min_load)
|
|
|
|
return NULL;
|
|
|
|
return idlest;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* find_idlest_queue - find the idlest runqueue among the cpus in group.
|
|
|
|
*/
|
2005-09-10 01:26:11 -06:00
|
|
|
static int
|
|
|
|
find_idlest_cpu(struct sched_group *group, struct task_struct *p, int this_cpu)
|
2005-06-25 15:57:19 -06:00
|
|
|
{
|
2005-09-10 01:26:09 -06:00
|
|
|
cpumask_t tmp;
|
2005-06-25 15:57:19 -06:00
|
|
|
unsigned long load, min_load = ULONG_MAX;
|
|
|
|
int idlest = -1;
|
|
|
|
int i;
|
|
|
|
|
2005-09-10 01:26:09 -06:00
|
|
|
/* Traverse only the allowed CPUs */
|
|
|
|
cpus_and(tmp, group->cpumask, p->cpus_allowed);
|
|
|
|
|
|
|
|
for_each_cpu_mask(i, tmp) {
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
load = weighted_cpuload(i);
|
2005-06-25 15:57:19 -06:00
|
|
|
|
|
|
|
if (load < min_load || (load == min_load && i == this_cpu)) {
|
|
|
|
min_load = load;
|
|
|
|
idlest = i;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return idlest;
|
|
|
|
}
|
|
|
|
|
2005-06-25 15:57:29 -06:00
|
|
|
/*
|
|
|
|
* sched_balance_self: balance the current task (running on cpu) in domains
|
|
|
|
* that have the 'flag' flag set. In practice, this is SD_BALANCE_FORK and
|
|
|
|
* SD_BALANCE_EXEC.
|
|
|
|
*
|
|
|
|
* Balance, ie. select the least loaded group.
|
|
|
|
*
|
|
|
|
* Returns the target CPU number, or the same CPU if no balancing is needed.
|
|
|
|
*
|
|
|
|
* preempt must be disabled.
|
|
|
|
*/
|
|
|
|
static int sched_balance_self(int cpu, int flag)
|
|
|
|
{
|
|
|
|
struct task_struct *t = current;
|
|
|
|
struct sched_domain *tmp, *sd = NULL;
|
2005-06-25 15:57:19 -06:00
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
for_each_domain(cpu, tmp) {
|
2006-06-27 03:54:42 -06:00
|
|
|
/*
|
|
|
|
* If power savings logic is enabled for a domain, stop there.
|
|
|
|
*/
|
|
|
|
if (tmp->flags & SD_POWERSAVINGS_BALANCE)
|
|
|
|
break;
|
2005-06-25 15:57:29 -06:00
|
|
|
if (tmp->flags & flag)
|
|
|
|
sd = tmp;
|
2006-06-27 03:54:28 -06:00
|
|
|
}
|
2005-06-25 15:57:29 -06:00
|
|
|
|
|
|
|
while (sd) {
|
|
|
|
cpumask_t span;
|
|
|
|
struct sched_group *group;
|
|
|
|
int new_cpu;
|
|
|
|
int weight;
|
|
|
|
|
|
|
|
span = sd->span;
|
|
|
|
group = find_idlest_group(sd, t, cpu);
|
|
|
|
if (!group)
|
|
|
|
goto nextlevel;
|
|
|
|
|
2005-09-10 01:26:09 -06:00
|
|
|
new_cpu = find_idlest_cpu(group, t, cpu);
|
2005-06-25 15:57:29 -06:00
|
|
|
if (new_cpu == -1 || new_cpu == cpu)
|
|
|
|
goto nextlevel;
|
|
|
|
|
|
|
|
/* Now try balancing at a lower domain level */
|
|
|
|
cpu = new_cpu;
|
|
|
|
nextlevel:
|
|
|
|
sd = NULL;
|
|
|
|
weight = cpus_weight(span);
|
|
|
|
for_each_domain(cpu, tmp) {
|
|
|
|
if (weight <= cpus_weight(tmp->span))
|
|
|
|
break;
|
|
|
|
if (tmp->flags & flag)
|
|
|
|
sd = tmp;
|
|
|
|
}
|
|
|
|
/* while loop will break here if sd == NULL */
|
|
|
|
}
|
|
|
|
|
|
|
|
return cpu;
|
|
|
|
}
|
|
|
|
|
|
|
|
#endif /* CONFIG_SMP */
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* wake_idle() will wake a task on an idle cpu if task->cpu is
|
|
|
|
* not idle and an idle cpu is available. The span of cpus to
|
|
|
|
* search starts with cpus closest then further out as needed,
|
|
|
|
* so we always favor a closer, idle cpu.
|
|
|
|
*
|
|
|
|
* Returns the CPU we should wake onto.
|
|
|
|
*/
|
|
|
|
#if defined(ARCH_HAS_SCHED_WAKE_IDLE)
|
2006-07-03 01:25:41 -06:00
|
|
|
static int wake_idle(int cpu, struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
cpumask_t tmp;
|
|
|
|
struct sched_domain *sd;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
if (idle_cpu(cpu))
|
|
|
|
return cpu;
|
|
|
|
|
|
|
|
for_each_domain(cpu, sd) {
|
|
|
|
if (sd->flags & SD_WAKE_IDLE) {
|
2005-06-25 15:57:06 -06:00
|
|
|
cpus_and(tmp, sd->span, p->cpus_allowed);
|
2005-04-16 16:20:36 -06:00
|
|
|
for_each_cpu_mask(i, tmp) {
|
|
|
|
if (idle_cpu(i))
|
|
|
|
return i;
|
|
|
|
}
|
|
|
|
}
|
2005-06-25 15:57:06 -06:00
|
|
|
else
|
|
|
|
break;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
return cpu;
|
|
|
|
}
|
|
|
|
#else
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline int wake_idle(int cpu, struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return cpu;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/***
|
|
|
|
* try_to_wake_up - wake up a thread
|
|
|
|
* @p: the to-be-woken-up thread
|
|
|
|
* @state: the mask of task states that can be woken
|
|
|
|
* @sync: do a synchronous wakeup?
|
|
|
|
*
|
|
|
|
* Put it on the run-queue if it's not already there. The "current"
|
|
|
|
* thread is always on the run-queue (except when the actual
|
|
|
|
* re-schedule is in progress), and as such you're allowed to do
|
|
|
|
* the simpler "current->state = TASK_RUNNING" to mark yourself
|
|
|
|
* runnable without the overhead of this.
|
|
|
|
*
|
|
|
|
* returns failure only if the task is already active.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static int try_to_wake_up(struct task_struct *p, unsigned int state, int sync)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
int cpu, this_cpu, success = 0;
|
|
|
|
unsigned long flags;
|
|
|
|
long old_state;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_SMP
|
2005-06-25 15:57:13 -06:00
|
|
|
struct sched_domain *sd, *this_sd = NULL;
|
2006-07-03 01:25:42 -06:00
|
|
|
unsigned long load, this_load;
|
2005-04-16 16:20:36 -06:00
|
|
|
int new_cpu;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
rq = task_rq_lock(p, &flags);
|
|
|
|
old_state = p->state;
|
|
|
|
if (!(old_state & state))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
if (p->array)
|
|
|
|
goto out_running;
|
|
|
|
|
|
|
|
cpu = task_cpu(p);
|
|
|
|
this_cpu = smp_processor_id();
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
if (unlikely(task_running(rq, p)))
|
|
|
|
goto out_activate;
|
|
|
|
|
2005-06-25 15:57:13 -06:00
|
|
|
new_cpu = cpu;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
schedstat_inc(rq, ttwu_cnt);
|
|
|
|
if (cpu == this_cpu) {
|
|
|
|
schedstat_inc(rq, ttwu_local);
|
2005-06-25 15:57:13 -06:00
|
|
|
goto out_set_cpu;
|
|
|
|
}
|
|
|
|
|
|
|
|
for_each_domain(this_cpu, sd) {
|
|
|
|
if (cpu_isset(cpu, sd->span)) {
|
|
|
|
schedstat_inc(sd, ttwu_wake_remote);
|
|
|
|
this_sd = sd;
|
|
|
|
break;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-06-25 15:57:13 -06:00
|
|
|
if (unlikely(!cpu_isset(this_cpu, p->cpus_allowed)))
|
2005-04-16 16:20:36 -06:00
|
|
|
goto out_set_cpu;
|
|
|
|
|
|
|
|
/*
|
2005-06-25 15:57:13 -06:00
|
|
|
* Check for affine wakeup and passive balancing possibilities.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2005-06-25 15:57:13 -06:00
|
|
|
if (this_sd) {
|
|
|
|
int idx = this_sd->wake_idx;
|
|
|
|
unsigned int imbalance;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-06-25 15:57:15 -06:00
|
|
|
imbalance = 100 + (this_sd->imbalance_pct - 100) / 2;
|
|
|
|
|
2005-06-25 15:57:13 -06:00
|
|
|
load = source_load(cpu, idx);
|
|
|
|
this_load = target_load(this_cpu, idx);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-06-25 15:57:13 -06:00
|
|
|
new_cpu = this_cpu; /* Wake to this CPU if we can */
|
|
|
|
|
2005-06-25 15:57:15 -06:00
|
|
|
if (this_sd->flags & SD_WAKE_AFFINE) {
|
|
|
|
unsigned long tl = this_load;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
unsigned long tl_per_task = cpu_avg_load_per_task(this_cpu);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
2005-06-25 15:57:15 -06:00
|
|
|
* If sync wakeup then subtract the (maximum possible)
|
|
|
|
* effect of the currently running task from the load
|
|
|
|
* of the current CPU:
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2005-06-25 15:57:15 -06:00
|
|
|
if (sync)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
tl -= current->load_weight;
|
2005-06-25 15:57:15 -06:00
|
|
|
|
|
|
|
if ((tl <= load &&
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
tl + target_load(cpu, idx) <= tl_per_task) ||
|
|
|
|
100*(tl + p->load_weight) <= imbalance*load) {
|
2005-06-25 15:57:15 -06:00
|
|
|
/*
|
|
|
|
* This domain has SD_WAKE_AFFINE and
|
|
|
|
* p is cache cold in this domain, and
|
|
|
|
* there is no bad imbalance.
|
|
|
|
*/
|
|
|
|
schedstat_inc(this_sd, ttwu_move_affine);
|
|
|
|
goto out_set_cpu;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Start passive balancing when half the imbalance_pct
|
|
|
|
* limit is reached.
|
|
|
|
*/
|
|
|
|
if (this_sd->flags & SD_WAKE_BALANCE) {
|
|
|
|
if (imbalance*this_load <= 100*load) {
|
|
|
|
schedstat_inc(this_sd, ttwu_move_balance);
|
|
|
|
goto out_set_cpu;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
new_cpu = cpu; /* Could not wake to this_cpu. Wake to cpu instead */
|
|
|
|
out_set_cpu:
|
|
|
|
new_cpu = wake_idle(new_cpu, p);
|
|
|
|
if (new_cpu != cpu) {
|
|
|
|
set_task_cpu(p, new_cpu);
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
/* might preempt at this point */
|
|
|
|
rq = task_rq_lock(p, &flags);
|
|
|
|
old_state = p->state;
|
|
|
|
if (!(old_state & state))
|
|
|
|
goto out;
|
|
|
|
if (p->array)
|
|
|
|
goto out_running;
|
|
|
|
|
|
|
|
this_cpu = smp_processor_id();
|
|
|
|
cpu = task_cpu(p);
|
|
|
|
}
|
|
|
|
|
|
|
|
out_activate:
|
|
|
|
#endif /* CONFIG_SMP */
|
|
|
|
if (old_state == TASK_UNINTERRUPTIBLE) {
|
|
|
|
rq->nr_uninterruptible--;
|
|
|
|
/*
|
|
|
|
* Tasks on involuntary sleep don't earn
|
|
|
|
* sleep_avg beyond just interactive state.
|
|
|
|
*/
|
2006-03-31 03:31:23 -07:00
|
|
|
p->sleep_type = SLEEP_NONINTERACTIVE;
|
2006-03-31 03:31:25 -07:00
|
|
|
} else
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-09-10 01:26:12 -06:00
|
|
|
/*
|
|
|
|
* Tasks that have marked their sleep as noninteractive get
|
2006-03-31 03:31:25 -07:00
|
|
|
* woken up with their sleep average not weighted in an
|
|
|
|
* interactive way.
|
2005-09-10 01:26:12 -06:00
|
|
|
*/
|
2006-03-31 03:31:25 -07:00
|
|
|
if (old_state & TASK_NONINTERACTIVE)
|
|
|
|
p->sleep_type = SLEEP_NONINTERACTIVE;
|
|
|
|
|
|
|
|
|
|
|
|
activate_task(p, rq, cpu == this_cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* Sync wakeups (i.e. those types of wakeups where the waker
|
|
|
|
* has indicated that it will leave the CPU in short order)
|
|
|
|
* don't trigger a preemption, if the woken up task will run on
|
|
|
|
* this cpu. (in this case the 'I will reschedule' promise of
|
|
|
|
* the waker guarantees that the freshly woken up task is going
|
|
|
|
* to be considered on this CPU.)
|
|
|
|
*/
|
|
|
|
if (!sync || cpu != this_cpu) {
|
|
|
|
if (TASK_PREEMPTS_CURR(p, rq))
|
|
|
|
resched_task(rq->curr);
|
|
|
|
}
|
|
|
|
success = 1;
|
|
|
|
|
|
|
|
out_running:
|
|
|
|
p->state = TASK_RUNNING;
|
|
|
|
out:
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
|
|
|
|
return success;
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
int fastcall wake_up_process(struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return try_to_wake_up(p, TASK_STOPPED | TASK_TRACED |
|
|
|
|
TASK_INTERRUPTIBLE | TASK_UNINTERRUPTIBLE, 0);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(wake_up_process);
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
int fastcall wake_up_state(struct task_struct *p, unsigned int state)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return try_to_wake_up(p, state, 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Perform scheduler related setup for a newly forked process p.
|
|
|
|
* p is forked by current.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
void fastcall sched_fork(struct task_struct *p, int clone_flags)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-06-25 15:57:29 -06:00
|
|
|
int cpu = get_cpu();
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
cpu = sched_balance_self(cpu, SD_BALANCE_FORK);
|
|
|
|
#endif
|
|
|
|
set_task_cpu(p, cpu);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* We mark the process as running here, but have not actually
|
|
|
|
* inserted it onto the runqueue yet. This guarantees that
|
|
|
|
* nobody will actually run it, and a signal or other external
|
|
|
|
* event cannot wake it up and insert it on the runqueue either.
|
|
|
|
*/
|
|
|
|
p->state = TASK_RUNNING;
|
2006-06-27 03:54:51 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Make sure we do not leak PI boosting priority to the child:
|
|
|
|
*/
|
|
|
|
p->prio = current->normal_prio;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
INIT_LIST_HEAD(&p->run_list);
|
|
|
|
p->array = NULL;
|
2006-07-14 01:24:38 -06:00
|
|
|
#if defined(CONFIG_SCHEDSTATS) || defined(CONFIG_TASK_DELAY_ACCT)
|
|
|
|
if (unlikely(sched_info_on()))
|
|
|
|
memset(&p->sched_info, 0, sizeof(p->sched_info));
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
[PATCH] sched: revert "filter affine wakeups"
Revert commit d7102e95b7b9c00277562c29aad421d2d521c5f6:
[PATCH] sched: filter affine wakeups
Apparently caused more than 10% performance regression for aim7 benchmark.
The setup in use is 16-cpu HP rx8620, 64Gb of memory and 12 MSA1000s with 144
disks. Each disk is 72Gb with a single ext3 filesystem (courtesy of HP, who
supplied benchmark results).
The problem is, for aim7, the wake-up pattern is random, but it still needs
load balancing action in the wake-up path to achieve best performance. With
the above commit, lack of load balancing hurts that workload.
However, for workloads like database transaction processing, the requirement
is exactly opposite. In the wake up path, best performance is achieved with
absolutely zero load balancing. We simply wake up the process on the CPU that
it was previously run. Worst performance is obtained when we do load
balancing at wake up.
There isn't an easy way to auto detect the workload characteristics. Ingo's
earlier patch that detects idle CPU and decide whether to load balance or not
doesn't perform with aim7 either since all CPUs are busy (it causes even
bigger perf. regression).
Revert commit d7102e95b7b9c00277562c29aad421d2d521c5f6, which causes more
than 10% performance regression with aim7.
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-02-14 14:53:10 -07:00
|
|
|
#if defined(CONFIG_SMP) && defined(__ARCH_WANT_UNLOCKED_CTXSW)
|
2005-06-25 15:57:23 -06:00
|
|
|
p->oncpu = 0;
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_PREEMPT
|
2005-06-25 15:57:23 -06:00
|
|
|
/* Want to start with kernel preemption disabled. */
|
2005-11-13 17:06:55 -07:00
|
|
|
task_thread_info(p)->preempt_count = 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
|
|
|
/*
|
|
|
|
* Share the timeslice between parent and child, thus the
|
|
|
|
* total amount of pending timeslices in the system doesn't change,
|
|
|
|
* resulting in more scheduling fairness.
|
|
|
|
*/
|
|
|
|
local_irq_disable();
|
|
|
|
p->time_slice = (current->time_slice + 1) >> 1;
|
|
|
|
/*
|
|
|
|
* The remainder of the first timeslice might be recovered by
|
|
|
|
* the parent if the child exits early enough.
|
|
|
|
*/
|
|
|
|
p->first_time_slice = 1;
|
|
|
|
current->time_slice >>= 1;
|
|
|
|
p->timestamp = sched_clock();
|
|
|
|
if (unlikely(!current->time_slice)) {
|
|
|
|
/*
|
|
|
|
* This case is rare, it happens when the parent has only
|
|
|
|
* a single jiffy left from its timeslice. Taking the
|
|
|
|
* runqueue lock is not a problem.
|
|
|
|
*/
|
|
|
|
current->time_slice = 1;
|
|
|
|
scheduler_tick();
|
2005-06-25 15:57:29 -06:00
|
|
|
}
|
|
|
|
local_irq_enable();
|
|
|
|
put_cpu();
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* wake_up_new_task - wake up a newly created task for the first time.
|
|
|
|
*
|
|
|
|
* This function will do some initial scheduler statistics housekeeping
|
|
|
|
* that must be done for every newly created context, then puts the task
|
|
|
|
* on the runqueue and wakes it.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
void fastcall wake_up_new_task(struct task_struct *p, unsigned long clone_flags)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq, *this_rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long flags;
|
|
|
|
int this_cpu, cpu;
|
|
|
|
|
|
|
|
rq = task_rq_lock(p, &flags);
|
2005-06-25 15:57:19 -06:00
|
|
|
BUG_ON(p->state != TASK_RUNNING);
|
2005-04-16 16:20:36 -06:00
|
|
|
this_cpu = smp_processor_id();
|
2005-06-25 15:57:19 -06:00
|
|
|
cpu = task_cpu(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We decrease the sleep average of forking parents
|
|
|
|
* and children as well, to keep max-interactive tasks
|
|
|
|
* from forking tasks that are max-interactive. The parent
|
|
|
|
* (current) is done further down, under its lock.
|
|
|
|
*/
|
|
|
|
p->sleep_avg = JIFFIES_TO_NS(CURRENT_BONUS(p) *
|
|
|
|
CHILD_PENALTY / 100 * MAX_SLEEP_AVG / MAX_BONUS);
|
|
|
|
|
|
|
|
p->prio = effective_prio(p);
|
|
|
|
|
|
|
|
if (likely(cpu == this_cpu)) {
|
|
|
|
if (!(clone_flags & CLONE_VM)) {
|
|
|
|
/*
|
|
|
|
* The VM isn't cloned, so we're in a good position to
|
|
|
|
* do child-runs-first in anticipation of an exec. This
|
|
|
|
* usually avoids a lot of COW overhead.
|
|
|
|
*/
|
|
|
|
if (unlikely(!current->array))
|
|
|
|
__activate_task(p, rq);
|
|
|
|
else {
|
|
|
|
p->prio = current->prio;
|
2006-06-27 03:54:51 -06:00
|
|
|
p->normal_prio = current->normal_prio;
|
2005-04-16 16:20:36 -06:00
|
|
|
list_add_tail(&p->run_list, ¤t->run_list);
|
|
|
|
p->array = current->array;
|
|
|
|
p->array->nr_active++;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
inc_nr_running(p, rq);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
set_need_resched();
|
|
|
|
} else
|
|
|
|
/* Run child last */
|
|
|
|
__activate_task(p, rq);
|
|
|
|
/*
|
|
|
|
* We skip the following code due to cpu == this_cpu
|
|
|
|
*
|
|
|
|
* task_rq_unlock(rq, &flags);
|
|
|
|
* this_rq = task_rq_lock(current, &flags);
|
|
|
|
*/
|
|
|
|
this_rq = rq;
|
|
|
|
} else {
|
|
|
|
this_rq = cpu_rq(this_cpu);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Not the local CPU - must adjust timestamp. This should
|
|
|
|
* get optimised away in the !CONFIG_SMP case.
|
|
|
|
*/
|
|
|
|
p->timestamp = (p->timestamp - this_rq->timestamp_last_tick)
|
|
|
|
+ rq->timestamp_last_tick;
|
|
|
|
__activate_task(p, rq);
|
|
|
|
if (TASK_PREEMPTS_CURR(p, rq))
|
|
|
|
resched_task(rq->curr);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Parent and child are on different CPUs, now get the
|
|
|
|
* parent runqueue to update the parent's ->sleep_avg:
|
|
|
|
*/
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
this_rq = task_rq_lock(current, &flags);
|
|
|
|
}
|
|
|
|
current->sleep_avg = JIFFIES_TO_NS(CURRENT_BONUS(current) *
|
|
|
|
PARENT_PENALTY / 100 * MAX_SLEEP_AVG / MAX_BONUS);
|
|
|
|
task_rq_unlock(this_rq, &flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Potentially available exiting-child timeslices are
|
|
|
|
* retrieved here - this way the parent does not get
|
|
|
|
* penalized for creating too many threads.
|
|
|
|
*
|
|
|
|
* (this cannot be used to 'generate' timeslices
|
|
|
|
* artificially, because any timeslice recovered here
|
|
|
|
* was given away by the parent in the first place.)
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
void fastcall sched_exit(struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
unsigned long flags;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If the child was a (relative-) CPU hog then decrease
|
|
|
|
* the sleep_avg of the parent as well.
|
|
|
|
*/
|
|
|
|
rq = task_rq_lock(p->parent, &flags);
|
2005-11-04 08:54:30 -07:00
|
|
|
if (p->first_time_slice && task_cpu(p) == task_cpu(p->parent)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
p->parent->time_slice += p->time_slice;
|
|
|
|
if (unlikely(p->parent->time_slice > task_timeslice(p)))
|
|
|
|
p->parent->time_slice = task_timeslice(p);
|
|
|
|
}
|
|
|
|
if (p->sleep_avg < p->parent->sleep_avg)
|
|
|
|
p->parent->sleep_avg = p->parent->sleep_avg /
|
|
|
|
(EXIT_WEIGHT + 1) * EXIT_WEIGHT + p->sleep_avg /
|
|
|
|
(EXIT_WEIGHT + 1);
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
}
|
|
|
|
|
2005-06-25 15:57:23 -06:00
|
|
|
/**
|
|
|
|
* prepare_task_switch - prepare to switch tasks
|
|
|
|
* @rq: the runqueue preparing to switch
|
|
|
|
* @next: the task we are going to switch to.
|
|
|
|
*
|
|
|
|
* This is called with the rq lock held and interrupts off. It must
|
|
|
|
* be paired with a subsequent finish_task_switch after the context
|
|
|
|
* switch.
|
|
|
|
*
|
|
|
|
* prepare_task_switch sets up locking and calls architecture specific
|
|
|
|
* hooks.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void prepare_task_switch(struct rq *rq, struct task_struct *next)
|
2005-06-25 15:57:23 -06:00
|
|
|
{
|
|
|
|
prepare_lock_switch(rq, next);
|
|
|
|
prepare_arch_switch(next);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/**
|
|
|
|
* finish_task_switch - clean up after a task-switch
|
2005-09-06 23:15:17 -06:00
|
|
|
* @rq: runqueue associated with task-switch
|
2005-04-16 16:20:36 -06:00
|
|
|
* @prev: the thread we just switched away from.
|
|
|
|
*
|
2005-06-25 15:57:23 -06:00
|
|
|
* finish_task_switch must be called after the context switch, paired
|
|
|
|
* with a prepare_task_switch call before the context switch.
|
|
|
|
* finish_task_switch will reconcile locking set up by prepare_task_switch,
|
|
|
|
* and do any other architecture-specific cleanup actions.
|
2005-04-16 16:20:36 -06:00
|
|
|
*
|
|
|
|
* Note that we may have delayed dropping an mm in context_switch(). If
|
|
|
|
* so, we finish that here outside of the runqueue lock. (Doing it
|
|
|
|
* with the lock held can cause deadlocks; see schedule() for
|
|
|
|
* details.)
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void finish_task_switch(struct rq *rq, struct task_struct *prev)
|
2005-04-16 16:20:36 -06:00
|
|
|
__releases(rq->lock)
|
|
|
|
{
|
|
|
|
struct mm_struct *mm = rq->prev_mm;
|
|
|
|
unsigned long prev_task_flags;
|
|
|
|
|
|
|
|
rq->prev_mm = NULL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* A task struct has one reference for the use as "current".
|
|
|
|
* If a task dies, then it sets EXIT_ZOMBIE in tsk->exit_state and
|
|
|
|
* calls schedule one last time. The schedule call will never return,
|
|
|
|
* and the scheduled task must drop that reference.
|
|
|
|
* The test for EXIT_ZOMBIE must occur while the runqueue locks are
|
|
|
|
* still held, otherwise prev could be scheduled on another cpu, die
|
|
|
|
* there before we look at prev->state, and then the reference would
|
|
|
|
* be dropped twice.
|
|
|
|
* Manfred Spraul <manfred@colorfullife.com>
|
|
|
|
*/
|
|
|
|
prev_task_flags = prev->flags;
|
2005-06-25 15:57:23 -06:00
|
|
|
finish_arch_switch(prev);
|
|
|
|
finish_lock_switch(rq, prev);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (mm)
|
|
|
|
mmdrop(mm);
|
2006-03-26 02:38:20 -07:00
|
|
|
if (unlikely(prev_task_flags & PF_DEAD)) {
|
|
|
|
/*
|
|
|
|
* Remove function-return probe instances associated with this
|
|
|
|
* task and put them back on the free list.
|
|
|
|
*/
|
|
|
|
kprobe_flush_task(prev);
|
2005-04-16 16:20:36 -06:00
|
|
|
put_task_struct(prev);
|
2006-03-26 02:38:20 -07:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* schedule_tail - first thing a freshly forked thread must call.
|
|
|
|
* @prev: the thread we just switched away from.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
asmlinkage void schedule_tail(struct task_struct *prev)
|
2005-04-16 16:20:36 -06:00
|
|
|
__releases(rq->lock)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = this_rq();
|
|
|
|
|
2005-06-25 15:57:23 -06:00
|
|
|
finish_task_switch(rq, prev);
|
|
|
|
#ifdef __ARCH_WANT_UNLOCKED_CTXSW
|
|
|
|
/* In this case, finish_task_switch does not reenable preemption */
|
|
|
|
preempt_enable();
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
if (current->set_child_tid)
|
|
|
|
put_user(current->pid, current->set_child_tid);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* context_switch - switch to the new MM and the new
|
|
|
|
* thread's register state.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline struct task_struct *
|
2006-07-03 01:25:42 -06:00
|
|
|
context_switch(struct rq *rq, struct task_struct *prev,
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *next)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct mm_struct *mm = next->mm;
|
|
|
|
struct mm_struct *oldmm = prev->active_mm;
|
|
|
|
|
|
|
|
if (unlikely(!mm)) {
|
|
|
|
next->active_mm = oldmm;
|
|
|
|
atomic_inc(&oldmm->mm_count);
|
|
|
|
enter_lazy_tlb(oldmm, next);
|
|
|
|
} else
|
|
|
|
switch_mm(oldmm, mm, next);
|
|
|
|
|
|
|
|
if (unlikely(!prev->mm)) {
|
|
|
|
prev->active_mm = NULL;
|
|
|
|
WARN_ON(rq->prev_mm);
|
|
|
|
rq->prev_mm = oldmm;
|
|
|
|
}
|
2006-07-14 01:24:27 -06:00
|
|
|
/*
|
|
|
|
* Since the runqueue lock will be released by the next
|
|
|
|
* task (which is an invalid locking op but in the case
|
|
|
|
* of the scheduler it's an obvious special-case), so we
|
|
|
|
* do an early lockdep release here:
|
|
|
|
*/
|
|
|
|
#ifndef __ARCH_WANT_UNLOCKED_CTXSW
|
2006-07-03 01:24:54 -06:00
|
|
|
spin_release(&rq->lock.dep_map, 1, _THIS_IP_);
|
2006-07-14 01:24:27 -06:00
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Here we just switch the register state and the stack. */
|
|
|
|
switch_to(prev, next, prev);
|
|
|
|
|
|
|
|
return prev;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* nr_running, nr_uninterruptible and nr_context_switches:
|
|
|
|
*
|
|
|
|
* externally visible scheduler statistics: current number of runnable
|
|
|
|
* threads, current number of uninterruptible-sleeping threads, total
|
|
|
|
* number of context switches performed since bootup.
|
|
|
|
*/
|
|
|
|
unsigned long nr_running(void)
|
|
|
|
{
|
|
|
|
unsigned long i, sum = 0;
|
|
|
|
|
|
|
|
for_each_online_cpu(i)
|
|
|
|
sum += cpu_rq(i)->nr_running;
|
|
|
|
|
|
|
|
return sum;
|
|
|
|
}
|
|
|
|
|
|
|
|
unsigned long nr_uninterruptible(void)
|
|
|
|
{
|
|
|
|
unsigned long i, sum = 0;
|
|
|
|
|
2006-03-28 02:56:37 -07:00
|
|
|
for_each_possible_cpu(i)
|
2005-04-16 16:20:36 -06:00
|
|
|
sum += cpu_rq(i)->nr_uninterruptible;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Since we read the counters lockless, it might be slightly
|
|
|
|
* inaccurate. Do not allow it to go below zero though:
|
|
|
|
*/
|
|
|
|
if (unlikely((long)sum < 0))
|
|
|
|
sum = 0;
|
|
|
|
|
|
|
|
return sum;
|
|
|
|
}
|
|
|
|
|
|
|
|
unsigned long long nr_context_switches(void)
|
|
|
|
{
|
2006-06-27 03:54:31 -06:00
|
|
|
int i;
|
|
|
|
unsigned long long sum = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-03-28 02:56:37 -07:00
|
|
|
for_each_possible_cpu(i)
|
2005-04-16 16:20:36 -06:00
|
|
|
sum += cpu_rq(i)->nr_switches;
|
|
|
|
|
|
|
|
return sum;
|
|
|
|
}
|
|
|
|
|
|
|
|
unsigned long nr_iowait(void)
|
|
|
|
{
|
|
|
|
unsigned long i, sum = 0;
|
|
|
|
|
2006-03-28 02:56:37 -07:00
|
|
|
for_each_possible_cpu(i)
|
2005-04-16 16:20:36 -06:00
|
|
|
sum += atomic_read(&cpu_rq(i)->nr_iowait);
|
|
|
|
|
|
|
|
return sum;
|
|
|
|
}
|
|
|
|
|
2006-03-31 03:31:21 -07:00
|
|
|
unsigned long nr_active(void)
|
|
|
|
{
|
|
|
|
unsigned long i, running = 0, uninterruptible = 0;
|
|
|
|
|
|
|
|
for_each_online_cpu(i) {
|
|
|
|
running += cpu_rq(i)->nr_running;
|
|
|
|
uninterruptible += cpu_rq(i)->nr_uninterruptible;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (unlikely((long)uninterruptible < 0))
|
|
|
|
uninterruptible = 0;
|
|
|
|
|
|
|
|
return running + uninterruptible;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
/*
|
|
|
|
* Is this task likely cache-hot:
|
|
|
|
*/
|
|
|
|
static inline int
|
|
|
|
task_hot(struct task_struct *p, unsigned long long now, struct sched_domain *sd)
|
|
|
|
{
|
|
|
|
return (long long)(now - p->last_ran) < (long long)sd->cache_hot_time;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* double_rq_lock - safely lock two runqueues
|
|
|
|
*
|
|
|
|
* Note this does not disable interrupts like task_rq_lock,
|
|
|
|
* you need to do so manually before calling.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void double_rq_lock(struct rq *rq1, struct rq *rq2)
|
2005-04-16 16:20:36 -06:00
|
|
|
__acquires(rq1->lock)
|
|
|
|
__acquires(rq2->lock)
|
|
|
|
{
|
|
|
|
if (rq1 == rq2) {
|
|
|
|
spin_lock(&rq1->lock);
|
|
|
|
__acquire(rq2->lock); /* Fake it out ;) */
|
|
|
|
} else {
|
2006-06-27 03:54:28 -06:00
|
|
|
if (rq1 < rq2) {
|
2005-04-16 16:20:36 -06:00
|
|
|
spin_lock(&rq1->lock);
|
|
|
|
spin_lock(&rq2->lock);
|
|
|
|
} else {
|
|
|
|
spin_lock(&rq2->lock);
|
|
|
|
spin_lock(&rq1->lock);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* double_rq_unlock - safely unlock two runqueues
|
|
|
|
*
|
|
|
|
* Note this does not restore interrupts like task_rq_unlock,
|
|
|
|
* you need to do so manually after calling.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void double_rq_unlock(struct rq *rq1, struct rq *rq2)
|
2005-04-16 16:20:36 -06:00
|
|
|
__releases(rq1->lock)
|
|
|
|
__releases(rq2->lock)
|
|
|
|
{
|
|
|
|
spin_unlock(&rq1->lock);
|
|
|
|
if (rq1 != rq2)
|
|
|
|
spin_unlock(&rq2->lock);
|
|
|
|
else
|
|
|
|
__release(rq2->lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* double_lock_balance - lock the busiest runqueue, this_rq is locked already.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void double_lock_balance(struct rq *this_rq, struct rq *busiest)
|
2005-04-16 16:20:36 -06:00
|
|
|
__releases(this_rq->lock)
|
|
|
|
__acquires(busiest->lock)
|
|
|
|
__acquires(this_rq->lock)
|
|
|
|
{
|
|
|
|
if (unlikely(!spin_trylock(&busiest->lock))) {
|
2006-06-27 03:54:28 -06:00
|
|
|
if (busiest < this_rq) {
|
2005-04-16 16:20:36 -06:00
|
|
|
spin_unlock(&this_rq->lock);
|
|
|
|
spin_lock(&busiest->lock);
|
|
|
|
spin_lock(&this_rq->lock);
|
|
|
|
} else
|
|
|
|
spin_lock(&busiest->lock);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If dest_cpu is allowed for this process, migrate the task to it.
|
|
|
|
* This is accomplished by forcing the cpu_allowed mask to only
|
|
|
|
* allow dest_cpu, which will force the cpu onto dest_cpu. Then
|
|
|
|
* the cpu_allowed mask is restored.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static void sched_migrate_task(struct task_struct *p, int dest_cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct migration_req req;
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long flags;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
rq = task_rq_lock(p, &flags);
|
|
|
|
if (!cpu_isset(dest_cpu, p->cpus_allowed)
|
|
|
|
|| unlikely(cpu_is_offline(dest_cpu)))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* force the process onto the specified CPU */
|
|
|
|
if (migrate_task(p, dest_cpu, &req)) {
|
|
|
|
/* Need to wait for migration thread (might exit: take ref). */
|
|
|
|
struct task_struct *mt = rq->migration_thread;
|
2006-07-03 01:25:41 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
get_task_struct(mt);
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
wake_up_process(mt);
|
|
|
|
put_task_struct(mt);
|
|
|
|
wait_for_completion(&req.done);
|
2006-07-03 01:25:41 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
out:
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2005-06-25 15:57:29 -06:00
|
|
|
* sched_exec - execve() is a valuable balancing opportunity, because at
|
|
|
|
* this point the task has the smallest effective memory and cache footprint.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
|
|
|
void sched_exec(void)
|
|
|
|
{
|
|
|
|
int new_cpu, this_cpu = get_cpu();
|
2005-06-25 15:57:29 -06:00
|
|
|
new_cpu = sched_balance_self(this_cpu, SD_BALANCE_EXEC);
|
2005-04-16 16:20:36 -06:00
|
|
|
put_cpu();
|
2005-06-25 15:57:29 -06:00
|
|
|
if (new_cpu != this_cpu)
|
|
|
|
sched_migrate_task(current, new_cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* pull_task - move a task from a remote runqueue to the local runqueue.
|
|
|
|
* Both runqueues must be locked.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void pull_task(struct rq *src_rq, struct prio_array *src_array,
|
|
|
|
struct task_struct *p, struct rq *this_rq,
|
|
|
|
struct prio_array *this_array, int this_cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
dequeue_task(p, src_array);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
dec_nr_running(p, src_rq);
|
2005-04-16 16:20:36 -06:00
|
|
|
set_task_cpu(p, this_cpu);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
inc_nr_running(p, this_rq);
|
2005-04-16 16:20:36 -06:00
|
|
|
enqueue_task(p, this_array);
|
|
|
|
p->timestamp = (p->timestamp - src_rq->timestamp_last_tick)
|
|
|
|
+ this_rq->timestamp_last_tick;
|
|
|
|
/*
|
|
|
|
* Note that idle threads have a prio of MAX_PRIO, for this test
|
|
|
|
* to be always true for them.
|
|
|
|
*/
|
|
|
|
if (TASK_PREEMPTS_CURR(p, this_rq))
|
|
|
|
resched_task(this_rq->curr);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* can_migrate_task - may task p from runqueue rq be migrated to this_cpu?
|
|
|
|
*/
|
2006-01-14 14:20:43 -07:00
|
|
|
static
|
2006-07-03 01:25:42 -06:00
|
|
|
int can_migrate_task(struct task_struct *p, struct rq *rq, int this_cpu,
|
2005-09-10 01:26:11 -06:00
|
|
|
struct sched_domain *sd, enum idle_type idle,
|
|
|
|
int *all_pinned)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We do not migrate tasks that are:
|
|
|
|
* 1) running (obviously), or
|
|
|
|
* 2) cannot be migrated to this CPU due to cpus_allowed, or
|
|
|
|
* 3) are cache-hot on their current CPU.
|
|
|
|
*/
|
|
|
|
if (!cpu_isset(this_cpu, p->cpus_allowed))
|
|
|
|
return 0;
|
2005-06-25 15:57:07 -06:00
|
|
|
*all_pinned = 0;
|
|
|
|
|
|
|
|
if (task_running(rq, p))
|
|
|
|
return 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Aggressive migration if:
|
2005-06-25 15:57:17 -06:00
|
|
|
* 1) task is cache cold, or
|
2005-04-16 16:20:36 -06:00
|
|
|
* 2) too many balance attempts have failed.
|
|
|
|
*/
|
|
|
|
|
2005-06-25 15:57:17 -06:00
|
|
|
if (sd->nr_balance_failed > sd->cache_nice_tries)
|
2005-04-16 16:20:36 -06:00
|
|
|
return 1;
|
|
|
|
|
|
|
|
if (task_hot(p, rq->timestamp_last_tick, sd))
|
2005-06-25 15:57:07 -06:00
|
|
|
return 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2006-06-27 03:54:37 -06:00
|
|
|
#define rq_best_prio(rq) min((rq)->curr->prio, (rq)->best_expired_prio)
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
* move_tasks tries to move up to max_nr_move tasks and max_load_move weighted
|
|
|
|
* load from busiest to this_rq, as part of a balancing operation within
|
|
|
|
* "domain". Returns the number of tasks moved.
|
2005-04-16 16:20:36 -06:00
|
|
|
*
|
|
|
|
* Called with both runqueues locked.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static int move_tasks(struct rq *this_rq, int this_cpu, struct rq *busiest,
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
unsigned long max_nr_move, unsigned long max_load_move,
|
|
|
|
struct sched_domain *sd, enum idle_type idle,
|
|
|
|
int *all_pinned)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
int idx, pulled = 0, pinned = 0, this_best_prio, best_prio,
|
|
|
|
best_prio_seen, skip_for_load;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct prio_array *array, *dst_array;
|
2005-04-16 16:20:36 -06:00
|
|
|
struct list_head *head, *curr;
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *tmp;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
long rem_load_move;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
if (max_nr_move == 0 || max_load_move == 0)
|
2005-04-16 16:20:36 -06:00
|
|
|
goto out;
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
rem_load_move = max_load_move;
|
2005-06-25 15:57:07 -06:00
|
|
|
pinned = 1;
|
2006-06-27 03:54:37 -06:00
|
|
|
this_best_prio = rq_best_prio(this_rq);
|
2006-07-03 01:25:40 -06:00
|
|
|
best_prio = rq_best_prio(busiest);
|
2006-06-27 03:54:37 -06:00
|
|
|
/*
|
|
|
|
* Enable handling of the case where there is more than one task
|
|
|
|
* with the best priority. If the current running task is one
|
2006-07-03 01:25:40 -06:00
|
|
|
* of those with prio==best_prio we know it won't be moved
|
2006-06-27 03:54:37 -06:00
|
|
|
* and therefore it's safe to override the skip (based on load) of
|
|
|
|
* any task we find with that prio.
|
|
|
|
*/
|
2006-07-03 01:25:40 -06:00
|
|
|
best_prio_seen = best_prio == busiest->curr->prio;
|
2005-06-25 15:57:07 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* We first consider expired tasks. Those will likely not be
|
|
|
|
* executed in the near future, and they are most likely to
|
|
|
|
* be cache-cold, thus switching CPUs has the least effect
|
|
|
|
* on them.
|
|
|
|
*/
|
|
|
|
if (busiest->expired->nr_active) {
|
|
|
|
array = busiest->expired;
|
|
|
|
dst_array = this_rq->expired;
|
|
|
|
} else {
|
|
|
|
array = busiest->active;
|
|
|
|
dst_array = this_rq->active;
|
|
|
|
}
|
|
|
|
|
|
|
|
new_array:
|
|
|
|
/* Start searching at priority 0: */
|
|
|
|
idx = 0;
|
|
|
|
skip_bitmap:
|
|
|
|
if (!idx)
|
|
|
|
idx = sched_find_first_bit(array->bitmap);
|
|
|
|
else
|
|
|
|
idx = find_next_bit(array->bitmap, MAX_PRIO, idx);
|
|
|
|
if (idx >= MAX_PRIO) {
|
|
|
|
if (array == busiest->expired && busiest->active->nr_active) {
|
|
|
|
array = busiest->active;
|
|
|
|
dst_array = this_rq->active;
|
|
|
|
goto new_array;
|
|
|
|
}
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
head = array->queue + idx;
|
|
|
|
curr = head->prev;
|
|
|
|
skip_queue:
|
2006-07-03 01:25:41 -06:00
|
|
|
tmp = list_entry(curr, struct task_struct, run_list);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
curr = curr->prev;
|
|
|
|
|
2006-06-27 03:54:36 -06:00
|
|
|
/*
|
|
|
|
* To help distribute high priority tasks accross CPUs we don't
|
|
|
|
* skip a task if it will be the highest priority task (i.e. smallest
|
|
|
|
* prio value) on its new queue regardless of its load weight
|
|
|
|
*/
|
2006-06-27 03:54:37 -06:00
|
|
|
skip_for_load = tmp->load_weight > rem_load_move;
|
|
|
|
if (skip_for_load && idx < this_best_prio)
|
2006-07-03 01:25:40 -06:00
|
|
|
skip_for_load = !best_prio_seen && idx == best_prio;
|
2006-06-27 03:54:37 -06:00
|
|
|
if (skip_for_load ||
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
!can_migrate_task(tmp, busiest, this_cpu, sd, idle, &pinned)) {
|
2006-07-03 01:25:40 -06:00
|
|
|
|
|
|
|
best_prio_seen |= idx == best_prio;
|
2005-04-16 16:20:36 -06:00
|
|
|
if (curr != head)
|
|
|
|
goto skip_queue;
|
|
|
|
idx++;
|
|
|
|
goto skip_bitmap;
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_SCHEDSTATS
|
|
|
|
if (task_hot(tmp, busiest->timestamp_last_tick, sd))
|
|
|
|
schedstat_inc(sd, lb_hot_gained[idle]);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
pull_task(busiest, array, tmp, this_rq, dst_array, this_cpu);
|
|
|
|
pulled++;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
rem_load_move -= tmp->load_weight;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
/*
|
|
|
|
* We only want to steal up to the prescribed number of tasks
|
|
|
|
* and the prescribed amount of weighted load.
|
|
|
|
*/
|
|
|
|
if (pulled < max_nr_move && rem_load_move > 0) {
|
2006-06-27 03:54:37 -06:00
|
|
|
if (idx < this_best_prio)
|
|
|
|
this_best_prio = idx;
|
2005-04-16 16:20:36 -06:00
|
|
|
if (curr != head)
|
|
|
|
goto skip_queue;
|
|
|
|
idx++;
|
|
|
|
goto skip_bitmap;
|
|
|
|
}
|
|
|
|
out:
|
|
|
|
/*
|
|
|
|
* Right now, this is the only place pull_task() is called,
|
|
|
|
* so we can safely collect pull_task() stats here rather than
|
|
|
|
* inside pull_task().
|
|
|
|
*/
|
|
|
|
schedstat_add(sd, lb_gained[idle], pulled);
|
2005-06-25 15:57:07 -06:00
|
|
|
|
|
|
|
if (all_pinned)
|
|
|
|
*all_pinned = pinned;
|
2005-04-16 16:20:36 -06:00
|
|
|
return pulled;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* find_busiest_group finds and returns the busiest CPU group within the
|
2006-07-03 01:25:40 -06:00
|
|
|
* domain. It calculates and returns the amount of weighted load which
|
|
|
|
* should be moved to restore balance via the imbalance parameter.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
|
|
|
static struct sched_group *
|
|
|
|
find_busiest_group(struct sched_domain *sd, int this_cpu,
|
2005-09-10 01:26:19 -06:00
|
|
|
unsigned long *imbalance, enum idle_type idle, int *sd_idle)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct sched_group *busiest = NULL, *this = NULL, *group = sd->groups;
|
|
|
|
unsigned long max_load, avg_load, total_load, this_load, total_pwr;
|
2005-09-10 01:26:21 -06:00
|
|
|
unsigned long max_pull;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
unsigned long busiest_load_per_task, busiest_nr_running;
|
|
|
|
unsigned long this_load_per_task, this_nr_running;
|
2005-06-25 15:57:13 -06:00
|
|
|
int load_idx;
|
2006-06-27 03:54:42 -06:00
|
|
|
#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
|
|
|
|
int power_savings_balance = 1;
|
|
|
|
unsigned long leader_nr_running = 0, min_load_per_task = 0;
|
|
|
|
unsigned long min_nr_running = ULONG_MAX;
|
|
|
|
struct sched_group *group_min = NULL, *group_leader = NULL;
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
max_load = this_load = total_load = total_pwr = 0;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
busiest_load_per_task = busiest_nr_running = 0;
|
|
|
|
this_load_per_task = this_nr_running = 0;
|
2005-06-25 15:57:13 -06:00
|
|
|
if (idle == NOT_IDLE)
|
|
|
|
load_idx = sd->busy_idx;
|
|
|
|
else if (idle == NEWLY_IDLE)
|
|
|
|
load_idx = sd->newidle_idx;
|
|
|
|
else
|
|
|
|
load_idx = sd->idle_idx;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
do {
|
2006-06-27 03:54:42 -06:00
|
|
|
unsigned long load, group_capacity;
|
2005-04-16 16:20:36 -06:00
|
|
|
int local_group;
|
|
|
|
int i;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
unsigned long sum_nr_running, sum_weighted_load;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
local_group = cpu_isset(this_cpu, group->cpumask);
|
|
|
|
|
|
|
|
/* Tally up the load of all CPUs in the group */
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
sum_weighted_load = sum_nr_running = avg_load = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
for_each_cpu_mask(i, group->cpumask) {
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(i);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
|
2005-09-10 01:26:19 -06:00
|
|
|
if (*sd_idle && !idle_cpu(i))
|
|
|
|
*sd_idle = 0;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Bias balancing toward cpus of our domain */
|
|
|
|
if (local_group)
|
[PATCH] sched: remove smpnice
I don't think the code is quite ready, which is why I asked for Peter's
additions to also be merged before I acked it (although it turned out that
it still isn't quite ready with his additions either).
Basically I have had similar observations to Suresh in that it does not
play nicely with the rest of the balancing infrastructure (and raised
similar concerns in my review).
The samples (group of 4) I got for "maximum recorded imbalance" on a 2x2
SMP+HT Xeon are as follows:
| Following boot | hackbench 20 | hackbench 40
-----------+----------------+---------------------+---------------------
2.6.16-rc2 | 30,37,100,112 | 5600,5530,6020,6090 | 6390,7090,8760,8470
+nosmpnice | 3, 2, 4, 2 | 28, 150, 294, 132 | 348, 348, 294, 347
Hackbench raw performance is down around 15% with smpnice (but that in
itself isn't a huge deal because it is just a benchmark). However, the
samples show that the imbalance passed into move_tasks is increased by
about a factor of 10-30. I think this would also go some way to explaining
latency blips turning up in the balancing code (though I haven't actually
measured that).
We'll probably have to revert this in the SUSE kernel.
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Peter Williams <pwil3058@bigpond.net.au>
Cc: "Martin J. Bligh" <mbligh@aracnet.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-02-10 02:51:02 -07:00
|
|
|
load = target_load(i, load_idx);
|
2005-04-16 16:20:36 -06:00
|
|
|
else
|
[PATCH] sched: remove smpnice
I don't think the code is quite ready, which is why I asked for Peter's
additions to also be merged before I acked it (although it turned out that
it still isn't quite ready with his additions either).
Basically I have had similar observations to Suresh in that it does not
play nicely with the rest of the balancing infrastructure (and raised
similar concerns in my review).
The samples (group of 4) I got for "maximum recorded imbalance" on a 2x2
SMP+HT Xeon are as follows:
| Following boot | hackbench 20 | hackbench 40
-----------+----------------+---------------------+---------------------
2.6.16-rc2 | 30,37,100,112 | 5600,5530,6020,6090 | 6390,7090,8760,8470
+nosmpnice | 3, 2, 4, 2 | 28, 150, 294, 132 | 348, 348, 294, 347
Hackbench raw performance is down around 15% with smpnice (but that in
itself isn't a huge deal because it is just a benchmark). However, the
samples show that the imbalance passed into move_tasks is increased by
about a factor of 10-30. I think this would also go some way to explaining
latency blips turning up in the balancing code (though I haven't actually
measured that).
We'll probably have to revert this in the SUSE kernel.
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Peter Williams <pwil3058@bigpond.net.au>
Cc: "Martin J. Bligh" <mbligh@aracnet.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-02-10 02:51:02 -07:00
|
|
|
load = source_load(i, load_idx);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
avg_load += load;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
sum_nr_running += rq->nr_running;
|
|
|
|
sum_weighted_load += rq->raw_weighted_load;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
total_load += avg_load;
|
|
|
|
total_pwr += group->cpu_power;
|
|
|
|
|
|
|
|
/* Adjust by relative CPU power of the group */
|
|
|
|
avg_load = (avg_load * SCHED_LOAD_SCALE) / group->cpu_power;
|
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
group_capacity = group->cpu_power / SCHED_LOAD_SCALE;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (local_group) {
|
|
|
|
this_load = avg_load;
|
|
|
|
this = group;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
this_nr_running = sum_nr_running;
|
|
|
|
this_load_per_task = sum_weighted_load;
|
|
|
|
} else if (avg_load > max_load &&
|
2006-06-27 03:54:42 -06:00
|
|
|
sum_nr_running > group_capacity) {
|
2005-04-16 16:20:36 -06:00
|
|
|
max_load = avg_load;
|
|
|
|
busiest = group;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
busiest_nr_running = sum_nr_running;
|
|
|
|
busiest_load_per_task = sum_weighted_load;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2006-06-27 03:54:42 -06:00
|
|
|
|
|
|
|
#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
|
|
|
|
/*
|
|
|
|
* Busy processors will not participate in power savings
|
|
|
|
* balance.
|
|
|
|
*/
|
|
|
|
if (idle == NOT_IDLE || !(sd->flags & SD_POWERSAVINGS_BALANCE))
|
|
|
|
goto group_next;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If the local group is idle or completely loaded
|
|
|
|
* no need to do power savings balance at this domain
|
|
|
|
*/
|
|
|
|
if (local_group && (this_nr_running >= group_capacity ||
|
|
|
|
!this_nr_running))
|
|
|
|
power_savings_balance = 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If a group is already running at full capacity or idle,
|
|
|
|
* don't include that group in power savings calculations
|
|
|
|
*/
|
|
|
|
if (!power_savings_balance || sum_nr_running >= group_capacity
|
|
|
|
|| !sum_nr_running)
|
|
|
|
goto group_next;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Calculate the group which has the least non-idle load.
|
|
|
|
* This is the group from where we need to pick up the load
|
|
|
|
* for saving power
|
|
|
|
*/
|
|
|
|
if ((sum_nr_running < min_nr_running) ||
|
|
|
|
(sum_nr_running == min_nr_running &&
|
|
|
|
first_cpu(group->cpumask) <
|
|
|
|
first_cpu(group_min->cpumask))) {
|
|
|
|
group_min = group;
|
|
|
|
min_nr_running = sum_nr_running;
|
|
|
|
min_load_per_task = sum_weighted_load /
|
|
|
|
sum_nr_running;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Calculate the group which is almost near its
|
|
|
|
* capacity but still has some space to pick up some load
|
|
|
|
* from other group and save more power
|
|
|
|
*/
|
2006-07-03 01:25:40 -06:00
|
|
|
if (sum_nr_running <= group_capacity - 1) {
|
2006-06-27 03:54:42 -06:00
|
|
|
if (sum_nr_running > leader_nr_running ||
|
|
|
|
(sum_nr_running == leader_nr_running &&
|
|
|
|
first_cpu(group->cpumask) >
|
|
|
|
first_cpu(group_leader->cpumask))) {
|
|
|
|
group_leader = group;
|
|
|
|
leader_nr_running = sum_nr_running;
|
|
|
|
}
|
2006-07-03 01:25:40 -06:00
|
|
|
}
|
2006-06-27 03:54:42 -06:00
|
|
|
group_next:
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
group = group->next;
|
|
|
|
} while (group != sd->groups);
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
if (!busiest || this_load >= max_load || busiest_nr_running == 0)
|
2005-04-16 16:20:36 -06:00
|
|
|
goto out_balanced;
|
|
|
|
|
|
|
|
avg_load = (SCHED_LOAD_SCALE * total_load) / total_pwr;
|
|
|
|
|
|
|
|
if (this_load >= avg_load ||
|
|
|
|
100*max_load <= sd->imbalance_pct*this_load)
|
|
|
|
goto out_balanced;
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
busiest_load_per_task /= busiest_nr_running;
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* We're trying to get all the cpus to the average_load, so we don't
|
|
|
|
* want to push ourselves above the average load, nor do we wish to
|
|
|
|
* reduce the max loaded cpu below the average load, as either of these
|
|
|
|
* actions would just result in more rebalancing later, and ping-pong
|
|
|
|
* tasks around. Thus we look for the minimum possible imbalance.
|
|
|
|
* Negative imbalances (*we* are more loaded than anyone else) will
|
|
|
|
* be counted as no imbalance for these purposes -- we can't fix that
|
|
|
|
* by pulling tasks to us. Be careful of negative numbers as they'll
|
|
|
|
* appear as very large values with unsigned longs.
|
|
|
|
*/
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
if (max_load <= busiest_load_per_task)
|
|
|
|
goto out_balanced;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In the presence of smp nice balancing, certain scenarios can have
|
|
|
|
* max load less than avg load(as we skip the groups at or below
|
|
|
|
* its cpu_power, while calculating max_load..)
|
|
|
|
*/
|
|
|
|
if (max_load < avg_load) {
|
|
|
|
*imbalance = 0;
|
|
|
|
goto small_imbalance;
|
|
|
|
}
|
2005-09-10 01:26:21 -06:00
|
|
|
|
|
|
|
/* Don't want to pull so many tasks that a group would go idle */
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
max_pull = min(max_load - avg_load, max_load - busiest_load_per_task);
|
2005-09-10 01:26:21 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* How much load to actually move to equalise the imbalance */
|
2005-09-10 01:26:21 -06:00
|
|
|
*imbalance = min(max_pull * busiest->cpu_power,
|
2005-04-16 16:20:36 -06:00
|
|
|
(avg_load - this_load) * this->cpu_power)
|
|
|
|
/ SCHED_LOAD_SCALE;
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
/*
|
|
|
|
* if *imbalance is less than the average load per runnable task
|
|
|
|
* there is no gaurantee that any tasks will be moved so we'll have
|
|
|
|
* a think about bumping its value to force at least one task to be
|
|
|
|
* moved
|
|
|
|
*/
|
|
|
|
if (*imbalance < busiest_load_per_task) {
|
2006-07-03 01:25:40 -06:00
|
|
|
unsigned long tmp, pwr_now, pwr_move;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
unsigned int imbn;
|
|
|
|
|
|
|
|
small_imbalance:
|
|
|
|
pwr_move = pwr_now = 0;
|
|
|
|
imbn = 2;
|
|
|
|
if (this_nr_running) {
|
|
|
|
this_load_per_task /= this_nr_running;
|
|
|
|
if (busiest_load_per_task > this_load_per_task)
|
|
|
|
imbn = 1;
|
|
|
|
} else
|
|
|
|
this_load_per_task = SCHED_LOAD_SCALE;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
if (max_load - this_load >= busiest_load_per_task * imbn) {
|
|
|
|
*imbalance = busiest_load_per_task;
|
2005-04-16 16:20:36 -06:00
|
|
|
return busiest;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* OK, we don't have enough imbalance to justify moving tasks,
|
|
|
|
* however we may be able to increase total CPU power used by
|
|
|
|
* moving them.
|
|
|
|
*/
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
pwr_now += busiest->cpu_power *
|
|
|
|
min(busiest_load_per_task, max_load);
|
|
|
|
pwr_now += this->cpu_power *
|
|
|
|
min(this_load_per_task, this_load);
|
2005-04-16 16:20:36 -06:00
|
|
|
pwr_now /= SCHED_LOAD_SCALE;
|
|
|
|
|
|
|
|
/* Amount of load we'd subtract */
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
tmp = busiest_load_per_task*SCHED_LOAD_SCALE/busiest->cpu_power;
|
2005-04-16 16:20:36 -06:00
|
|
|
if (max_load > tmp)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
pwr_move += busiest->cpu_power *
|
|
|
|
min(busiest_load_per_task, max_load - tmp);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Amount of load we'd add */
|
|
|
|
if (max_load*busiest->cpu_power <
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
busiest_load_per_task*SCHED_LOAD_SCALE)
|
2005-04-16 16:20:36 -06:00
|
|
|
tmp = max_load*busiest->cpu_power/this->cpu_power;
|
|
|
|
else
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
tmp = busiest_load_per_task*SCHED_LOAD_SCALE/this->cpu_power;
|
|
|
|
pwr_move += this->cpu_power*min(this_load_per_task, this_load + tmp);
|
2005-04-16 16:20:36 -06:00
|
|
|
pwr_move /= SCHED_LOAD_SCALE;
|
|
|
|
|
|
|
|
/* Move if we gain throughput */
|
|
|
|
if (pwr_move <= pwr_now)
|
|
|
|
goto out_balanced;
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
*imbalance = busiest_load_per_task;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
return busiest;
|
|
|
|
|
|
|
|
out_balanced:
|
2006-06-27 03:54:42 -06:00
|
|
|
#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
|
|
|
|
if (idle == NOT_IDLE || !(sd->flags & SD_POWERSAVINGS_BALANCE))
|
|
|
|
goto ret;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
if (this == group_leader && group_leader != group_min) {
|
|
|
|
*imbalance = min_load_per_task;
|
|
|
|
return group_min;
|
|
|
|
}
|
|
|
|
ret:
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
*imbalance = 0;
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* find_busiest_queue - find the busiest runqueue among the cpus in group.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static struct rq *
|
2006-07-03 01:25:40 -06:00
|
|
|
find_busiest_queue(struct sched_group *group, enum idle_type idle,
|
|
|
|
unsigned long imbalance)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *busiest = NULL, *rq;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
unsigned long max_load = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
int i;
|
|
|
|
|
|
|
|
for_each_cpu_mask(i, group->cpumask) {
|
2006-07-03 01:25:40 -06:00
|
|
|
rq = cpu_rq(i);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
if (rq->nr_running == 1 && rq->raw_weighted_load > imbalance)
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
continue;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
if (rq->raw_weighted_load > max_load) {
|
|
|
|
max_load = rq->raw_weighted_load;
|
|
|
|
busiest = rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return busiest;
|
|
|
|
}
|
|
|
|
|
2005-06-25 15:57:30 -06:00
|
|
|
/*
|
|
|
|
* Max backoff if we encounter pinned tasks. Pretty arbitrary value, but
|
|
|
|
* so long as it is large enough.
|
|
|
|
*/
|
|
|
|
#define MAX_PINNED_INTERVAL 512
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
static inline unsigned long minus_1_or_zero(unsigned long n)
|
|
|
|
{
|
|
|
|
return n > 0 ? n - 1 : 0;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* Check this_cpu to ensure it is balanced within domain. Attempt to move
|
|
|
|
* tasks if there is an imbalance.
|
|
|
|
*
|
|
|
|
* Called with this_rq unlocked.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static int load_balance(int this_cpu, struct rq *this_rq,
|
2005-04-16 16:20:36 -06:00
|
|
|
struct sched_domain *sd, enum idle_type idle)
|
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
int nr_moved, all_pinned = 0, active_balance = 0, sd_idle = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
struct sched_group *group;
|
|
|
|
unsigned long imbalance;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *busiest;
|
2005-09-10 01:26:19 -06:00
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
if (idle != NOT_IDLE && sd->flags & SD_SHARE_CPUPOWER &&
|
|
|
|
!sched_smt_power_savings)
|
2005-09-10 01:26:19 -06:00
|
|
|
sd_idle = 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
schedstat_inc(sd, lb_cnt[idle]);
|
|
|
|
|
2005-09-10 01:26:19 -06:00
|
|
|
group = find_busiest_group(sd, this_cpu, &imbalance, idle, &sd_idle);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!group) {
|
|
|
|
schedstat_inc(sd, lb_nobusyg[idle]);
|
|
|
|
goto out_balanced;
|
|
|
|
}
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
busiest = find_busiest_queue(group, idle, imbalance);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!busiest) {
|
|
|
|
schedstat_inc(sd, lb_nobusyq[idle]);
|
|
|
|
goto out_balanced;
|
|
|
|
}
|
|
|
|
|
2005-06-25 15:57:11 -06:00
|
|
|
BUG_ON(busiest == this_rq);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
schedstat_add(sd, lb_imbalance[idle], imbalance);
|
|
|
|
|
|
|
|
nr_moved = 0;
|
|
|
|
if (busiest->nr_running > 1) {
|
|
|
|
/*
|
|
|
|
* Attempt to move tasks. If find_busiest_group has found
|
|
|
|
* an imbalance but busiest->nr_running <= 1, the group is
|
|
|
|
* still unbalanced. nr_moved simply stays zero, so it is
|
|
|
|
* correctly treated as an imbalance.
|
|
|
|
*/
|
2005-09-10 01:26:18 -06:00
|
|
|
double_rq_lock(this_rq, busiest);
|
2005-04-16 16:20:36 -06:00
|
|
|
nr_moved = move_tasks(this_rq, this_cpu, busiest,
|
2006-07-03 01:25:40 -06:00
|
|
|
minus_1_or_zero(busiest->nr_running),
|
|
|
|
imbalance, sd, idle, &all_pinned);
|
2005-09-10 01:26:18 -06:00
|
|
|
double_rq_unlock(this_rq, busiest);
|
2005-06-25 15:57:07 -06:00
|
|
|
|
|
|
|
/* All tasks on this runqueue were pinned by CPU affinity */
|
|
|
|
if (unlikely(all_pinned))
|
|
|
|
goto out_balanced;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2005-06-25 15:57:07 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!nr_moved) {
|
|
|
|
schedstat_inc(sd, lb_failed[idle]);
|
|
|
|
sd->nr_balance_failed++;
|
|
|
|
|
|
|
|
if (unlikely(sd->nr_balance_failed > sd->cache_nice_tries+2)) {
|
|
|
|
|
|
|
|
spin_lock(&busiest->lock);
|
2005-09-10 01:26:21 -06:00
|
|
|
|
|
|
|
/* don't kick the migration_thread, if the curr
|
|
|
|
* task on busiest cpu can't be moved to this_cpu
|
|
|
|
*/
|
|
|
|
if (!cpu_isset(this_cpu, busiest->curr->cpus_allowed)) {
|
|
|
|
spin_unlock(&busiest->lock);
|
|
|
|
all_pinned = 1;
|
|
|
|
goto out_one_pinned;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!busiest->active_balance) {
|
|
|
|
busiest->active_balance = 1;
|
|
|
|
busiest->push_cpu = this_cpu;
|
2005-06-25 15:57:07 -06:00
|
|
|
active_balance = 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
spin_unlock(&busiest->lock);
|
2005-06-25 15:57:07 -06:00
|
|
|
if (active_balance)
|
2005-04-16 16:20:36 -06:00
|
|
|
wake_up_process(busiest->migration_thread);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We've kicked active balancing, reset the failure
|
|
|
|
* counter.
|
|
|
|
*/
|
2005-06-25 15:57:09 -06:00
|
|
|
sd->nr_balance_failed = sd->cache_nice_tries+1;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2005-06-25 15:57:07 -06:00
|
|
|
} else
|
2005-04-16 16:20:36 -06:00
|
|
|
sd->nr_balance_failed = 0;
|
|
|
|
|
2005-06-25 15:57:07 -06:00
|
|
|
if (likely(!active_balance)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
/* We were unbalanced, so reset the balancing interval */
|
|
|
|
sd->balance_interval = sd->min_interval;
|
2005-06-25 15:57:07 -06:00
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* If we've begun active balancing, start to back off. This
|
|
|
|
* case may not be covered by the all_pinned logic if there
|
|
|
|
* is only 1 task on the busy runqueue (because we don't call
|
|
|
|
* move_tasks).
|
|
|
|
*/
|
|
|
|
if (sd->balance_interval < sd->max_interval)
|
|
|
|
sd->balance_interval *= 2;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
if (!nr_moved && !sd_idle && sd->flags & SD_SHARE_CPUPOWER &&
|
|
|
|
!sched_smt_power_savings)
|
2005-09-10 01:26:19 -06:00
|
|
|
return -1;
|
2005-04-16 16:20:36 -06:00
|
|
|
return nr_moved;
|
|
|
|
|
|
|
|
out_balanced:
|
|
|
|
schedstat_inc(sd, lb_balanced[idle]);
|
|
|
|
|
2005-06-25 15:57:08 -06:00
|
|
|
sd->nr_balance_failed = 0;
|
2005-09-10 01:26:21 -06:00
|
|
|
|
|
|
|
out_one_pinned:
|
2005-04-16 16:20:36 -06:00
|
|
|
/* tune up the balancing interval */
|
2005-06-25 15:57:30 -06:00
|
|
|
if ((all_pinned && sd->balance_interval < MAX_PINNED_INTERVAL) ||
|
|
|
|
(sd->balance_interval < sd->max_interval))
|
2005-04-16 16:20:36 -06:00
|
|
|
sd->balance_interval *= 2;
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
if (!sd_idle && sd->flags & SD_SHARE_CPUPOWER &&
|
|
|
|
!sched_smt_power_savings)
|
2005-09-10 01:26:19 -06:00
|
|
|
return -1;
|
2005-04-16 16:20:36 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check this_cpu to ensure it is balanced within domain. Attempt to move
|
|
|
|
* tasks if there is an imbalance.
|
|
|
|
*
|
|
|
|
* Called from schedule when this_rq is about to become idle (NEWLY_IDLE).
|
|
|
|
* this_rq is locked.
|
|
|
|
*/
|
2006-07-03 01:25:40 -06:00
|
|
|
static int
|
2006-07-03 01:25:42 -06:00
|
|
|
load_balance_newidle(int this_cpu, struct rq *this_rq, struct sched_domain *sd)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct sched_group *group;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *busiest = NULL;
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long imbalance;
|
|
|
|
int nr_moved = 0;
|
2005-09-10 01:26:19 -06:00
|
|
|
int sd_idle = 0;
|
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
if (sd->flags & SD_SHARE_CPUPOWER && !sched_smt_power_savings)
|
2005-09-10 01:26:19 -06:00
|
|
|
sd_idle = 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
schedstat_inc(sd, lb_cnt[NEWLY_IDLE]);
|
2005-09-10 01:26:19 -06:00
|
|
|
group = find_busiest_group(sd, this_cpu, &imbalance, NEWLY_IDLE, &sd_idle);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!group) {
|
|
|
|
schedstat_inc(sd, lb_nobusyg[NEWLY_IDLE]);
|
2005-06-25 15:57:08 -06:00
|
|
|
goto out_balanced;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
busiest = find_busiest_queue(group, NEWLY_IDLE, imbalance);
|
2005-06-25 15:57:11 -06:00
|
|
|
if (!busiest) {
|
2005-04-16 16:20:36 -06:00
|
|
|
schedstat_inc(sd, lb_nobusyq[NEWLY_IDLE]);
|
2005-06-25 15:57:08 -06:00
|
|
|
goto out_balanced;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2005-06-25 15:57:11 -06:00
|
|
|
BUG_ON(busiest == this_rq);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
schedstat_add(sd, lb_imbalance[NEWLY_IDLE], imbalance);
|
2005-09-10 01:26:16 -06:00
|
|
|
|
|
|
|
nr_moved = 0;
|
|
|
|
if (busiest->nr_running > 1) {
|
|
|
|
/* Attempt to move tasks */
|
|
|
|
double_lock_balance(this_rq, busiest);
|
|
|
|
nr_moved = move_tasks(this_rq, this_cpu, busiest,
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
minus_1_or_zero(busiest->nr_running),
|
2005-06-25 15:57:07 -06:00
|
|
|
imbalance, sd, NEWLY_IDLE, NULL);
|
2005-09-10 01:26:16 -06:00
|
|
|
spin_unlock(&busiest->lock);
|
|
|
|
}
|
|
|
|
|
2005-09-10 01:26:19 -06:00
|
|
|
if (!nr_moved) {
|
2005-04-16 16:20:36 -06:00
|
|
|
schedstat_inc(sd, lb_failed[NEWLY_IDLE]);
|
2005-09-10 01:26:19 -06:00
|
|
|
if (!sd_idle && sd->flags & SD_SHARE_CPUPOWER)
|
|
|
|
return -1;
|
|
|
|
} else
|
2005-06-25 15:57:08 -06:00
|
|
|
sd->nr_balance_failed = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
return nr_moved;
|
2005-06-25 15:57:08 -06:00
|
|
|
|
|
|
|
out_balanced:
|
|
|
|
schedstat_inc(sd, lb_balanced[NEWLY_IDLE]);
|
2006-07-03 01:25:40 -06:00
|
|
|
if (!sd_idle && sd->flags & SD_SHARE_CPUPOWER &&
|
|
|
|
!sched_smt_power_savings)
|
2005-09-10 01:26:19 -06:00
|
|
|
return -1;
|
2005-06-25 15:57:08 -06:00
|
|
|
sd->nr_balance_failed = 0;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-06-25 15:57:08 -06:00
|
|
|
return 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* idle_balance is called by schedule() if this_cpu is about to become
|
|
|
|
* idle. Attempts to pull tasks from other CPUs.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void idle_balance(int this_cpu, struct rq *this_rq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct sched_domain *sd;
|
|
|
|
|
|
|
|
for_each_domain(this_cpu, sd) {
|
|
|
|
if (sd->flags & SD_BALANCE_NEWIDLE) {
|
2006-07-03 01:25:40 -06:00
|
|
|
/* If we've pulled tasks over stop searching: */
|
|
|
|
if (load_balance_newidle(this_cpu, this_rq, sd))
|
2005-04-16 16:20:36 -06:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* active_load_balance is run by migration threads. It pushes running tasks
|
|
|
|
* off the busiest CPU onto idle CPUs. It requires at least 1 task to be
|
|
|
|
* running on each physical CPU where possible, and avoids physical /
|
|
|
|
* logical imbalances.
|
|
|
|
*
|
|
|
|
* Called with busiest_rq locked.
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void active_load_balance(struct rq *busiest_rq, int busiest_cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-06-25 15:57:09 -06:00
|
|
|
int target_cpu = busiest_rq->push_cpu;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct sched_domain *sd;
|
|
|
|
struct rq *target_rq;
|
2005-06-25 15:57:09 -06:00
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
/* Is there any task to move? */
|
2005-06-25 15:57:09 -06:00
|
|
|
if (busiest_rq->nr_running <= 1)
|
|
|
|
return;
|
|
|
|
|
|
|
|
target_rq = cpu_rq(target_cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
2005-06-25 15:57:09 -06:00
|
|
|
* This condition is "impossible", if it occurs
|
|
|
|
* we need to fix it. Originally reported by
|
|
|
|
* Bjorn Helgaas on a 128-cpu setup.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2005-06-25 15:57:09 -06:00
|
|
|
BUG_ON(busiest_rq == target_rq);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-06-25 15:57:09 -06:00
|
|
|
/* move a task from busiest_rq to target_rq */
|
|
|
|
double_lock_balance(busiest_rq, target_rq);
|
|
|
|
|
|
|
|
/* Search for an sd spanning us and the target CPU. */
|
2006-06-27 03:54:28 -06:00
|
|
|
for_each_domain(target_cpu, sd) {
|
2005-06-25 15:57:09 -06:00
|
|
|
if ((sd->flags & SD_LOAD_BALANCE) &&
|
2006-07-03 01:25:40 -06:00
|
|
|
cpu_isset(busiest_cpu, sd->span))
|
2005-06-25 15:57:09 -06:00
|
|
|
break;
|
2006-06-27 03:54:28 -06:00
|
|
|
}
|
2005-06-25 15:57:09 -06:00
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
if (likely(sd)) {
|
|
|
|
schedstat_inc(sd, alb_cnt);
|
2005-06-25 15:57:09 -06:00
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
if (move_tasks(target_rq, target_cpu, busiest_rq, 1,
|
|
|
|
RTPRIO_TO_LOAD_WEIGHT(100), sd, SCHED_IDLE,
|
|
|
|
NULL))
|
|
|
|
schedstat_inc(sd, alb_pushed);
|
|
|
|
else
|
|
|
|
schedstat_inc(sd, alb_failed);
|
|
|
|
}
|
2005-06-25 15:57:09 -06:00
|
|
|
spin_unlock(&target_rq->lock);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* rebalance_tick will get called every timer tick, on every CPU.
|
|
|
|
*
|
|
|
|
* It checks each scheduling domain to see if it is due to be balanced,
|
|
|
|
* and initiates a balancing operation if so.
|
|
|
|
*
|
|
|
|
* Balancing parameters are set up in arch_init_sched_domains.
|
|
|
|
*/
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
/* Don't have all balancing operations going off at once: */
|
|
|
|
static inline unsigned long cpu_offset(int cpu)
|
|
|
|
{
|
|
|
|
return jiffies + cpu * HZ / NR_CPUS;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
static void
|
2006-07-03 01:25:42 -06:00
|
|
|
rebalance_tick(int this_cpu, struct rq *this_rq, enum idle_type idle)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
unsigned long this_load, interval, j = cpu_offset(this_cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
struct sched_domain *sd;
|
2006-07-03 01:25:40 -06:00
|
|
|
int i, scale;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
this_load = this_rq->raw_weighted_load;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
|
|
|
/* Update our load: */
|
|
|
|
for (i = 0, scale = 1; i < 3; i++, scale <<= 1) {
|
|
|
|
unsigned long old_load, new_load;
|
|
|
|
|
2005-06-25 15:57:13 -06:00
|
|
|
old_load = this_rq->cpu_load[i];
|
2006-07-03 01:25:40 -06:00
|
|
|
new_load = this_load;
|
2005-06-25 15:57:13 -06:00
|
|
|
/*
|
|
|
|
* Round up the averaging division if load is increasing. This
|
|
|
|
* prevents us from getting stuck on 9 if the load is 10, for
|
|
|
|
* example.
|
|
|
|
*/
|
|
|
|
if (new_load > old_load)
|
|
|
|
new_load += scale-1;
|
|
|
|
this_rq->cpu_load[i] = (old_load*(scale-1) + new_load) / scale;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
for_each_domain(this_cpu, sd) {
|
|
|
|
if (!(sd->flags & SD_LOAD_BALANCE))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
interval = sd->balance_interval;
|
|
|
|
if (idle != SCHED_IDLE)
|
|
|
|
interval *= sd->busy_factor;
|
|
|
|
|
|
|
|
/* scale ms to jiffies */
|
|
|
|
interval = msecs_to_jiffies(interval);
|
|
|
|
if (unlikely(!interval))
|
|
|
|
interval = 1;
|
|
|
|
|
|
|
|
if (j - sd->last_balance >= interval) {
|
|
|
|
if (load_balance(this_cpu, this_rq, sd, idle)) {
|
2005-09-10 01:26:21 -06:00
|
|
|
/*
|
|
|
|
* We've pulled tasks over so either we're no
|
2005-09-10 01:26:19 -06:00
|
|
|
* longer idle, or one of our SMT siblings is
|
|
|
|
* not idle.
|
|
|
|
*/
|
2005-04-16 16:20:36 -06:00
|
|
|
idle = NOT_IDLE;
|
|
|
|
}
|
|
|
|
sd->last_balance += interval;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#else
|
|
|
|
/*
|
|
|
|
* on UP we do not need to balance between CPUs:
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void rebalance_tick(int cpu, struct rq *rq, enum idle_type idle)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
}
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void idle_balance(int cpu, struct rq *rq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline int wake_priority_sleeper(struct rq *rq)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
int ret = 0;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_SCHED_SMT
|
|
|
|
spin_lock(&rq->lock);
|
|
|
|
/*
|
|
|
|
* If an SMT sibling task has been put to sleep for priority
|
|
|
|
* reasons reschedule the idle task to see if it can now run.
|
|
|
|
*/
|
|
|
|
if (rq->nr_running) {
|
|
|
|
resched_task(rq->idle);
|
|
|
|
ret = 1;
|
|
|
|
}
|
|
|
|
spin_unlock(&rq->lock);
|
|
|
|
#endif
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
DEFINE_PER_CPU(struct kernel_stat, kstat);
|
|
|
|
|
|
|
|
EXPORT_PER_CPU_SYMBOL(kstat);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This is called on clock ticks and on context switches.
|
|
|
|
* Bank in p->sched_time the ns elapsed since the last tick or switch.
|
|
|
|
*/
|
2006-07-03 01:25:40 -06:00
|
|
|
static inline void
|
2006-07-03 01:25:42 -06:00
|
|
|
update_cpu_clock(struct task_struct *p, struct rq *rq, unsigned long long now)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
p->sched_time += now - max(p->timestamp, rq->timestamp_last_tick);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Return current->sched_time plus any more ns on the sched_clock
|
|
|
|
* that have not yet been banked.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
unsigned long long current_sched_time(const struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
unsigned long long ns;
|
|
|
|
unsigned long flags;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
local_irq_save(flags);
|
2006-07-03 01:25:40 -06:00
|
|
|
ns = max(p->timestamp, task_rq(p)->timestamp_last_tick);
|
|
|
|
ns = p->sched_time + sched_clock() - ns;
|
2005-04-16 16:20:36 -06:00
|
|
|
local_irq_restore(flags);
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
return ns;
|
|
|
|
}
|
|
|
|
|
2006-05-21 19:54:09 -06:00
|
|
|
/*
|
|
|
|
* We place interactive tasks back into the active array, if possible.
|
|
|
|
*
|
|
|
|
* To guarantee that this does not starve expired tasks we ignore the
|
|
|
|
* interactivity of a task if the first expired task had to wait more
|
|
|
|
* than a 'reasonable' amount of time. This deadline timeout is
|
|
|
|
* load-dependent, as the frequency of array switched decreases with
|
|
|
|
* increasing number of running tasks. We also ignore the interactivity
|
|
|
|
* if a better static_prio task has expired:
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline int expired_starving(struct rq *rq)
|
2006-07-03 01:25:40 -06:00
|
|
|
{
|
|
|
|
if (rq->curr->static_prio > rq->best_expired_prio)
|
|
|
|
return 1;
|
|
|
|
if (!STARVATION_LIMIT || !rq->expired_timestamp)
|
|
|
|
return 0;
|
|
|
|
if (jiffies - rq->expired_timestamp > STARVATION_LIMIT * rq->nr_running)
|
|
|
|
return 1;
|
|
|
|
return 0;
|
|
|
|
}
|
2006-05-21 19:54:09 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* Account user cpu time to a process.
|
|
|
|
* @p: the process that the cpu time gets accounted to
|
|
|
|
* @hardirq_offset: the offset to subtract from hardirq_count()
|
|
|
|
* @cputime: the cpu time spent in user space since the last update
|
|
|
|
*/
|
|
|
|
void account_user_time(struct task_struct *p, cputime_t cputime)
|
|
|
|
{
|
|
|
|
struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
|
|
|
|
cputime64_t tmp;
|
|
|
|
|
|
|
|
p->utime = cputime_add(p->utime, cputime);
|
|
|
|
|
|
|
|
/* Add user time to cpustat. */
|
|
|
|
tmp = cputime_to_cputime64(cputime);
|
|
|
|
if (TASK_NICE(p) > 0)
|
|
|
|
cpustat->nice = cputime64_add(cpustat->nice, tmp);
|
|
|
|
else
|
|
|
|
cpustat->user = cputime64_add(cpustat->user, tmp);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Account system cpu time to a process.
|
|
|
|
* @p: the process that the cpu time gets accounted to
|
|
|
|
* @hardirq_offset: the offset to subtract from hardirq_count()
|
|
|
|
* @cputime: the cpu time spent in kernel space since the last update
|
|
|
|
*/
|
|
|
|
void account_system_time(struct task_struct *p, int hardirq_offset,
|
|
|
|
cputime_t cputime)
|
|
|
|
{
|
|
|
|
struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = this_rq();
|
2005-04-16 16:20:36 -06:00
|
|
|
cputime64_t tmp;
|
|
|
|
|
|
|
|
p->stime = cputime_add(p->stime, cputime);
|
|
|
|
|
|
|
|
/* Add system time to cpustat. */
|
|
|
|
tmp = cputime_to_cputime64(cputime);
|
|
|
|
if (hardirq_count() - hardirq_offset)
|
|
|
|
cpustat->irq = cputime64_add(cpustat->irq, tmp);
|
|
|
|
else if (softirq_count())
|
|
|
|
cpustat->softirq = cputime64_add(cpustat->softirq, tmp);
|
|
|
|
else if (p != rq->idle)
|
|
|
|
cpustat->system = cputime64_add(cpustat->system, tmp);
|
|
|
|
else if (atomic_read(&rq->nr_iowait) > 0)
|
|
|
|
cpustat->iowait = cputime64_add(cpustat->iowait, tmp);
|
|
|
|
else
|
|
|
|
cpustat->idle = cputime64_add(cpustat->idle, tmp);
|
|
|
|
/* Account for system time used */
|
|
|
|
acct_update_integrals(p);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Account for involuntary wait time.
|
|
|
|
* @p: the process from which the cpu time has been stolen
|
|
|
|
* @steal: the cpu time spent in involuntary wait
|
|
|
|
*/
|
|
|
|
void account_steal_time(struct task_struct *p, cputime_t steal)
|
|
|
|
{
|
|
|
|
struct cpu_usage_stat *cpustat = &kstat_this_cpu.cpustat;
|
|
|
|
cputime64_t tmp = cputime_to_cputime64(steal);
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = this_rq();
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (p == rq->idle) {
|
|
|
|
p->stime = cputime_add(p->stime, steal);
|
|
|
|
if (atomic_read(&rq->nr_iowait) > 0)
|
|
|
|
cpustat->iowait = cputime64_add(cpustat->iowait, tmp);
|
|
|
|
else
|
|
|
|
cpustat->idle = cputime64_add(cpustat->idle, tmp);
|
|
|
|
} else
|
|
|
|
cpustat->steal = cputime64_add(cpustat->steal, tmp);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function gets called by the timer code, with HZ frequency.
|
|
|
|
* We call it with interrupts disabled.
|
|
|
|
*
|
|
|
|
* It also gets called by the fork code, when changing the parent's
|
|
|
|
* timeslices.
|
|
|
|
*/
|
|
|
|
void scheduler_tick(void)
|
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
unsigned long long now = sched_clock();
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *p = current;
|
2005-04-16 16:20:36 -06:00
|
|
|
int cpu = smp_processor_id();
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
update_cpu_clock(p, rq, now);
|
|
|
|
|
|
|
|
rq->timestamp_last_tick = now;
|
|
|
|
|
|
|
|
if (p == rq->idle) {
|
|
|
|
if (wake_priority_sleeper(rq))
|
|
|
|
goto out;
|
|
|
|
rebalance_tick(cpu, rq, SCHED_IDLE);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Task might have expired already, but not scheduled off yet */
|
|
|
|
if (p->array != rq->active) {
|
|
|
|
set_tsk_need_resched(p);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
spin_lock(&rq->lock);
|
|
|
|
/*
|
|
|
|
* The task was running during this tick - update the
|
|
|
|
* time slice counter. Note: we do not update a thread's
|
|
|
|
* priority until it either goes to sleep or uses up its
|
|
|
|
* timeslice. This makes it possible for interactive tasks
|
|
|
|
* to use up their timeslices at their highest priority levels.
|
|
|
|
*/
|
|
|
|
if (rt_task(p)) {
|
|
|
|
/*
|
|
|
|
* RR tasks need a special form of timeslice management.
|
|
|
|
* FIFO tasks have no timeslices.
|
|
|
|
*/
|
|
|
|
if ((p->policy == SCHED_RR) && !--p->time_slice) {
|
|
|
|
p->time_slice = task_timeslice(p);
|
|
|
|
p->first_time_slice = 0;
|
|
|
|
set_tsk_need_resched(p);
|
|
|
|
|
|
|
|
/* put it at the end of the queue: */
|
|
|
|
requeue_task(p, rq->active);
|
|
|
|
}
|
|
|
|
goto out_unlock;
|
|
|
|
}
|
|
|
|
if (!--p->time_slice) {
|
|
|
|
dequeue_task(p, rq->active);
|
|
|
|
set_tsk_need_resched(p);
|
|
|
|
p->prio = effective_prio(p);
|
|
|
|
p->time_slice = task_timeslice(p);
|
|
|
|
p->first_time_slice = 0;
|
|
|
|
|
|
|
|
if (!rq->expired_timestamp)
|
|
|
|
rq->expired_timestamp = jiffies;
|
2006-07-03 01:25:40 -06:00
|
|
|
if (!TASK_INTERACTIVE(p) || expired_starving(rq)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
enqueue_task(p, rq->expired);
|
|
|
|
if (p->static_prio < rq->best_expired_prio)
|
|
|
|
rq->best_expired_prio = p->static_prio;
|
|
|
|
} else
|
|
|
|
enqueue_task(p, rq->active);
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Prevent a too long timeslice allowing a task to monopolize
|
|
|
|
* the CPU. We do this by splitting up the timeslice into
|
|
|
|
* smaller pieces.
|
|
|
|
*
|
|
|
|
* Note: this does not mean the task's timeslices expire or
|
|
|
|
* get lost in any way, they just might be preempted by
|
|
|
|
* another task of equal priority. (one with higher
|
|
|
|
* priority would have preempted this task already.) We
|
|
|
|
* requeue this task to the end of the list on this priority
|
|
|
|
* level, which is in essence a round-robin of tasks with
|
|
|
|
* equal priority.
|
|
|
|
*
|
|
|
|
* This only applies to tasks in the interactive
|
|
|
|
* delta range with at least TIMESLICE_GRANULARITY to requeue.
|
|
|
|
*/
|
|
|
|
if (TASK_INTERACTIVE(p) && !((task_timeslice(p) -
|
|
|
|
p->time_slice) % TIMESLICE_GRANULARITY(p)) &&
|
|
|
|
(p->time_slice >= TIMESLICE_GRANULARITY(p)) &&
|
|
|
|
(p->array == rq->active)) {
|
|
|
|
|
|
|
|
requeue_task(p, rq->active);
|
|
|
|
set_tsk_need_resched(p);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
out_unlock:
|
|
|
|
spin_unlock(&rq->lock);
|
|
|
|
out:
|
|
|
|
rebalance_tick(cpu, rq, NOT_IDLE);
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_SCHED_SMT
|
2006-07-03 01:25:42 -06:00
|
|
|
static inline void wakeup_busy_runqueue(struct rq *rq)
|
2005-09-10 01:26:08 -06:00
|
|
|
{
|
|
|
|
/* If an SMT runqueue is sleeping due to priority reasons wake it up */
|
|
|
|
if (rq->curr == rq->idle && rq->nr_running)
|
|
|
|
resched_task(rq->idle);
|
|
|
|
}
|
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
/*
|
|
|
|
* Called with interrupt disabled and this_rq's runqueue locked.
|
|
|
|
*/
|
|
|
|
static void wake_sleeping_dependent(int this_cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-06-25 15:57:24 -06:00
|
|
|
struct sched_domain *tmp, *sd = NULL;
|
2005-04-16 16:20:36 -06:00
|
|
|
int i;
|
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
for_each_domain(this_cpu, tmp) {
|
|
|
|
if (tmp->flags & SD_SHARE_CPUPOWER) {
|
2005-06-25 15:57:24 -06:00
|
|
|
sd = tmp;
|
2006-06-27 03:54:28 -06:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2005-06-25 15:57:24 -06:00
|
|
|
|
|
|
|
if (!sd)
|
2005-04-16 16:20:36 -06:00
|
|
|
return;
|
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
for_each_cpu_mask(i, sd->span) {
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *smt_rq = cpu_rq(i);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
if (i == this_cpu)
|
|
|
|
continue;
|
|
|
|
if (unlikely(!spin_trylock(&smt_rq->lock)))
|
|
|
|
continue;
|
|
|
|
|
2005-09-10 01:26:08 -06:00
|
|
|
wakeup_busy_runqueue(smt_rq);
|
2006-06-27 03:54:28 -06:00
|
|
|
spin_unlock(&smt_rq->lock);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[PATCH] sched: fix SMT scheduler latency bug
William Weston reported unusually high scheduling latencies on his x86 HT
box, on the -RT kernel. I managed to reproduce it on my HT box and the
latency tracer shows the incident in action:
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
du-2803 3Dnh2 0us : __trace_start_sched_wakeup (try_to_wake_up)
..............................................................
... we are running on CPU#3, PID 2778 gets woken to CPU#1: ...
..............................................................
du-2803 3Dnh2 0us : __trace_start_sched_wakeup <<...>-2778> (73 1)
du-2803 3Dnh2 0us : _raw_spin_unlock (try_to_wake_up)
................................................
... still on CPU#3, we send an IPI to CPU#1: ...
................................................
du-2803 3Dnh1 0us : resched_task (try_to_wake_up)
du-2803 3Dnh1 1us : smp_send_reschedule (try_to_wake_up)
du-2803 3Dnh1 1us : send_IPI_mask_bitmask (smp_send_reschedule)
du-2803 3Dnh1 2us : _raw_spin_unlock_irqrestore (try_to_wake_up)
...............................................
... 1 usec later, the IPI arrives on CPU#1: ...
...............................................
<idle>-0 1Dnh. 2us : smp_reschedule_interrupt (c0100c5a 0 0)
So far so good, this is the normal wakeup/preemption mechanism. But here
comes the scheduler anomaly on CPU#1:
<idle>-0 1Dnh. 2us : preempt_schedule_irq (need_resched)
<idle>-0 1Dnh. 2us : preempt_schedule_irq (need_resched)
<idle>-0 1Dnh. 3us : __schedule (preempt_schedule_irq)
<idle>-0 1Dnh. 3us : profile_hit (__schedule)
<idle>-0 1Dnh1 3us : sched_clock (__schedule)
<idle>-0 1Dnh1 4us : _raw_spin_lock_irq (__schedule)
<idle>-0 1Dnh1 4us : _raw_spin_lock_irqsave (__schedule)
<idle>-0 1Dnh2 5us : _raw_spin_unlock (__schedule)
<idle>-0 1Dnh1 5us : preempt_schedule (__schedule)
<idle>-0 1Dnh1 6us : _raw_spin_lock (__schedule)
<idle>-0 1Dnh2 6us : find_next_bit (__schedule)
<idle>-0 1Dnh2 6us : _raw_spin_lock (__schedule)
<idle>-0 1Dnh3 7us : find_next_bit (__schedule)
<idle>-0 1Dnh3 7us : find_next_bit (__schedule)
<idle>-0 1Dnh3 8us : _raw_spin_unlock (__schedule)
<idle>-0 1Dnh2 8us : preempt_schedule (__schedule)
<idle>-0 1Dnh2 8us : find_next_bit (__schedule)
<idle>-0 1Dnh2 9us : trace_stop_sched_switched (__schedule)
<idle>-0 1Dnh2 9us : _raw_spin_lock (trace_stop_sched_switched)
<idle>-0 1Dnh3 10us : trace_stop_sched_switched <<...>-2778> (73 8c)
<idle>-0 1Dnh3 10us : _raw_spin_unlock (trace_stop_sched_switched)
<idle>-0 1Dnh1 10us : _raw_spin_unlock (__schedule)
<idle>-0 1Dnh. 11us : local_irq_enable_noresched (preempt_schedule_irq)
<idle>-0 1Dnh. 11us < (0)
we didnt pick up pid 2778! It only gets scheduled much later:
<...>-2778 1Dnh2 412us : __switch_to (__schedule)
<...>-2778 1Dnh2 413us : __schedule <<idle>-0> (8c 73)
<...>-2778 1Dnh2 413us : _raw_spin_unlock (__schedule)
<...>-2778 1Dnh1 413us : trace_stop_sched_switched (__schedule)
<...>-2778 1Dnh1 414us : _raw_spin_lock (trace_stop_sched_switched)
<...>-2778 1Dnh2 414us : trace_stop_sched_switched <<...>-2778> (73 1)
<...>-2778 1Dnh2 414us : _raw_spin_unlock (trace_stop_sched_switched)
<...>-2778 1Dnh1 415us : trace_stop_sched_switched (__schedule)
the reason for this anomaly is the following code in dependent_sleeper():
/*
* If a user task with lower static priority than the
* running task on the SMT sibling is trying to schedule,
* delay it till there is proportionately less timeslice
* left of the sibling task to prevent a lower priority
* task from using an unfair proportion of the
* physical cpu's resources. -ck
*/
[...]
if (((smt_curr->time_slice * (100 - sd->per_cpu_gain) /
100) > task_timeslice(p)))
ret = 1;
Note that in contrast to the comment above, we dont actually do the check
based on static priority, we do the check based on timeslices. But
timeslices go up and down, and even highprio tasks can randomly have very
low timeslices (just before their next refill) and can thus be judged as
'lowprio' by the above piece of code. This condition is clearly buggy.
The correct test is to check for static_prio _and_ to check for the
preemption priority. Even on different static priority levels, a
higher-prio interactive task should not be delayed due to a
higher-static-prio CPU hog.
There is a symmetric bug in the 'kick SMT sibling' code of this function as
well, which can be solved in a similar way.
The patch below (against the current scheduler queue in -mm) fixes both
bugs. I have build and boot-tested this on x86 SMT, and nice +20 tasks
still get properly throttled - so the dependent-sleeper logic is still in
action.
btw., these bugs pessimised the SMT scheduler because the 'delay wakeup'
property was applied too liberally, so this fix is likely a throughput
improvement as well.
I separated out a smt_slice() function to make the code easier to read.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-09-10 01:26:16 -06:00
|
|
|
/*
|
|
|
|
* number of 'lost' timeslices this task wont be able to fully
|
|
|
|
* utilize, if another task runs on a sibling. This models the
|
|
|
|
* slowdown effect of other tasks running on siblings:
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline unsigned long
|
|
|
|
smt_slice(struct task_struct *p, struct sched_domain *sd)
|
[PATCH] sched: fix SMT scheduler latency bug
William Weston reported unusually high scheduling latencies on his x86 HT
box, on the -RT kernel. I managed to reproduce it on my HT box and the
latency tracer shows the incident in action:
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
du-2803 3Dnh2 0us : __trace_start_sched_wakeup (try_to_wake_up)
..............................................................
... we are running on CPU#3, PID 2778 gets woken to CPU#1: ...
..............................................................
du-2803 3Dnh2 0us : __trace_start_sched_wakeup <<...>-2778> (73 1)
du-2803 3Dnh2 0us : _raw_spin_unlock (try_to_wake_up)
................................................
... still on CPU#3, we send an IPI to CPU#1: ...
................................................
du-2803 3Dnh1 0us : resched_task (try_to_wake_up)
du-2803 3Dnh1 1us : smp_send_reschedule (try_to_wake_up)
du-2803 3Dnh1 1us : send_IPI_mask_bitmask (smp_send_reschedule)
du-2803 3Dnh1 2us : _raw_spin_unlock_irqrestore (try_to_wake_up)
...............................................
... 1 usec later, the IPI arrives on CPU#1: ...
...............................................
<idle>-0 1Dnh. 2us : smp_reschedule_interrupt (c0100c5a 0 0)
So far so good, this is the normal wakeup/preemption mechanism. But here
comes the scheduler anomaly on CPU#1:
<idle>-0 1Dnh. 2us : preempt_schedule_irq (need_resched)
<idle>-0 1Dnh. 2us : preempt_schedule_irq (need_resched)
<idle>-0 1Dnh. 3us : __schedule (preempt_schedule_irq)
<idle>-0 1Dnh. 3us : profile_hit (__schedule)
<idle>-0 1Dnh1 3us : sched_clock (__schedule)
<idle>-0 1Dnh1 4us : _raw_spin_lock_irq (__schedule)
<idle>-0 1Dnh1 4us : _raw_spin_lock_irqsave (__schedule)
<idle>-0 1Dnh2 5us : _raw_spin_unlock (__schedule)
<idle>-0 1Dnh1 5us : preempt_schedule (__schedule)
<idle>-0 1Dnh1 6us : _raw_spin_lock (__schedule)
<idle>-0 1Dnh2 6us : find_next_bit (__schedule)
<idle>-0 1Dnh2 6us : _raw_spin_lock (__schedule)
<idle>-0 1Dnh3 7us : find_next_bit (__schedule)
<idle>-0 1Dnh3 7us : find_next_bit (__schedule)
<idle>-0 1Dnh3 8us : _raw_spin_unlock (__schedule)
<idle>-0 1Dnh2 8us : preempt_schedule (__schedule)
<idle>-0 1Dnh2 8us : find_next_bit (__schedule)
<idle>-0 1Dnh2 9us : trace_stop_sched_switched (__schedule)
<idle>-0 1Dnh2 9us : _raw_spin_lock (trace_stop_sched_switched)
<idle>-0 1Dnh3 10us : trace_stop_sched_switched <<...>-2778> (73 8c)
<idle>-0 1Dnh3 10us : _raw_spin_unlock (trace_stop_sched_switched)
<idle>-0 1Dnh1 10us : _raw_spin_unlock (__schedule)
<idle>-0 1Dnh. 11us : local_irq_enable_noresched (preempt_schedule_irq)
<idle>-0 1Dnh. 11us < (0)
we didnt pick up pid 2778! It only gets scheduled much later:
<...>-2778 1Dnh2 412us : __switch_to (__schedule)
<...>-2778 1Dnh2 413us : __schedule <<idle>-0> (8c 73)
<...>-2778 1Dnh2 413us : _raw_spin_unlock (__schedule)
<...>-2778 1Dnh1 413us : trace_stop_sched_switched (__schedule)
<...>-2778 1Dnh1 414us : _raw_spin_lock (trace_stop_sched_switched)
<...>-2778 1Dnh2 414us : trace_stop_sched_switched <<...>-2778> (73 1)
<...>-2778 1Dnh2 414us : _raw_spin_unlock (trace_stop_sched_switched)
<...>-2778 1Dnh1 415us : trace_stop_sched_switched (__schedule)
the reason for this anomaly is the following code in dependent_sleeper():
/*
* If a user task with lower static priority than the
* running task on the SMT sibling is trying to schedule,
* delay it till there is proportionately less timeslice
* left of the sibling task to prevent a lower priority
* task from using an unfair proportion of the
* physical cpu's resources. -ck
*/
[...]
if (((smt_curr->time_slice * (100 - sd->per_cpu_gain) /
100) > task_timeslice(p)))
ret = 1;
Note that in contrast to the comment above, we dont actually do the check
based on static priority, we do the check based on timeslices. But
timeslices go up and down, and even highprio tasks can randomly have very
low timeslices (just before their next refill) and can thus be judged as
'lowprio' by the above piece of code. This condition is clearly buggy.
The correct test is to check for static_prio _and_ to check for the
preemption priority. Even on different static priority levels, a
higher-prio interactive task should not be delayed due to a
higher-static-prio CPU hog.
There is a symmetric bug in the 'kick SMT sibling' code of this function as
well, which can be solved in a similar way.
The patch below (against the current scheduler queue in -mm) fixes both
bugs. I have build and boot-tested this on x86 SMT, and nice +20 tasks
still get properly throttled - so the dependent-sleeper logic is still in
action.
btw., these bugs pessimised the SMT scheduler because the 'delay wakeup'
property was applied too liberally, so this fix is likely a throughput
improvement as well.
I separated out a smt_slice() function to make the code easier to read.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-09-10 01:26:16 -06:00
|
|
|
{
|
|
|
|
return p->time_slice * (100 - sd->per_cpu_gain) / 100;
|
|
|
|
}
|
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
/*
|
|
|
|
* To minimise lock contention and not have to drop this_rq's runlock we only
|
|
|
|
* trylock the sibling runqueues and bypass those runqueues if we fail to
|
|
|
|
* acquire their lock. As we only trylock the normal locking order does not
|
|
|
|
* need to be obeyed.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static int
|
2006-07-03 01:25:42 -06:00
|
|
|
dependent_sleeper(int this_cpu, struct rq *this_rq, struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-06-25 15:57:24 -06:00
|
|
|
struct sched_domain *tmp, *sd = NULL;
|
2005-04-16 16:20:36 -06:00
|
|
|
int ret = 0, i;
|
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
/* kernel/rt threads do not participate in dependent sleeping */
|
|
|
|
if (!p->mm || rt_task(p))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
for_each_domain(this_cpu, tmp) {
|
|
|
|
if (tmp->flags & SD_SHARE_CPUPOWER) {
|
2005-06-25 15:57:24 -06:00
|
|
|
sd = tmp;
|
2006-06-27 03:54:28 -06:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2005-06-25 15:57:24 -06:00
|
|
|
|
|
|
|
if (!sd)
|
2005-04-16 16:20:36 -06:00
|
|
|
return 0;
|
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
for_each_cpu_mask(i, sd->span) {
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *smt_curr;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *smt_rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
if (i == this_cpu)
|
|
|
|
continue;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
smt_rq = cpu_rq(i);
|
|
|
|
if (unlikely(!spin_trylock(&smt_rq->lock)))
|
|
|
|
continue;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
smt_curr = smt_rq->curr;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:28 -06:00
|
|
|
if (!smt_curr->mm)
|
|
|
|
goto unlock;
|
2005-09-10 01:26:08 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* If a user task with lower static priority than the
|
|
|
|
* running task on the SMT sibling is trying to schedule,
|
|
|
|
* delay it till there is proportionately less timeslice
|
|
|
|
* left of the sibling task to prevent a lower priority
|
|
|
|
* task from using an unfair proportion of the
|
|
|
|
* physical cpu's resources. -ck
|
|
|
|
*/
|
2005-09-10 01:26:08 -06:00
|
|
|
if (rt_task(smt_curr)) {
|
|
|
|
/*
|
|
|
|
* With real time tasks we run non-rt tasks only
|
|
|
|
* per_cpu_gain% of the time.
|
|
|
|
*/
|
|
|
|
if ((jiffies % DEF_TIMESLICE) >
|
|
|
|
(sd->per_cpu_gain * DEF_TIMESLICE / 100))
|
|
|
|
ret = 1;
|
2006-06-27 03:54:28 -06:00
|
|
|
} else {
|
[PATCH] sched: fix SMT scheduler latency bug
William Weston reported unusually high scheduling latencies on his x86 HT
box, on the -RT kernel. I managed to reproduce it on my HT box and the
latency tracer shows the incident in action:
_------=> CPU#
/ _-----=> irqs-off
| / _----=> need-resched
|| / _---=> hardirq/softirq
||| / _--=> preempt-depth
|||| /
||||| delay
cmd pid ||||| time | caller
\ / ||||| \ | /
du-2803 3Dnh2 0us : __trace_start_sched_wakeup (try_to_wake_up)
..............................................................
... we are running on CPU#3, PID 2778 gets woken to CPU#1: ...
..............................................................
du-2803 3Dnh2 0us : __trace_start_sched_wakeup <<...>-2778> (73 1)
du-2803 3Dnh2 0us : _raw_spin_unlock (try_to_wake_up)
................................................
... still on CPU#3, we send an IPI to CPU#1: ...
................................................
du-2803 3Dnh1 0us : resched_task (try_to_wake_up)
du-2803 3Dnh1 1us : smp_send_reschedule (try_to_wake_up)
du-2803 3Dnh1 1us : send_IPI_mask_bitmask (smp_send_reschedule)
du-2803 3Dnh1 2us : _raw_spin_unlock_irqrestore (try_to_wake_up)
...............................................
... 1 usec later, the IPI arrives on CPU#1: ...
...............................................
<idle>-0 1Dnh. 2us : smp_reschedule_interrupt (c0100c5a 0 0)
So far so good, this is the normal wakeup/preemption mechanism. But here
comes the scheduler anomaly on CPU#1:
<idle>-0 1Dnh. 2us : preempt_schedule_irq (need_resched)
<idle>-0 1Dnh. 2us : preempt_schedule_irq (need_resched)
<idle>-0 1Dnh. 3us : __schedule (preempt_schedule_irq)
<idle>-0 1Dnh. 3us : profile_hit (__schedule)
<idle>-0 1Dnh1 3us : sched_clock (__schedule)
<idle>-0 1Dnh1 4us : _raw_spin_lock_irq (__schedule)
<idle>-0 1Dnh1 4us : _raw_spin_lock_irqsave (__schedule)
<idle>-0 1Dnh2 5us : _raw_spin_unlock (__schedule)
<idle>-0 1Dnh1 5us : preempt_schedule (__schedule)
<idle>-0 1Dnh1 6us : _raw_spin_lock (__schedule)
<idle>-0 1Dnh2 6us : find_next_bit (__schedule)
<idle>-0 1Dnh2 6us : _raw_spin_lock (__schedule)
<idle>-0 1Dnh3 7us : find_next_bit (__schedule)
<idle>-0 1Dnh3 7us : find_next_bit (__schedule)
<idle>-0 1Dnh3 8us : _raw_spin_unlock (__schedule)
<idle>-0 1Dnh2 8us : preempt_schedule (__schedule)
<idle>-0 1Dnh2 8us : find_next_bit (__schedule)
<idle>-0 1Dnh2 9us : trace_stop_sched_switched (__schedule)
<idle>-0 1Dnh2 9us : _raw_spin_lock (trace_stop_sched_switched)
<idle>-0 1Dnh3 10us : trace_stop_sched_switched <<...>-2778> (73 8c)
<idle>-0 1Dnh3 10us : _raw_spin_unlock (trace_stop_sched_switched)
<idle>-0 1Dnh1 10us : _raw_spin_unlock (__schedule)
<idle>-0 1Dnh. 11us : local_irq_enable_noresched (preempt_schedule_irq)
<idle>-0 1Dnh. 11us < (0)
we didnt pick up pid 2778! It only gets scheduled much later:
<...>-2778 1Dnh2 412us : __switch_to (__schedule)
<...>-2778 1Dnh2 413us : __schedule <<idle>-0> (8c 73)
<...>-2778 1Dnh2 413us : _raw_spin_unlock (__schedule)
<...>-2778 1Dnh1 413us : trace_stop_sched_switched (__schedule)
<...>-2778 1Dnh1 414us : _raw_spin_lock (trace_stop_sched_switched)
<...>-2778 1Dnh2 414us : trace_stop_sched_switched <<...>-2778> (73 1)
<...>-2778 1Dnh2 414us : _raw_spin_unlock (trace_stop_sched_switched)
<...>-2778 1Dnh1 415us : trace_stop_sched_switched (__schedule)
the reason for this anomaly is the following code in dependent_sleeper():
/*
* If a user task with lower static priority than the
* running task on the SMT sibling is trying to schedule,
* delay it till there is proportionately less timeslice
* left of the sibling task to prevent a lower priority
* task from using an unfair proportion of the
* physical cpu's resources. -ck
*/
[...]
if (((smt_curr->time_slice * (100 - sd->per_cpu_gain) /
100) > task_timeslice(p)))
ret = 1;
Note that in contrast to the comment above, we dont actually do the check
based on static priority, we do the check based on timeslices. But
timeslices go up and down, and even highprio tasks can randomly have very
low timeslices (just before their next refill) and can thus be judged as
'lowprio' by the above piece of code. This condition is clearly buggy.
The correct test is to check for static_prio _and_ to check for the
preemption priority. Even on different static priority levels, a
higher-prio interactive task should not be delayed due to a
higher-static-prio CPU hog.
There is a symmetric bug in the 'kick SMT sibling' code of this function as
well, which can be solved in a similar way.
The patch below (against the current scheduler queue in -mm) fixes both
bugs. I have build and boot-tested this on x86 SMT, and nice +20 tasks
still get properly throttled - so the dependent-sleeper logic is still in
action.
btw., these bugs pessimised the SMT scheduler because the 'delay wakeup'
property was applied too liberally, so this fix is likely a throughput
improvement as well.
I separated out a smt_slice() function to make the code easier to read.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-09-10 01:26:16 -06:00
|
|
|
if (smt_curr->static_prio < p->static_prio &&
|
|
|
|
!TASK_PREEMPTS_CURR(p, smt_rq) &&
|
|
|
|
smt_slice(smt_curr, sd) > task_timeslice(p))
|
2005-09-10 01:26:08 -06:00
|
|
|
ret = 1;
|
|
|
|
}
|
2006-06-27 03:54:28 -06:00
|
|
|
unlock:
|
|
|
|
spin_unlock(&smt_rq->lock);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
#else
|
2006-06-27 03:54:28 -06:00
|
|
|
static inline void wake_sleeping_dependent(int this_cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
}
|
2006-07-03 01:25:40 -06:00
|
|
|
static inline int
|
2006-07-03 01:25:42 -06:00
|
|
|
dependent_sleeper(int this_cpu, struct rq *this_rq, struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#if defined(CONFIG_PREEMPT) && defined(CONFIG_DEBUG_PREEMPT)
|
|
|
|
|
|
|
|
void fastcall add_preempt_count(int val)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Underflow?
|
|
|
|
*/
|
2006-07-03 01:24:33 -06:00
|
|
|
if (DEBUG_LOCKS_WARN_ON((preempt_count() < 0)))
|
|
|
|
return;
|
2005-04-16 16:20:36 -06:00
|
|
|
preempt_count() += val;
|
|
|
|
/*
|
|
|
|
* Spinlock count overflowing soon?
|
|
|
|
*/
|
2006-07-03 01:24:33 -06:00
|
|
|
DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >= PREEMPT_MASK-10);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(add_preempt_count);
|
|
|
|
|
|
|
|
void fastcall sub_preempt_count(int val)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Underflow?
|
|
|
|
*/
|
2006-07-03 01:24:33 -06:00
|
|
|
if (DEBUG_LOCKS_WARN_ON(val > preempt_count()))
|
|
|
|
return;
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* Is the spinlock portion underflowing?
|
|
|
|
*/
|
2006-07-03 01:24:33 -06:00
|
|
|
if (DEBUG_LOCKS_WARN_ON((val < PREEMPT_MASK) &&
|
|
|
|
!(preempt_count() & PREEMPT_MASK)))
|
|
|
|
return;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
preempt_count() -= val;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(sub_preempt_count);
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
2006-03-31 03:31:23 -07:00
|
|
|
static inline int interactive_sleep(enum sleep_type sleep_type)
|
|
|
|
{
|
|
|
|
return (sleep_type == SLEEP_INTERACTIVE ||
|
|
|
|
sleep_type == SLEEP_INTERRUPTED);
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* schedule() is the main scheduler function.
|
|
|
|
*/
|
|
|
|
asmlinkage void __sched schedule(void)
|
|
|
|
{
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *prev, *next;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct prio_array *array;
|
2005-04-16 16:20:36 -06:00
|
|
|
struct list_head *queue;
|
|
|
|
unsigned long long now;
|
|
|
|
unsigned long run_time;
|
2005-06-25 15:57:31 -06:00
|
|
|
int cpu, idx, new_prio;
|
2006-07-03 01:25:40 -06:00
|
|
|
long *switch_count;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Test if we are atomic. Since do_exit() needs to call into
|
|
|
|
* schedule() atomically, we ignore that path for now.
|
|
|
|
* Otherwise, whine if we are scheduling when we should not be.
|
|
|
|
*/
|
2006-03-27 02:15:20 -07:00
|
|
|
if (unlikely(in_atomic() && !current->exit_state)) {
|
|
|
|
printk(KERN_ERR "BUG: scheduling while atomic: "
|
|
|
|
"%s/0x%08x/%d\n",
|
|
|
|
current->comm, preempt_count(), current->pid);
|
|
|
|
dump_stack();
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
profile_hit(SCHED_PROFILING, __builtin_return_address(0));
|
|
|
|
|
|
|
|
need_resched:
|
|
|
|
preempt_disable();
|
|
|
|
prev = current;
|
|
|
|
release_kernel_lock(prev);
|
|
|
|
need_resched_nonpreemptible:
|
|
|
|
rq = this_rq();
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The idle thread is not allowed to schedule!
|
|
|
|
* Remove this check after it has been exercised a bit.
|
|
|
|
*/
|
|
|
|
if (unlikely(prev == rq->idle) && prev->state != TASK_RUNNING) {
|
|
|
|
printk(KERN_ERR "bad: scheduling from the idle thread!\n");
|
|
|
|
dump_stack();
|
|
|
|
}
|
|
|
|
|
|
|
|
schedstat_inc(rq, sched_cnt);
|
|
|
|
now = sched_clock();
|
2005-04-18 11:58:36 -06:00
|
|
|
if (likely((long long)(now - prev->timestamp) < NS_MAX_SLEEP_AVG)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
run_time = now - prev->timestamp;
|
2005-04-18 11:58:36 -06:00
|
|
|
if (unlikely((long long)(now - prev->timestamp) < 0))
|
2005-04-16 16:20:36 -06:00
|
|
|
run_time = 0;
|
|
|
|
} else
|
|
|
|
run_time = NS_MAX_SLEEP_AVG;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Tasks charged proportionately less run_time at high sleep_avg to
|
|
|
|
* delay them losing their interactive status
|
|
|
|
*/
|
|
|
|
run_time /= (CURRENT_BONUS(prev) ? : 1);
|
|
|
|
|
|
|
|
spin_lock_irq(&rq->lock);
|
|
|
|
|
|
|
|
if (unlikely(prev->flags & PF_DEAD))
|
|
|
|
prev->state = EXIT_DEAD;
|
|
|
|
|
|
|
|
switch_count = &prev->nivcsw;
|
|
|
|
if (prev->state && !(preempt_count() & PREEMPT_ACTIVE)) {
|
|
|
|
switch_count = &prev->nvcsw;
|
|
|
|
if (unlikely((prev->state & TASK_INTERRUPTIBLE) &&
|
|
|
|
unlikely(signal_pending(prev))))
|
|
|
|
prev->state = TASK_RUNNING;
|
|
|
|
else {
|
|
|
|
if (prev->state == TASK_UNINTERRUPTIBLE)
|
|
|
|
rq->nr_uninterruptible++;
|
|
|
|
deactivate_task(prev, rq);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
cpu = smp_processor_id();
|
|
|
|
if (unlikely(!rq->nr_running)) {
|
|
|
|
idle_balance(cpu, rq);
|
|
|
|
if (!rq->nr_running) {
|
|
|
|
next = rq->idle;
|
|
|
|
rq->expired_timestamp = 0;
|
2006-06-27 03:54:28 -06:00
|
|
|
wake_sleeping_dependent(cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
goto switch_tasks;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
array = rq->active;
|
|
|
|
if (unlikely(!array->nr_active)) {
|
|
|
|
/*
|
|
|
|
* Switch the active and expired arrays.
|
|
|
|
*/
|
|
|
|
schedstat_inc(rq, sched_switch);
|
|
|
|
rq->active = rq->expired;
|
|
|
|
rq->expired = array;
|
|
|
|
array = rq->active;
|
|
|
|
rq->expired_timestamp = 0;
|
|
|
|
rq->best_expired_prio = MAX_PRIO;
|
|
|
|
}
|
|
|
|
|
|
|
|
idx = sched_find_first_bit(array->bitmap);
|
|
|
|
queue = array->queue + idx;
|
2006-07-03 01:25:41 -06:00
|
|
|
next = list_entry(queue->next, struct task_struct, run_list);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-03-31 03:31:23 -07:00
|
|
|
if (!rt_task(next) && interactive_sleep(next->sleep_type)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long long delta = now - next->timestamp;
|
2005-04-18 11:58:36 -06:00
|
|
|
if (unlikely((long long)(now - next->timestamp) < 0))
|
2005-04-16 16:20:36 -06:00
|
|
|
delta = 0;
|
|
|
|
|
2006-03-31 03:31:23 -07:00
|
|
|
if (next->sleep_type == SLEEP_INTERACTIVE)
|
2005-04-16 16:20:36 -06:00
|
|
|
delta = delta * (ON_RUNQUEUE_WEIGHT * 128 / 100) / 128;
|
|
|
|
|
|
|
|
array = next->array;
|
2005-06-25 15:57:31 -06:00
|
|
|
new_prio = recalc_task_prio(next, next->timestamp + delta);
|
|
|
|
|
|
|
|
if (unlikely(next->prio != new_prio)) {
|
|
|
|
dequeue_task(next, array);
|
|
|
|
next->prio = new_prio;
|
|
|
|
enqueue_task(next, array);
|
2006-03-31 03:31:29 -07:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2006-03-31 03:31:23 -07:00
|
|
|
next->sleep_type = SLEEP_NORMAL;
|
2006-06-27 03:54:28 -06:00
|
|
|
if (dependent_sleeper(cpu, rq, next))
|
|
|
|
next = rq->idle;
|
2005-04-16 16:20:36 -06:00
|
|
|
switch_tasks:
|
|
|
|
if (next == rq->idle)
|
|
|
|
schedstat_inc(rq, sched_goidle);
|
|
|
|
prefetch(next);
|
2005-09-09 14:02:02 -06:00
|
|
|
prefetch_stack(next);
|
2005-04-16 16:20:36 -06:00
|
|
|
clear_tsk_need_resched(prev);
|
|
|
|
rcu_qsctr_inc(task_cpu(prev));
|
|
|
|
|
|
|
|
update_cpu_clock(prev, rq, now);
|
|
|
|
|
|
|
|
prev->sleep_avg -= run_time;
|
|
|
|
if ((long)prev->sleep_avg <= 0)
|
|
|
|
prev->sleep_avg = 0;
|
|
|
|
prev->timestamp = prev->last_ran = now;
|
|
|
|
|
|
|
|
sched_info_switch(prev, next);
|
|
|
|
if (likely(prev != next)) {
|
|
|
|
next->timestamp = now;
|
|
|
|
rq->nr_switches++;
|
|
|
|
rq->curr = next;
|
|
|
|
++*switch_count;
|
|
|
|
|
2005-06-25 15:57:23 -06:00
|
|
|
prepare_task_switch(rq, next);
|
2005-04-16 16:20:36 -06:00
|
|
|
prev = context_switch(rq, prev, next);
|
|
|
|
barrier();
|
2005-06-25 15:57:23 -06:00
|
|
|
/*
|
|
|
|
* this_rq must be evaluated again because prev may have moved
|
|
|
|
* CPUs since it called schedule(), thus the 'rq' on its stack
|
|
|
|
* frame will be invalid.
|
|
|
|
*/
|
|
|
|
finish_task_switch(this_rq(), prev);
|
2005-04-16 16:20:36 -06:00
|
|
|
} else
|
|
|
|
spin_unlock_irq(&rq->lock);
|
|
|
|
|
|
|
|
prev = current;
|
|
|
|
if (unlikely(reacquire_kernel_lock(prev) < 0))
|
|
|
|
goto need_resched_nonpreemptible;
|
|
|
|
preempt_enable_no_resched();
|
|
|
|
if (unlikely(test_thread_flag(TIF_NEED_RESCHED)))
|
|
|
|
goto need_resched;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(schedule);
|
|
|
|
|
|
|
|
#ifdef CONFIG_PREEMPT
|
|
|
|
/*
|
2006-07-10 05:43:52 -06:00
|
|
|
* this is the entry point to schedule() from in-kernel preemption
|
2005-04-16 16:20:36 -06:00
|
|
|
* off of preempt_enable. Kernel preemptions off return from interrupt
|
|
|
|
* occur there and call schedule directly.
|
|
|
|
*/
|
|
|
|
asmlinkage void __sched preempt_schedule(void)
|
|
|
|
{
|
|
|
|
struct thread_info *ti = current_thread_info();
|
|
|
|
#ifdef CONFIG_PREEMPT_BKL
|
|
|
|
struct task_struct *task = current;
|
|
|
|
int saved_lock_depth;
|
|
|
|
#endif
|
|
|
|
/*
|
|
|
|
* If there is a non-zero preempt_count or interrupts are disabled,
|
|
|
|
* we do not want to preempt the current task. Just return..
|
|
|
|
*/
|
|
|
|
if (unlikely(ti->preempt_count || irqs_disabled()))
|
|
|
|
return;
|
|
|
|
|
|
|
|
need_resched:
|
|
|
|
add_preempt_count(PREEMPT_ACTIVE);
|
|
|
|
/*
|
|
|
|
* We keep the big kernel semaphore locked, but we
|
|
|
|
* clear ->lock_depth so that schedule() doesnt
|
|
|
|
* auto-release the semaphore:
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_PREEMPT_BKL
|
|
|
|
saved_lock_depth = task->lock_depth;
|
|
|
|
task->lock_depth = -1;
|
|
|
|
#endif
|
|
|
|
schedule();
|
|
|
|
#ifdef CONFIG_PREEMPT_BKL
|
|
|
|
task->lock_depth = saved_lock_depth;
|
|
|
|
#endif
|
|
|
|
sub_preempt_count(PREEMPT_ACTIVE);
|
|
|
|
|
|
|
|
/* we could miss a preemption opportunity between schedule and now */
|
|
|
|
barrier();
|
|
|
|
if (unlikely(test_thread_flag(TIF_NEED_RESCHED)))
|
|
|
|
goto need_resched;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(preempt_schedule);
|
|
|
|
|
|
|
|
/*
|
2006-07-10 05:43:52 -06:00
|
|
|
* this is the entry point to schedule() from kernel preemption
|
2005-04-16 16:20:36 -06:00
|
|
|
* off of irq context.
|
|
|
|
* Note, that this is called and return with irqs disabled. This will
|
|
|
|
* protect us against recursive calling from irq.
|
|
|
|
*/
|
|
|
|
asmlinkage void __sched preempt_schedule_irq(void)
|
|
|
|
{
|
|
|
|
struct thread_info *ti = current_thread_info();
|
|
|
|
#ifdef CONFIG_PREEMPT_BKL
|
|
|
|
struct task_struct *task = current;
|
|
|
|
int saved_lock_depth;
|
|
|
|
#endif
|
2006-07-10 05:43:52 -06:00
|
|
|
/* Catch callers which need to be fixed */
|
2005-04-16 16:20:36 -06:00
|
|
|
BUG_ON(ti->preempt_count || !irqs_disabled());
|
|
|
|
|
|
|
|
need_resched:
|
|
|
|
add_preempt_count(PREEMPT_ACTIVE);
|
|
|
|
/*
|
|
|
|
* We keep the big kernel semaphore locked, but we
|
|
|
|
* clear ->lock_depth so that schedule() doesnt
|
|
|
|
* auto-release the semaphore:
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_PREEMPT_BKL
|
|
|
|
saved_lock_depth = task->lock_depth;
|
|
|
|
task->lock_depth = -1;
|
|
|
|
#endif
|
|
|
|
local_irq_enable();
|
|
|
|
schedule();
|
|
|
|
local_irq_disable();
|
|
|
|
#ifdef CONFIG_PREEMPT_BKL
|
|
|
|
task->lock_depth = saved_lock_depth;
|
|
|
|
#endif
|
|
|
|
sub_preempt_count(PREEMPT_ACTIVE);
|
|
|
|
|
|
|
|
/* we could miss a preemption opportunity between schedule and now */
|
|
|
|
barrier();
|
|
|
|
if (unlikely(test_thread_flag(TIF_NEED_RESCHED)))
|
|
|
|
goto need_resched;
|
|
|
|
}
|
|
|
|
|
|
|
|
#endif /* CONFIG_PREEMPT */
|
|
|
|
|
2005-09-10 01:26:11 -06:00
|
|
|
int default_wake_function(wait_queue_t *curr, unsigned mode, int sync,
|
|
|
|
void *key)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
return try_to_wake_up(curr->private, mode, sync);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(default_wake_function);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The core wakeup function. Non-exclusive wakeups (nr_exclusive == 0) just
|
|
|
|
* wake everything up. If it's an exclusive wakeup (nr_exclusive == small +ve
|
|
|
|
* number) then we wake all the non-exclusive tasks and one exclusive task.
|
|
|
|
*
|
|
|
|
* There are circumstances in which we can try to wake a task which has already
|
|
|
|
* started to run but is not in state TASK_RUNNING. try_to_wake_up() returns
|
|
|
|
* zero in this (rare) case, and we handle it by continuing to scan the queue.
|
|
|
|
*/
|
|
|
|
static void __wake_up_common(wait_queue_head_t *q, unsigned int mode,
|
|
|
|
int nr_exclusive, int sync, void *key)
|
|
|
|
{
|
|
|
|
struct list_head *tmp, *next;
|
|
|
|
|
|
|
|
list_for_each_safe(tmp, next, &q->task_list) {
|
2006-07-03 01:25:40 -06:00
|
|
|
wait_queue_t *curr = list_entry(tmp, wait_queue_t, task_list);
|
|
|
|
unsigned flags = curr->flags;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (curr->func(curr, mode, sync, key) &&
|
2006-07-03 01:25:40 -06:00
|
|
|
(flags & WQ_FLAG_EXCLUSIVE) && !--nr_exclusive)
|
2005-04-16 16:20:36 -06:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* __wake_up - wake up threads blocked on a waitqueue.
|
|
|
|
* @q: the waitqueue
|
|
|
|
* @mode: which threads
|
|
|
|
* @nr_exclusive: how many wake-one or wake-many threads to wake up
|
2005-05-01 09:59:26 -06:00
|
|
|
* @key: is directly passed to the wakeup function
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
|
|
|
void fastcall __wake_up(wait_queue_head_t *q, unsigned int mode,
|
2005-09-10 01:26:11 -06:00
|
|
|
int nr_exclusive, void *key)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&q->lock, flags);
|
|
|
|
__wake_up_common(q, mode, nr_exclusive, 0, key);
|
|
|
|
spin_unlock_irqrestore(&q->lock, flags);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__wake_up);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Same as __wake_up but called with the spinlock in wait_queue_head_t held.
|
|
|
|
*/
|
|
|
|
void fastcall __wake_up_locked(wait_queue_head_t *q, unsigned int mode)
|
|
|
|
{
|
|
|
|
__wake_up_common(q, mode, 1, 0, NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
2005-05-01 09:59:26 -06:00
|
|
|
* __wake_up_sync - wake up threads blocked on a waitqueue.
|
2005-04-16 16:20:36 -06:00
|
|
|
* @q: the waitqueue
|
|
|
|
* @mode: which threads
|
|
|
|
* @nr_exclusive: how many wake-one or wake-many threads to wake up
|
|
|
|
*
|
|
|
|
* The sync wakeup differs that the waker knows that it will schedule
|
|
|
|
* away soon, so while the target thread will be woken up, it will not
|
|
|
|
* be migrated to another CPU - ie. the two threads are 'synchronized'
|
|
|
|
* with each other. This can prevent needless bouncing between CPUs.
|
|
|
|
*
|
|
|
|
* On UP it can prevent extra preemption.
|
|
|
|
*/
|
2005-09-10 01:26:11 -06:00
|
|
|
void fastcall
|
|
|
|
__wake_up_sync(wait_queue_head_t *q, unsigned int mode, int nr_exclusive)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
int sync = 1;
|
|
|
|
|
|
|
|
if (unlikely(!q))
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (unlikely(!nr_exclusive))
|
|
|
|
sync = 0;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&q->lock, flags);
|
|
|
|
__wake_up_common(q, mode, nr_exclusive, sync, NULL);
|
|
|
|
spin_unlock_irqrestore(&q->lock, flags);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(__wake_up_sync); /* For internal use only */
|
|
|
|
|
|
|
|
void fastcall complete(struct completion *x)
|
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&x->wait.lock, flags);
|
|
|
|
x->done++;
|
|
|
|
__wake_up_common(&x->wait, TASK_UNINTERRUPTIBLE | TASK_INTERRUPTIBLE,
|
|
|
|
1, 0, NULL);
|
|
|
|
spin_unlock_irqrestore(&x->wait.lock, flags);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(complete);
|
|
|
|
|
|
|
|
void fastcall complete_all(struct completion *x)
|
|
|
|
{
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
spin_lock_irqsave(&x->wait.lock, flags);
|
|
|
|
x->done += UINT_MAX/2;
|
|
|
|
__wake_up_common(&x->wait, TASK_UNINTERRUPTIBLE | TASK_INTERRUPTIBLE,
|
|
|
|
0, 0, NULL);
|
|
|
|
spin_unlock_irqrestore(&x->wait.lock, flags);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(complete_all);
|
|
|
|
|
|
|
|
void fastcall __sched wait_for_completion(struct completion *x)
|
|
|
|
{
|
|
|
|
might_sleep();
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
spin_lock_irq(&x->wait.lock);
|
|
|
|
if (!x->done) {
|
|
|
|
DECLARE_WAITQUEUE(wait, current);
|
|
|
|
|
|
|
|
wait.flags |= WQ_FLAG_EXCLUSIVE;
|
|
|
|
__add_wait_queue_tail(&x->wait, &wait);
|
|
|
|
do {
|
|
|
|
__set_current_state(TASK_UNINTERRUPTIBLE);
|
|
|
|
spin_unlock_irq(&x->wait.lock);
|
|
|
|
schedule();
|
|
|
|
spin_lock_irq(&x->wait.lock);
|
|
|
|
} while (!x->done);
|
|
|
|
__remove_wait_queue(&x->wait, &wait);
|
|
|
|
}
|
|
|
|
x->done--;
|
|
|
|
spin_unlock_irq(&x->wait.lock);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(wait_for_completion);
|
|
|
|
|
|
|
|
unsigned long fastcall __sched
|
|
|
|
wait_for_completion_timeout(struct completion *x, unsigned long timeout)
|
|
|
|
{
|
|
|
|
might_sleep();
|
|
|
|
|
|
|
|
spin_lock_irq(&x->wait.lock);
|
|
|
|
if (!x->done) {
|
|
|
|
DECLARE_WAITQUEUE(wait, current);
|
|
|
|
|
|
|
|
wait.flags |= WQ_FLAG_EXCLUSIVE;
|
|
|
|
__add_wait_queue_tail(&x->wait, &wait);
|
|
|
|
do {
|
|
|
|
__set_current_state(TASK_UNINTERRUPTIBLE);
|
|
|
|
spin_unlock_irq(&x->wait.lock);
|
|
|
|
timeout = schedule_timeout(timeout);
|
|
|
|
spin_lock_irq(&x->wait.lock);
|
|
|
|
if (!timeout) {
|
|
|
|
__remove_wait_queue(&x->wait, &wait);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
} while (!x->done);
|
|
|
|
__remove_wait_queue(&x->wait, &wait);
|
|
|
|
}
|
|
|
|
x->done--;
|
|
|
|
out:
|
|
|
|
spin_unlock_irq(&x->wait.lock);
|
|
|
|
return timeout;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(wait_for_completion_timeout);
|
|
|
|
|
|
|
|
int fastcall __sched wait_for_completion_interruptible(struct completion *x)
|
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
might_sleep();
|
|
|
|
|
|
|
|
spin_lock_irq(&x->wait.lock);
|
|
|
|
if (!x->done) {
|
|
|
|
DECLARE_WAITQUEUE(wait, current);
|
|
|
|
|
|
|
|
wait.flags |= WQ_FLAG_EXCLUSIVE;
|
|
|
|
__add_wait_queue_tail(&x->wait, &wait);
|
|
|
|
do {
|
|
|
|
if (signal_pending(current)) {
|
|
|
|
ret = -ERESTARTSYS;
|
|
|
|
__remove_wait_queue(&x->wait, &wait);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
__set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
spin_unlock_irq(&x->wait.lock);
|
|
|
|
schedule();
|
|
|
|
spin_lock_irq(&x->wait.lock);
|
|
|
|
} while (!x->done);
|
|
|
|
__remove_wait_queue(&x->wait, &wait);
|
|
|
|
}
|
|
|
|
x->done--;
|
|
|
|
out:
|
|
|
|
spin_unlock_irq(&x->wait.lock);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(wait_for_completion_interruptible);
|
|
|
|
|
|
|
|
unsigned long fastcall __sched
|
|
|
|
wait_for_completion_interruptible_timeout(struct completion *x,
|
|
|
|
unsigned long timeout)
|
|
|
|
{
|
|
|
|
might_sleep();
|
|
|
|
|
|
|
|
spin_lock_irq(&x->wait.lock);
|
|
|
|
if (!x->done) {
|
|
|
|
DECLARE_WAITQUEUE(wait, current);
|
|
|
|
|
|
|
|
wait.flags |= WQ_FLAG_EXCLUSIVE;
|
|
|
|
__add_wait_queue_tail(&x->wait, &wait);
|
|
|
|
do {
|
|
|
|
if (signal_pending(current)) {
|
|
|
|
timeout = -ERESTARTSYS;
|
|
|
|
__remove_wait_queue(&x->wait, &wait);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
__set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
spin_unlock_irq(&x->wait.lock);
|
|
|
|
timeout = schedule_timeout(timeout);
|
|
|
|
spin_lock_irq(&x->wait.lock);
|
|
|
|
if (!timeout) {
|
|
|
|
__remove_wait_queue(&x->wait, &wait);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
} while (!x->done);
|
|
|
|
__remove_wait_queue(&x->wait, &wait);
|
|
|
|
}
|
|
|
|
x->done--;
|
|
|
|
out:
|
|
|
|
spin_unlock_irq(&x->wait.lock);
|
|
|
|
return timeout;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(wait_for_completion_interruptible_timeout);
|
|
|
|
|
|
|
|
|
|
|
|
#define SLEEP_ON_VAR \
|
|
|
|
unsigned long flags; \
|
|
|
|
wait_queue_t wait; \
|
|
|
|
init_waitqueue_entry(&wait, current);
|
|
|
|
|
|
|
|
#define SLEEP_ON_HEAD \
|
|
|
|
spin_lock_irqsave(&q->lock,flags); \
|
|
|
|
__add_wait_queue(q, &wait); \
|
|
|
|
spin_unlock(&q->lock);
|
|
|
|
|
|
|
|
#define SLEEP_ON_TAIL \
|
|
|
|
spin_lock_irq(&q->lock); \
|
|
|
|
__remove_wait_queue(q, &wait); \
|
|
|
|
spin_unlock_irqrestore(&q->lock, flags);
|
|
|
|
|
|
|
|
void fastcall __sched interruptible_sleep_on(wait_queue_head_t *q)
|
|
|
|
{
|
|
|
|
SLEEP_ON_VAR
|
|
|
|
|
|
|
|
current->state = TASK_INTERRUPTIBLE;
|
|
|
|
|
|
|
|
SLEEP_ON_HEAD
|
|
|
|
schedule();
|
|
|
|
SLEEP_ON_TAIL
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(interruptible_sleep_on);
|
|
|
|
|
2005-09-10 01:26:11 -06:00
|
|
|
long fastcall __sched
|
|
|
|
interruptible_sleep_on_timeout(wait_queue_head_t *q, long timeout)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
SLEEP_ON_VAR
|
|
|
|
|
|
|
|
current->state = TASK_INTERRUPTIBLE;
|
|
|
|
|
|
|
|
SLEEP_ON_HEAD
|
|
|
|
timeout = schedule_timeout(timeout);
|
|
|
|
SLEEP_ON_TAIL
|
|
|
|
|
|
|
|
return timeout;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(interruptible_sleep_on_timeout);
|
|
|
|
|
|
|
|
void fastcall __sched sleep_on(wait_queue_head_t *q)
|
|
|
|
{
|
|
|
|
SLEEP_ON_VAR
|
|
|
|
|
|
|
|
current->state = TASK_UNINTERRUPTIBLE;
|
|
|
|
|
|
|
|
SLEEP_ON_HEAD
|
|
|
|
schedule();
|
|
|
|
SLEEP_ON_TAIL
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(sleep_on);
|
|
|
|
|
|
|
|
long fastcall __sched sleep_on_timeout(wait_queue_head_t *q, long timeout)
|
|
|
|
{
|
|
|
|
SLEEP_ON_VAR
|
|
|
|
|
|
|
|
current->state = TASK_UNINTERRUPTIBLE;
|
|
|
|
|
|
|
|
SLEEP_ON_HEAD
|
|
|
|
timeout = schedule_timeout(timeout);
|
|
|
|
SLEEP_ON_TAIL
|
|
|
|
|
|
|
|
return timeout;
|
|
|
|
}
|
|
|
|
|
|
|
|
EXPORT_SYMBOL(sleep_on_timeout);
|
|
|
|
|
2006-06-27 03:54:51 -06:00
|
|
|
#ifdef CONFIG_RT_MUTEXES
|
|
|
|
|
|
|
|
/*
|
|
|
|
* rt_mutex_setprio - set the current priority of a task
|
|
|
|
* @p: task
|
|
|
|
* @prio: prio value (kernel-internal form)
|
|
|
|
*
|
|
|
|
* This function changes the 'effective' priority of a task. It does
|
|
|
|
* not touch ->normal_prio like __setscheduler().
|
|
|
|
*
|
|
|
|
* Used by the rt_mutex code to implement priority inheritance logic.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
void rt_mutex_setprio(struct task_struct *p, int prio)
|
2006-06-27 03:54:51 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct prio_array *array;
|
2006-06-27 03:54:51 -06:00
|
|
|
unsigned long flags;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2006-06-27 03:54:51 -06:00
|
|
|
int oldprio;
|
|
|
|
|
|
|
|
BUG_ON(prio < 0 || prio > MAX_PRIO);
|
|
|
|
|
|
|
|
rq = task_rq_lock(p, &flags);
|
|
|
|
|
|
|
|
oldprio = p->prio;
|
|
|
|
array = p->array;
|
|
|
|
if (array)
|
|
|
|
dequeue_task(p, array);
|
|
|
|
p->prio = prio;
|
|
|
|
|
|
|
|
if (array) {
|
|
|
|
/*
|
|
|
|
* If changing to an RT priority then queue it
|
|
|
|
* in the active array!
|
|
|
|
*/
|
|
|
|
if (rt_task(p))
|
|
|
|
array = rq->active;
|
|
|
|
enqueue_task(p, array);
|
|
|
|
/*
|
|
|
|
* Reschedule if we are currently running on this runqueue and
|
|
|
|
* our priority decreased, or if we are not currently running on
|
|
|
|
* this runqueue and our priority is higher than the current's
|
|
|
|
*/
|
|
|
|
if (task_running(rq, p)) {
|
|
|
|
if (p->prio > oldprio)
|
|
|
|
resched_task(rq->curr);
|
|
|
|
} else if (TASK_PREEMPTS_CURR(p, rq))
|
|
|
|
resched_task(rq->curr);
|
|
|
|
}
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
void set_user_nice(struct task_struct *p, long nice)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct prio_array *array;
|
2006-07-03 01:25:40 -06:00
|
|
|
int old_prio, delta;
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long flags;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (TASK_NICE(p) == nice || nice < -20 || nice > 19)
|
|
|
|
return;
|
|
|
|
/*
|
|
|
|
* We have to be careful, if called from sys_setpriority(),
|
|
|
|
* the task might be in the middle of scheduling on another CPU.
|
|
|
|
*/
|
|
|
|
rq = task_rq_lock(p, &flags);
|
|
|
|
/*
|
|
|
|
* The RT priorities are set via sched_setscheduler(), but we still
|
|
|
|
* allow the 'normal' nice value to be set - but as expected
|
|
|
|
* it wont have any effect on scheduling until the task is
|
2006-01-14 14:20:41 -07:00
|
|
|
* not SCHED_NORMAL/SCHED_BATCH:
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2006-06-27 03:54:51 -06:00
|
|
|
if (has_rt_policy(p)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
p->static_prio = NICE_TO_PRIO(nice);
|
|
|
|
goto out_unlock;
|
|
|
|
}
|
|
|
|
array = p->array;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
if (array) {
|
2005-04-16 16:20:36 -06:00
|
|
|
dequeue_task(p, array);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
dec_raw_weighted_load(rq, p);
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
p->static_prio = NICE_TO_PRIO(nice);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
set_load_weight(p);
|
2006-06-27 03:54:51 -06:00
|
|
|
old_prio = p->prio;
|
|
|
|
p->prio = effective_prio(p);
|
|
|
|
delta = p->prio - old_prio;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (array) {
|
|
|
|
enqueue_task(p, array);
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
inc_raw_weighted_load(rq, p);
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* If the task increased its priority or is running and
|
|
|
|
* lowered its priority, then reschedule its CPU:
|
|
|
|
*/
|
|
|
|
if (delta < 0 || (delta > 0 && task_running(rq, p)))
|
|
|
|
resched_task(rq->curr);
|
|
|
|
}
|
|
|
|
out_unlock:
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(set_user_nice);
|
|
|
|
|
2005-05-01 09:59:00 -06:00
|
|
|
/*
|
|
|
|
* can_nice - check if a task can reduce its nice value
|
|
|
|
* @p: task
|
|
|
|
* @nice: nice value
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
int can_nice(const struct task_struct *p, const int nice)
|
2005-05-01 09:59:00 -06:00
|
|
|
{
|
2005-08-18 12:24:19 -06:00
|
|
|
/* convert nice value [19,-20] to rlimit style value [1,40] */
|
|
|
|
int nice_rlim = 20 - nice;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-05-01 09:59:00 -06:00
|
|
|
return (nice_rlim <= p->signal->rlim[RLIMIT_NICE].rlim_cur ||
|
|
|
|
capable(CAP_SYS_NICE));
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef __ARCH_WANT_SYS_NICE
|
|
|
|
|
|
|
|
/*
|
|
|
|
* sys_nice - change the priority of the current process.
|
|
|
|
* @increment: priority increment
|
|
|
|
*
|
|
|
|
* sys_setpriority is a more generic, but much slower function that
|
|
|
|
* does similar things.
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_nice(int increment)
|
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
long nice, retval;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Setpriority might change our priority at the same moment.
|
|
|
|
* We don't have to worry. Conceptually one call occurs first
|
|
|
|
* and we have a single winner.
|
|
|
|
*/
|
2005-05-01 09:59:00 -06:00
|
|
|
if (increment < -40)
|
|
|
|
increment = -40;
|
2005-04-16 16:20:36 -06:00
|
|
|
if (increment > 40)
|
|
|
|
increment = 40;
|
|
|
|
|
|
|
|
nice = PRIO_TO_NICE(current->static_prio) + increment;
|
|
|
|
if (nice < -20)
|
|
|
|
nice = -20;
|
|
|
|
if (nice > 19)
|
|
|
|
nice = 19;
|
|
|
|
|
2005-05-01 09:59:00 -06:00
|
|
|
if (increment < 0 && !can_nice(current, nice))
|
|
|
|
return -EPERM;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
retval = security_task_setnice(current, nice);
|
|
|
|
if (retval)
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
set_user_nice(current, nice);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
/**
|
|
|
|
* task_prio - return the priority value of a given task.
|
|
|
|
* @p: the task in question.
|
|
|
|
*
|
|
|
|
* This is the priority value as seen by users in /proc.
|
|
|
|
* RT tasks are offset by -200. Normal tasks are centered
|
|
|
|
* around 0, value goes from -16 to +15.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
int task_prio(const struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return p->prio - MAX_RT_PRIO;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* task_nice - return the nice value of a given task.
|
|
|
|
* @p: the task in question.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
int task_nice(const struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return TASK_NICE(p);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(task_nice);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* idle_cpu - is a given cpu idle currently?
|
|
|
|
* @cpu: the processor in question.
|
|
|
|
*/
|
|
|
|
int idle_cpu(int cpu)
|
|
|
|
{
|
|
|
|
return cpu_curr(cpu) == cpu_rq(cpu)->idle;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* idle_task - return the idle task for a given cpu.
|
|
|
|
* @cpu: the processor in question.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *idle_task(int cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return cpu_rq(cpu)->idle;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* find_process_by_pid - find a process with a matching PID value.
|
|
|
|
* @pid: the pid in question.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
static inline struct task_struct *find_process_by_pid(pid_t pid)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return pid ? find_task_by_pid(pid) : current;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Actually do priority change: must hold rq lock. */
|
|
|
|
static void __setscheduler(struct task_struct *p, int policy, int prio)
|
|
|
|
{
|
|
|
|
BUG_ON(p->array);
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
p->policy = policy;
|
|
|
|
p->rt_priority = prio;
|
2006-06-27 03:54:51 -06:00
|
|
|
p->normal_prio = normal_prio(p);
|
|
|
|
/* we are holding p->pi_lock already */
|
|
|
|
p->prio = rt_mutex_getprio(p);
|
|
|
|
/*
|
|
|
|
* SCHED_BATCH tasks are treated as perpetual CPU hogs:
|
|
|
|
*/
|
|
|
|
if (policy == SCHED_BATCH)
|
|
|
|
p->sleep_avg = 0;
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
set_load_weight(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sched_setscheduler - change the scheduling policy and/or RT priority of
|
|
|
|
* a thread.
|
|
|
|
* @p: the task in question.
|
|
|
|
* @policy: new policy.
|
|
|
|
* @param: structure containing the new RT priority.
|
|
|
|
*/
|
2005-09-10 01:26:11 -06:00
|
|
|
int sched_setscheduler(struct task_struct *p, int policy,
|
|
|
|
struct sched_param *param)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
int retval, oldprio, oldpolicy = -1;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct prio_array *array;
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long flags;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:44 -06:00
|
|
|
/* may grab non-irq protected spin_locks */
|
|
|
|
BUG_ON(in_interrupt());
|
2005-04-16 16:20:36 -06:00
|
|
|
recheck:
|
|
|
|
/* double check policy once rq lock held */
|
|
|
|
if (policy < 0)
|
|
|
|
policy = oldpolicy = p->policy;
|
|
|
|
else if (policy != SCHED_FIFO && policy != SCHED_RR &&
|
2006-01-14 14:20:41 -07:00
|
|
|
policy != SCHED_NORMAL && policy != SCHED_BATCH)
|
|
|
|
return -EINVAL;
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* Valid priorities for SCHED_FIFO and SCHED_RR are
|
2006-01-14 14:20:41 -07:00
|
|
|
* 1..MAX_USER_RT_PRIO-1, valid priority for SCHED_NORMAL and
|
|
|
|
* SCHED_BATCH is 0.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
|
|
|
if (param->sched_priority < 0 ||
|
2005-09-10 01:26:11 -06:00
|
|
|
(p->mm && param->sched_priority > MAX_USER_RT_PRIO-1) ||
|
2005-07-25 14:28:39 -06:00
|
|
|
(!p->mm && param->sched_priority > MAX_RT_PRIO-1))
|
2005-04-16 16:20:36 -06:00
|
|
|
return -EINVAL;
|
2006-01-14 14:20:41 -07:00
|
|
|
if ((policy == SCHED_NORMAL || policy == SCHED_BATCH)
|
|
|
|
!= (param->sched_priority == 0))
|
2005-04-16 16:20:36 -06:00
|
|
|
return -EINVAL;
|
|
|
|
|
[PATCH] Changing RT priority without CAP_SYS_NICE
Presently, a process without the capability CAP_SYS_NICE can not change
its own policy, which is OK.
But it can also not decrease its RT priority (if scheduled with policy
SCHED_RR or SCHED_FIFO), which is what this patch changes.
The rationale is the same as for the nice value: a process should be
able to require less priority for itself. Increasing the priority is
still not allowed.
This is for example useful if you give a multithreaded user process a RT
priority, and the process would like to organize its internal threads
using priorities also. Then you can give the process the highest
priority needed N, and the process starts its threads with lower
priorities: N-1, N-2...
The POSIX norm says that the permissions are implementation specific, so
I think we can do that.
In a sense, it makes the permissions consistent whatever the policy is:
with this patch, process scheduled by SCHED_FIFO, SCHED_RR and
SCHED_OTHER can all decrease their priority.
From: Ingo Molnar <mingo@elte.hu>
cleaned up and merged to -mm.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-25 15:57:32 -06:00
|
|
|
/*
|
|
|
|
* Allow unprivileged RT tasks to decrease priority:
|
|
|
|
*/
|
|
|
|
if (!capable(CAP_SYS_NICE)) {
|
2006-01-14 14:20:41 -07:00
|
|
|
/*
|
|
|
|
* can't change policy, except between SCHED_NORMAL
|
|
|
|
* and SCHED_BATCH:
|
|
|
|
*/
|
|
|
|
if (((policy != SCHED_NORMAL && p->policy != SCHED_BATCH) &&
|
|
|
|
(policy != SCHED_BATCH && p->policy != SCHED_NORMAL)) &&
|
|
|
|
!p->signal->rlim[RLIMIT_RTPRIO].rlim_cur)
|
[PATCH] Changing RT priority without CAP_SYS_NICE
Presently, a process without the capability CAP_SYS_NICE can not change
its own policy, which is OK.
But it can also not decrease its RT priority (if scheduled with policy
SCHED_RR or SCHED_FIFO), which is what this patch changes.
The rationale is the same as for the nice value: a process should be
able to require less priority for itself. Increasing the priority is
still not allowed.
This is for example useful if you give a multithreaded user process a RT
priority, and the process would like to organize its internal threads
using priorities also. Then you can give the process the highest
priority needed N, and the process starts its threads with lower
priorities: N-1, N-2...
The POSIX norm says that the permissions are implementation specific, so
I think we can do that.
In a sense, it makes the permissions consistent whatever the policy is:
with this patch, process scheduled by SCHED_FIFO, SCHED_RR and
SCHED_OTHER can all decrease their priority.
From: Ingo Molnar <mingo@elte.hu>
cleaned up and merged to -mm.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-25 15:57:32 -06:00
|
|
|
return -EPERM;
|
|
|
|
/* can't increase priority */
|
2006-01-14 14:20:41 -07:00
|
|
|
if ((policy != SCHED_NORMAL && policy != SCHED_BATCH) &&
|
[PATCH] Changing RT priority without CAP_SYS_NICE
Presently, a process without the capability CAP_SYS_NICE can not change
its own policy, which is OK.
But it can also not decrease its RT priority (if scheduled with policy
SCHED_RR or SCHED_FIFO), which is what this patch changes.
The rationale is the same as for the nice value: a process should be
able to require less priority for itself. Increasing the priority is
still not allowed.
This is for example useful if you give a multithreaded user process a RT
priority, and the process would like to organize its internal threads
using priorities also. Then you can give the process the highest
priority needed N, and the process starts its threads with lower
priorities: N-1, N-2...
The POSIX norm says that the permissions are implementation specific, so
I think we can do that.
In a sense, it makes the permissions consistent whatever the policy is:
with this patch, process scheduled by SCHED_FIFO, SCHED_RR and
SCHED_OTHER can all decrease their priority.
From: Ingo Molnar <mingo@elte.hu>
cleaned up and merged to -mm.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2005-06-25 15:57:32 -06:00
|
|
|
param->sched_priority > p->rt_priority &&
|
|
|
|
param->sched_priority >
|
|
|
|
p->signal->rlim[RLIMIT_RTPRIO].rlim_cur)
|
|
|
|
return -EPERM;
|
|
|
|
/* can't change other user's priorities */
|
|
|
|
if ((current->euid != p->euid) &&
|
|
|
|
(current->euid != p->uid))
|
|
|
|
return -EPERM;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
retval = security_task_setscheduler(p, policy, param);
|
|
|
|
if (retval)
|
|
|
|
return retval;
|
2006-06-27 03:54:51 -06:00
|
|
|
/*
|
|
|
|
* make sure no PI-waiters arrive (or leave) while we are
|
|
|
|
* changing the priority of the task:
|
|
|
|
*/
|
|
|
|
spin_lock_irqsave(&p->pi_lock, flags);
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* To be able to change p->policy safely, the apropriate
|
|
|
|
* runqueue lock must be held.
|
|
|
|
*/
|
2006-06-27 03:54:51 -06:00
|
|
|
rq = __task_rq_lock(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
/* recheck policy now with rq lock held */
|
|
|
|
if (unlikely(oldpolicy != -1 && oldpolicy != p->policy)) {
|
|
|
|
policy = oldpolicy = -1;
|
2006-06-27 03:54:51 -06:00
|
|
|
__task_rq_unlock(rq);
|
|
|
|
spin_unlock_irqrestore(&p->pi_lock, flags);
|
2005-04-16 16:20:36 -06:00
|
|
|
goto recheck;
|
|
|
|
}
|
|
|
|
array = p->array;
|
|
|
|
if (array)
|
|
|
|
deactivate_task(p, rq);
|
|
|
|
oldprio = p->prio;
|
|
|
|
__setscheduler(p, policy, param->sched_priority);
|
|
|
|
if (array) {
|
|
|
|
__activate_task(p, rq);
|
|
|
|
/*
|
|
|
|
* Reschedule if we are currently running on this runqueue and
|
|
|
|
* our priority decreased, or if we are not currently running on
|
|
|
|
* this runqueue and our priority is higher than the current's
|
|
|
|
*/
|
|
|
|
if (task_running(rq, p)) {
|
|
|
|
if (p->prio > oldprio)
|
|
|
|
resched_task(rq->curr);
|
|
|
|
} else if (TASK_PREEMPTS_CURR(p, rq))
|
|
|
|
resched_task(rq->curr);
|
|
|
|
}
|
2006-06-27 03:54:51 -06:00
|
|
|
__task_rq_unlock(rq);
|
|
|
|
spin_unlock_irqrestore(&p->pi_lock, flags);
|
|
|
|
|
2006-06-27 03:55:02 -06:00
|
|
|
rt_mutex_adjust_pi(p);
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(sched_setscheduler);
|
|
|
|
|
2005-09-10 01:26:11 -06:00
|
|
|
static int
|
|
|
|
do_sched_setscheduler(pid_t pid, int policy, struct sched_param __user *param)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct sched_param lparam;
|
|
|
|
struct task_struct *p;
|
2006-07-03 01:25:41 -06:00
|
|
|
int retval;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (!param || pid < 0)
|
|
|
|
return -EINVAL;
|
|
|
|
if (copy_from_user(&lparam, param, sizeof(struct sched_param)))
|
|
|
|
return -EFAULT;
|
|
|
|
read_lock_irq(&tasklist_lock);
|
|
|
|
p = find_process_by_pid(pid);
|
|
|
|
if (!p) {
|
|
|
|
read_unlock_irq(&tasklist_lock);
|
|
|
|
return -ESRCH;
|
|
|
|
}
|
2006-06-27 03:55:00 -06:00
|
|
|
get_task_struct(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
read_unlock_irq(&tasklist_lock);
|
2006-06-27 03:55:00 -06:00
|
|
|
retval = sched_setscheduler(p, policy, &lparam);
|
|
|
|
put_task_struct(p);
|
2006-07-03 01:25:41 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_setscheduler - set/change the scheduler policy and RT priority
|
|
|
|
* @pid: the pid in question.
|
|
|
|
* @policy: new policy.
|
|
|
|
* @param: structure containing the new RT priority.
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_sched_setscheduler(pid_t pid, int policy,
|
|
|
|
struct sched_param __user *param)
|
|
|
|
{
|
2006-01-18 18:43:03 -07:00
|
|
|
/* negative values for policy are not valid */
|
|
|
|
if (policy < 0)
|
|
|
|
return -EINVAL;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
return do_sched_setscheduler(pid, policy, param);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_setparam - set/change the RT priority of a thread
|
|
|
|
* @pid: the pid in question.
|
|
|
|
* @param: structure containing the new RT priority.
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_sched_setparam(pid_t pid, struct sched_param __user *param)
|
|
|
|
{
|
|
|
|
return do_sched_setscheduler(pid, -1, param);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_getscheduler - get the policy (scheduling class) of a thread
|
|
|
|
* @pid: the pid in question.
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_sched_getscheduler(pid_t pid)
|
|
|
|
{
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *p;
|
2005-04-16 16:20:36 -06:00
|
|
|
int retval = -EINVAL;
|
|
|
|
|
|
|
|
if (pid < 0)
|
|
|
|
goto out_nounlock;
|
|
|
|
|
|
|
|
retval = -ESRCH;
|
|
|
|
read_lock(&tasklist_lock);
|
|
|
|
p = find_process_by_pid(pid);
|
|
|
|
if (p) {
|
|
|
|
retval = security_task_getscheduler(p);
|
|
|
|
if (!retval)
|
|
|
|
retval = p->policy;
|
|
|
|
}
|
|
|
|
read_unlock(&tasklist_lock);
|
|
|
|
|
|
|
|
out_nounlock:
|
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_getscheduler - get the RT priority of a thread
|
|
|
|
* @pid: the pid in question.
|
|
|
|
* @param: structure containing the RT priority.
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_sched_getparam(pid_t pid, struct sched_param __user *param)
|
|
|
|
{
|
|
|
|
struct sched_param lp;
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *p;
|
2005-04-16 16:20:36 -06:00
|
|
|
int retval = -EINVAL;
|
|
|
|
|
|
|
|
if (!param || pid < 0)
|
|
|
|
goto out_nounlock;
|
|
|
|
|
|
|
|
read_lock(&tasklist_lock);
|
|
|
|
p = find_process_by_pid(pid);
|
|
|
|
retval = -ESRCH;
|
|
|
|
if (!p)
|
|
|
|
goto out_unlock;
|
|
|
|
|
|
|
|
retval = security_task_getscheduler(p);
|
|
|
|
if (retval)
|
|
|
|
goto out_unlock;
|
|
|
|
|
|
|
|
lp.sched_priority = p->rt_priority;
|
|
|
|
read_unlock(&tasklist_lock);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This one might sleep, we cannot do it with a spinlock held ...
|
|
|
|
*/
|
|
|
|
retval = copy_to_user(param, &lp, sizeof(*param)) ? -EFAULT : 0;
|
|
|
|
|
|
|
|
out_nounlock:
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
out_unlock:
|
|
|
|
read_unlock(&tasklist_lock);
|
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
|
|
|
long sched_setaffinity(pid_t pid, cpumask_t new_mask)
|
|
|
|
{
|
|
|
|
cpumask_t cpus_allowed;
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *p;
|
|
|
|
int retval;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
lock_cpu_hotplug();
|
|
|
|
read_lock(&tasklist_lock);
|
|
|
|
|
|
|
|
p = find_process_by_pid(pid);
|
|
|
|
if (!p) {
|
|
|
|
read_unlock(&tasklist_lock);
|
|
|
|
unlock_cpu_hotplug();
|
|
|
|
return -ESRCH;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* It is not safe to call set_cpus_allowed with the
|
|
|
|
* tasklist_lock held. We will bump the task_struct's
|
|
|
|
* usage count and then drop tasklist_lock.
|
|
|
|
*/
|
|
|
|
get_task_struct(p);
|
|
|
|
read_unlock(&tasklist_lock);
|
|
|
|
|
|
|
|
retval = -EPERM;
|
|
|
|
if ((current->euid != p->euid) && (current->euid != p->uid) &&
|
|
|
|
!capable(CAP_SYS_NICE))
|
|
|
|
goto out_unlock;
|
|
|
|
|
2006-06-23 03:03:59 -06:00
|
|
|
retval = security_task_setscheduler(p, 0, NULL);
|
|
|
|
if (retval)
|
|
|
|
goto out_unlock;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
cpus_allowed = cpuset_cpus_allowed(p);
|
|
|
|
cpus_and(new_mask, new_mask, cpus_allowed);
|
|
|
|
retval = set_cpus_allowed(p, new_mask);
|
|
|
|
|
|
|
|
out_unlock:
|
|
|
|
put_task_struct(p);
|
|
|
|
unlock_cpu_hotplug();
|
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int get_user_cpu_mask(unsigned long __user *user_mask_ptr, unsigned len,
|
|
|
|
cpumask_t *new_mask)
|
|
|
|
{
|
|
|
|
if (len < sizeof(cpumask_t)) {
|
|
|
|
memset(new_mask, 0, sizeof(cpumask_t));
|
|
|
|
} else if (len > sizeof(cpumask_t)) {
|
|
|
|
len = sizeof(cpumask_t);
|
|
|
|
}
|
|
|
|
return copy_from_user(new_mask, user_mask_ptr, len) ? -EFAULT : 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_setaffinity - set the cpu affinity of a process
|
|
|
|
* @pid: pid of the process
|
|
|
|
* @len: length in bytes of the bitmask pointed to by user_mask_ptr
|
|
|
|
* @user_mask_ptr: user-space pointer to the new cpu mask
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_sched_setaffinity(pid_t pid, unsigned int len,
|
|
|
|
unsigned long __user *user_mask_ptr)
|
|
|
|
{
|
|
|
|
cpumask_t new_mask;
|
|
|
|
int retval;
|
|
|
|
|
|
|
|
retval = get_user_cpu_mask(user_mask_ptr, len, &new_mask);
|
|
|
|
if (retval)
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
return sched_setaffinity(pid, new_mask);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Represents all cpu's present in the system
|
|
|
|
* In systems capable of hotplug, this map could dynamically grow
|
|
|
|
* as new cpu's are detected in the system via any platform specific
|
|
|
|
* method, such as ACPI for e.g.
|
|
|
|
*/
|
|
|
|
|
2006-01-11 14:44:57 -07:00
|
|
|
cpumask_t cpu_present_map __read_mostly;
|
2005-04-16 16:20:36 -06:00
|
|
|
EXPORT_SYMBOL(cpu_present_map);
|
|
|
|
|
|
|
|
#ifndef CONFIG_SMP
|
2006-01-11 14:44:57 -07:00
|
|
|
cpumask_t cpu_online_map __read_mostly = CPU_MASK_ALL;
|
|
|
|
cpumask_t cpu_possible_map __read_mostly = CPU_MASK_ALL;
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
|
|
|
|
|
|
|
long sched_getaffinity(pid_t pid, cpumask_t *mask)
|
|
|
|
{
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *p;
|
2005-04-16 16:20:36 -06:00
|
|
|
int retval;
|
|
|
|
|
|
|
|
lock_cpu_hotplug();
|
|
|
|
read_lock(&tasklist_lock);
|
|
|
|
|
|
|
|
retval = -ESRCH;
|
|
|
|
p = find_process_by_pid(pid);
|
|
|
|
if (!p)
|
|
|
|
goto out_unlock;
|
|
|
|
|
2006-06-23 03:03:59 -06:00
|
|
|
retval = security_task_getscheduler(p);
|
|
|
|
if (retval)
|
|
|
|
goto out_unlock;
|
|
|
|
|
2006-02-01 04:05:18 -07:00
|
|
|
cpus_and(*mask, p->cpus_allowed, cpu_online_map);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
out_unlock:
|
|
|
|
read_unlock(&tasklist_lock);
|
|
|
|
unlock_cpu_hotplug();
|
|
|
|
if (retval)
|
|
|
|
return retval;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_getaffinity - get the cpu affinity of a process
|
|
|
|
* @pid: pid of the process
|
|
|
|
* @len: length in bytes of the bitmask pointed to by user_mask_ptr
|
|
|
|
* @user_mask_ptr: user-space pointer to hold the current cpu mask
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_sched_getaffinity(pid_t pid, unsigned int len,
|
|
|
|
unsigned long __user *user_mask_ptr)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
cpumask_t mask;
|
|
|
|
|
|
|
|
if (len < sizeof(cpumask_t))
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
ret = sched_getaffinity(pid, &mask);
|
|
|
|
if (ret < 0)
|
|
|
|
return ret;
|
|
|
|
|
|
|
|
if (copy_to_user(user_mask_ptr, &mask, sizeof(cpumask_t)))
|
|
|
|
return -EFAULT;
|
|
|
|
|
|
|
|
return sizeof(cpumask_t);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_yield - yield the current processor to other threads.
|
|
|
|
*
|
|
|
|
* this function yields the current CPU by moving the calling thread
|
|
|
|
* to the expired array. If there are no other threads running on this
|
|
|
|
* CPU then this function will return.
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_sched_yield(void)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = this_rq_lock();
|
|
|
|
struct prio_array *array = current->array, *target = rq->expired;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
schedstat_inc(rq, yld_cnt);
|
|
|
|
/*
|
|
|
|
* We implement yielding by moving the task into the expired
|
|
|
|
* queue.
|
|
|
|
*
|
|
|
|
* (special rule: RT tasks will just roundrobin in the active
|
|
|
|
* array.)
|
|
|
|
*/
|
|
|
|
if (rt_task(current))
|
|
|
|
target = rq->active;
|
|
|
|
|
2005-09-10 01:26:20 -06:00
|
|
|
if (array->nr_active == 1) {
|
2005-04-16 16:20:36 -06:00
|
|
|
schedstat_inc(rq, yld_act_empty);
|
|
|
|
if (!rq->expired->nr_active)
|
|
|
|
schedstat_inc(rq, yld_both_empty);
|
|
|
|
} else if (!rq->expired->nr_active)
|
|
|
|
schedstat_inc(rq, yld_exp_empty);
|
|
|
|
|
|
|
|
if (array != target) {
|
|
|
|
dequeue_task(current, array);
|
|
|
|
enqueue_task(current, target);
|
|
|
|
} else
|
|
|
|
/*
|
|
|
|
* requeue_task is cheaper so perform that if possible.
|
|
|
|
*/
|
|
|
|
requeue_task(current, array);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Since we are going to call schedule() anyway, there's
|
|
|
|
* no need to preempt or enable interrupts:
|
|
|
|
*/
|
|
|
|
__release(rq->lock);
|
2006-07-03 01:24:54 -06:00
|
|
|
spin_release(&rq->lock.dep_map, 1, _THIS_IP_);
|
2005-04-16 16:20:36 -06:00
|
|
|
_raw_spin_unlock(&rq->lock);
|
|
|
|
preempt_enable_no_resched();
|
|
|
|
|
|
|
|
schedule();
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-07-30 04:03:39 -06:00
|
|
|
static inline int __resched_legal(int expected_preempt_count)
|
2006-06-30 02:56:00 -06:00
|
|
|
{
|
2006-07-30 04:03:39 -06:00
|
|
|
if (unlikely(preempt_count() != expected_preempt_count))
|
2006-06-30 02:56:00 -06:00
|
|
|
return 0;
|
|
|
|
if (unlikely(system_state != SYSTEM_RUNNING))
|
|
|
|
return 0;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void __cond_resched(void)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-06-23 03:05:23 -06:00
|
|
|
#ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
|
|
|
|
__might_sleep(__FILE__, __LINE__);
|
|
|
|
#endif
|
2005-07-07 18:57:04 -06:00
|
|
|
/*
|
|
|
|
* The BKS might be reacquired before we have dropped
|
|
|
|
* PREEMPT_ACTIVE, which could trigger a second
|
|
|
|
* cond_resched() call.
|
|
|
|
*/
|
2005-04-16 16:20:36 -06:00
|
|
|
do {
|
|
|
|
add_preempt_count(PREEMPT_ACTIVE);
|
|
|
|
schedule();
|
|
|
|
sub_preempt_count(PREEMPT_ACTIVE);
|
|
|
|
} while (need_resched());
|
|
|
|
}
|
|
|
|
|
|
|
|
int __sched cond_resched(void)
|
|
|
|
{
|
2006-07-30 04:03:39 -06:00
|
|
|
if (need_resched() && __resched_legal(0)) {
|
2005-04-16 16:20:36 -06:00
|
|
|
__cond_resched();
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(cond_resched);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* cond_resched_lock() - if a reschedule is pending, drop the given lock,
|
|
|
|
* call schedule, and on return reacquire the lock.
|
|
|
|
*
|
|
|
|
* This works OK both with and without CONFIG_PREEMPT. We do strange low-level
|
|
|
|
* operations here to prevent schedule() from being called twice (once via
|
|
|
|
* spin_unlock(), once by hand).
|
|
|
|
*/
|
2005-09-10 01:26:11 -06:00
|
|
|
int cond_resched_lock(spinlock_t *lock)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2005-06-13 16:52:32 -06:00
|
|
|
int ret = 0;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
if (need_lockbreak(lock)) {
|
|
|
|
spin_unlock(lock);
|
|
|
|
cpu_relax();
|
2005-06-13 16:52:32 -06:00
|
|
|
ret = 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
spin_lock(lock);
|
|
|
|
}
|
2006-07-30 04:03:39 -06:00
|
|
|
if (need_resched() && __resched_legal(1)) {
|
2006-07-03 01:24:54 -06:00
|
|
|
spin_release(&lock->dep_map, 1, _THIS_IP_);
|
2005-04-16 16:20:36 -06:00
|
|
|
_raw_spin_unlock(lock);
|
|
|
|
preempt_enable_no_resched();
|
|
|
|
__cond_resched();
|
2005-06-13 16:52:32 -06:00
|
|
|
ret = 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
spin_lock(lock);
|
|
|
|
}
|
2005-06-13 16:52:32 -06:00
|
|
|
return ret;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(cond_resched_lock);
|
|
|
|
|
|
|
|
int __sched cond_resched_softirq(void)
|
|
|
|
{
|
|
|
|
BUG_ON(!in_softirq());
|
|
|
|
|
2006-07-30 04:03:39 -06:00
|
|
|
if (need_resched() && __resched_legal(0)) {
|
2006-07-03 01:24:42 -06:00
|
|
|
raw_local_irq_disable();
|
|
|
|
_local_bh_enable();
|
|
|
|
raw_local_irq_enable();
|
2005-04-16 16:20:36 -06:00
|
|
|
__cond_resched();
|
|
|
|
local_bh_disable();
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(cond_resched_softirq);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* yield - yield the current processor to other threads.
|
|
|
|
*
|
|
|
|
* this is a shortcut for kernel-space yielding - it marks the
|
|
|
|
* thread runnable and calls sys_sched_yield().
|
|
|
|
*/
|
|
|
|
void __sched yield(void)
|
|
|
|
{
|
|
|
|
set_current_state(TASK_RUNNING);
|
|
|
|
sys_sched_yield();
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(yield);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This task is about to go to sleep on IO. Increment rq->nr_iowait so
|
|
|
|
* that process accounting knows that this is a task in IO wait state.
|
|
|
|
*
|
|
|
|
* But don't do that if it is a deliberate, throttling IO wait (this task
|
|
|
|
* has set its backing_dev_info: the queue against which it should throttle)
|
|
|
|
*/
|
|
|
|
void __sched io_schedule(void)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = &__raw_get_cpu_var(runqueues);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-07-14 01:24:37 -06:00
|
|
|
delayacct_blkio_start();
|
2005-04-16 16:20:36 -06:00
|
|
|
atomic_inc(&rq->nr_iowait);
|
|
|
|
schedule();
|
|
|
|
atomic_dec(&rq->nr_iowait);
|
2006-07-14 01:24:37 -06:00
|
|
|
delayacct_blkio_end();
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(io_schedule);
|
|
|
|
|
|
|
|
long __sched io_schedule_timeout(long timeout)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = &__raw_get_cpu_var(runqueues);
|
2005-04-16 16:20:36 -06:00
|
|
|
long ret;
|
|
|
|
|
2006-07-14 01:24:37 -06:00
|
|
|
delayacct_blkio_start();
|
2005-04-16 16:20:36 -06:00
|
|
|
atomic_inc(&rq->nr_iowait);
|
|
|
|
ret = schedule_timeout(timeout);
|
|
|
|
atomic_dec(&rq->nr_iowait);
|
2006-07-14 01:24:37 -06:00
|
|
|
delayacct_blkio_end();
|
2005-04-16 16:20:36 -06:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_get_priority_max - return maximum RT priority.
|
|
|
|
* @policy: scheduling class.
|
|
|
|
*
|
|
|
|
* this syscall returns the maximum rt_priority that can be used
|
|
|
|
* by a given scheduling class.
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_sched_get_priority_max(int policy)
|
|
|
|
{
|
|
|
|
int ret = -EINVAL;
|
|
|
|
|
|
|
|
switch (policy) {
|
|
|
|
case SCHED_FIFO:
|
|
|
|
case SCHED_RR:
|
|
|
|
ret = MAX_USER_RT_PRIO-1;
|
|
|
|
break;
|
|
|
|
case SCHED_NORMAL:
|
2006-01-14 14:20:41 -07:00
|
|
|
case SCHED_BATCH:
|
2005-04-16 16:20:36 -06:00
|
|
|
ret = 0;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_get_priority_min - return minimum RT priority.
|
|
|
|
* @policy: scheduling class.
|
|
|
|
*
|
|
|
|
* this syscall returns the minimum rt_priority that can be used
|
|
|
|
* by a given scheduling class.
|
|
|
|
*/
|
|
|
|
asmlinkage long sys_sched_get_priority_min(int policy)
|
|
|
|
{
|
|
|
|
int ret = -EINVAL;
|
|
|
|
|
|
|
|
switch (policy) {
|
|
|
|
case SCHED_FIFO:
|
|
|
|
case SCHED_RR:
|
|
|
|
ret = 1;
|
|
|
|
break;
|
|
|
|
case SCHED_NORMAL:
|
2006-01-14 14:20:41 -07:00
|
|
|
case SCHED_BATCH:
|
2005-04-16 16:20:36 -06:00
|
|
|
ret = 0;
|
|
|
|
}
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sys_sched_rr_get_interval - return the default timeslice of a process.
|
|
|
|
* @pid: pid of the process.
|
|
|
|
* @interval: userspace pointer to the timeslice value.
|
|
|
|
*
|
|
|
|
* this syscall writes the default timeslice value of a given process
|
|
|
|
* into the user-space timespec buffer. A value of '0' means infinity.
|
|
|
|
*/
|
|
|
|
asmlinkage
|
|
|
|
long sys_sched_rr_get_interval(pid_t pid, struct timespec __user *interval)
|
|
|
|
{
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *p;
|
2005-04-16 16:20:36 -06:00
|
|
|
int retval = -EINVAL;
|
|
|
|
struct timespec t;
|
|
|
|
|
|
|
|
if (pid < 0)
|
|
|
|
goto out_nounlock;
|
|
|
|
|
|
|
|
retval = -ESRCH;
|
|
|
|
read_lock(&tasklist_lock);
|
|
|
|
p = find_process_by_pid(pid);
|
|
|
|
if (!p)
|
|
|
|
goto out_unlock;
|
|
|
|
|
|
|
|
retval = security_task_getscheduler(p);
|
|
|
|
if (retval)
|
|
|
|
goto out_unlock;
|
|
|
|
|
2006-06-26 00:58:00 -06:00
|
|
|
jiffies_to_timespec(p->policy == SCHED_FIFO ?
|
2005-04-16 16:20:36 -06:00
|
|
|
0 : task_timeslice(p), &t);
|
|
|
|
read_unlock(&tasklist_lock);
|
|
|
|
retval = copy_to_user(interval, &t, sizeof(t)) ? -EFAULT : 0;
|
|
|
|
out_nounlock:
|
|
|
|
return retval;
|
|
|
|
out_unlock:
|
|
|
|
read_unlock(&tasklist_lock);
|
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline struct task_struct *eldest_child(struct task_struct *p)
|
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
if (list_empty(&p->children))
|
|
|
|
return NULL;
|
2005-04-16 16:20:36 -06:00
|
|
|
return list_entry(p->children.next,struct task_struct,sibling);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline struct task_struct *older_sibling(struct task_struct *p)
|
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
if (p->sibling.prev==&p->parent->children)
|
|
|
|
return NULL;
|
2005-04-16 16:20:36 -06:00
|
|
|
return list_entry(p->sibling.prev,struct task_struct,sibling);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline struct task_struct *younger_sibling(struct task_struct *p)
|
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
if (p->sibling.next==&p->parent->children)
|
|
|
|
return NULL;
|
2005-04-16 16:20:36 -06:00
|
|
|
return list_entry(p->sibling.next,struct task_struct,sibling);
|
|
|
|
}
|
|
|
|
|
2006-07-10 05:43:52 -06:00
|
|
|
static const char stat_nam[] = "RSDTtZX";
|
2006-07-03 01:25:41 -06:00
|
|
|
|
|
|
|
static void show_task(struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *relative;
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long free = 0;
|
2006-07-03 01:25:41 -06:00
|
|
|
unsigned state;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
state = p->state ? __ffs(p->state) + 1 : 0;
|
2006-07-10 05:43:52 -06:00
|
|
|
printk("%-13.13s %c", p->comm,
|
|
|
|
state < sizeof(stat_nam) - 1 ? stat_nam[state] : '?');
|
2005-04-16 16:20:36 -06:00
|
|
|
#if (BITS_PER_LONG == 32)
|
|
|
|
if (state == TASK_RUNNING)
|
|
|
|
printk(" running ");
|
|
|
|
else
|
|
|
|
printk(" %08lX ", thread_saved_pc(p));
|
|
|
|
#else
|
|
|
|
if (state == TASK_RUNNING)
|
|
|
|
printk(" running task ");
|
|
|
|
else
|
|
|
|
printk(" %016lx ", thread_saved_pc(p));
|
|
|
|
#endif
|
|
|
|
#ifdef CONFIG_DEBUG_STACK_USAGE
|
|
|
|
{
|
2005-11-13 17:06:56 -07:00
|
|
|
unsigned long *n = end_of_stack(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
while (!*n)
|
|
|
|
n++;
|
2005-11-13 17:06:56 -07:00
|
|
|
free = (unsigned long)n - (unsigned long)end_of_stack(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
printk("%5lu %5d %6d ", free, p->pid, p->parent->pid);
|
|
|
|
if ((relative = eldest_child(p)))
|
|
|
|
printk("%5d ", relative->pid);
|
|
|
|
else
|
|
|
|
printk(" ");
|
|
|
|
if ((relative = younger_sibling(p)))
|
|
|
|
printk("%7d", relative->pid);
|
|
|
|
else
|
|
|
|
printk(" ");
|
|
|
|
if ((relative = older_sibling(p)))
|
|
|
|
printk(" %5d", relative->pid);
|
|
|
|
else
|
|
|
|
printk(" ");
|
|
|
|
if (!p->mm)
|
|
|
|
printk(" (L-TLB)\n");
|
|
|
|
else
|
|
|
|
printk(" (NOTLB)\n");
|
|
|
|
|
|
|
|
if (state != TASK_RUNNING)
|
|
|
|
show_stack(p, NULL);
|
|
|
|
}
|
|
|
|
|
|
|
|
void show_state(void)
|
|
|
|
{
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *g, *p;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
#if (BITS_PER_LONG == 32)
|
|
|
|
printk("\n"
|
|
|
|
" sibling\n");
|
|
|
|
printk(" task PC pid father child younger older\n");
|
|
|
|
#else
|
|
|
|
printk("\n"
|
|
|
|
" sibling\n");
|
|
|
|
printk(" task PC pid father child younger older\n");
|
|
|
|
#endif
|
|
|
|
read_lock(&tasklist_lock);
|
|
|
|
do_each_thread(g, p) {
|
|
|
|
/*
|
|
|
|
* reset the NMI-timeout, listing all files on a slow
|
|
|
|
* console might take alot of time:
|
|
|
|
*/
|
|
|
|
touch_nmi_watchdog();
|
|
|
|
show_task(p);
|
|
|
|
} while_each_thread(g, p);
|
|
|
|
|
|
|
|
read_unlock(&tasklist_lock);
|
2006-07-03 01:24:33 -06:00
|
|
|
debug_show_all_locks();
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2005-06-28 08:40:42 -06:00
|
|
|
/**
|
|
|
|
* init_idle - set up an idle thread for a given CPU
|
|
|
|
* @idle: task in question
|
|
|
|
* @cpu: cpu the idle task belongs to
|
|
|
|
*
|
|
|
|
* NOTE: this function does not set the idle thread's NEED_RESCHED
|
|
|
|
* flag, to make booting more robust.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
void __devinit init_idle(struct task_struct *idle, int cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long flags;
|
|
|
|
|
2006-03-07 22:55:27 -07:00
|
|
|
idle->timestamp = sched_clock();
|
2005-04-16 16:20:36 -06:00
|
|
|
idle->sleep_avg = 0;
|
|
|
|
idle->array = NULL;
|
2006-06-27 03:54:51 -06:00
|
|
|
idle->prio = idle->normal_prio = MAX_PRIO;
|
2005-04-16 16:20:36 -06:00
|
|
|
idle->state = TASK_RUNNING;
|
|
|
|
idle->cpus_allowed = cpumask_of_cpu(cpu);
|
|
|
|
set_task_cpu(idle, cpu);
|
|
|
|
|
|
|
|
spin_lock_irqsave(&rq->lock, flags);
|
|
|
|
rq->curr = rq->idle = idle;
|
2005-06-25 15:57:23 -06:00
|
|
|
#if defined(CONFIG_SMP) && defined(__ARCH_WANT_UNLOCKED_CTXSW)
|
|
|
|
idle->oncpu = 1;
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
spin_unlock_irqrestore(&rq->lock, flags);
|
|
|
|
|
|
|
|
/* Set the preempt count _outside_ the spinlocks! */
|
|
|
|
#if defined(CONFIG_PREEMPT) && !defined(CONFIG_PREEMPT_BKL)
|
2005-11-13 17:06:55 -07:00
|
|
|
task_thread_info(idle)->preempt_count = (idle->lock_depth >= 0);
|
2005-04-16 16:20:36 -06:00
|
|
|
#else
|
2005-11-13 17:06:55 -07:00
|
|
|
task_thread_info(idle)->preempt_count = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* In a system that switches off the HZ timer nohz_cpu_mask
|
|
|
|
* indicates which cpus entered this state. This is used
|
|
|
|
* in the rcu update to wait only for active cpus. For system
|
|
|
|
* which do not switch off the HZ timer nohz_cpu_mask should
|
|
|
|
* always be CPU_MASK_NONE.
|
|
|
|
*/
|
|
|
|
cpumask_t nohz_cpu_mask = CPU_MASK_NONE;
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
/*
|
|
|
|
* This is how migration works:
|
|
|
|
*
|
2006-07-03 01:25:42 -06:00
|
|
|
* 1) we queue a struct migration_req structure in the source CPU's
|
2005-04-16 16:20:36 -06:00
|
|
|
* runqueue and wake up that CPU's migration thread.
|
|
|
|
* 2) we down() the locked semaphore => thread blocks.
|
|
|
|
* 3) migration thread wakes up (implicitly it forces the migrated
|
|
|
|
* thread off the CPU)
|
|
|
|
* 4) it gets the migration request and checks whether the migrated
|
|
|
|
* task is still in the wrong runqueue.
|
|
|
|
* 5) if it's in the wrong runqueue then the migration thread removes
|
|
|
|
* it and puts it into the right queue.
|
|
|
|
* 6) migration thread up()s the semaphore.
|
|
|
|
* 7) we wake up and the migration is done.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Change a given task's CPU affinity. Migrate the thread to a
|
|
|
|
* proper CPU and schedule it away if the CPU it's executing on
|
|
|
|
* is removed from the allowed bitmask.
|
|
|
|
*
|
|
|
|
* NOTE: the caller must have a valid reference to the task, the
|
|
|
|
* task must not exit() & deallocate itself prematurely. The
|
|
|
|
* call is not atomic; no spinlocks may be held.
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
int set_cpus_allowed(struct task_struct *p, cpumask_t new_mask)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct migration_req req;
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long flags;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2006-07-03 01:25:40 -06:00
|
|
|
int ret = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
rq = task_rq_lock(p, &flags);
|
|
|
|
if (!cpus_intersects(new_mask, cpu_online_map)) {
|
|
|
|
ret = -EINVAL;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
p->cpus_allowed = new_mask;
|
|
|
|
/* Can the task run on the task's current CPU? If so, we're done */
|
|
|
|
if (cpu_isset(task_cpu(p), new_mask))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
if (migrate_task(p, any_online_cpu(new_mask), &req)) {
|
|
|
|
/* Need help from migration thread: drop lock and wait. */
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
wake_up_process(rq->migration_thread);
|
|
|
|
wait_for_completion(&req.done);
|
|
|
|
tlb_migrate_finish(p->mm);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
out:
|
|
|
|
task_rq_unlock(rq, &flags);
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(set_cpus_allowed);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Move (not current) task off this cpu, onto dest cpu. We're doing
|
|
|
|
* this because either it can't run here any more (set_cpus_allowed()
|
|
|
|
* away from this CPU, or CPU going down), or because we're
|
|
|
|
* attempting to rebalance this task on exec (sched_exec).
|
|
|
|
*
|
|
|
|
* So we race with normal scheduler movements, but that's OK, as long
|
|
|
|
* as the task is no longer on this CPU.
|
2006-06-27 03:54:32 -06:00
|
|
|
*
|
|
|
|
* Returns non-zero if task was successfully migrated.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2006-06-27 03:54:32 -06:00
|
|
|
static int __migrate_task(struct task_struct *p, int src_cpu, int dest_cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq_dest, *rq_src;
|
2006-06-27 03:54:32 -06:00
|
|
|
int ret = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
if (unlikely(cpu_is_offline(dest_cpu)))
|
2006-06-27 03:54:32 -06:00
|
|
|
return ret;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
rq_src = cpu_rq(src_cpu);
|
|
|
|
rq_dest = cpu_rq(dest_cpu);
|
|
|
|
|
|
|
|
double_rq_lock(rq_src, rq_dest);
|
|
|
|
/* Already moved. */
|
|
|
|
if (task_cpu(p) != src_cpu)
|
|
|
|
goto out;
|
|
|
|
/* Affinity changed (again). */
|
|
|
|
if (!cpu_isset(dest_cpu, p->cpus_allowed))
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
set_task_cpu(p, dest_cpu);
|
|
|
|
if (p->array) {
|
|
|
|
/*
|
|
|
|
* Sync timestamp with rq_dest's before activating.
|
|
|
|
* The same thing could be achieved by doing this step
|
|
|
|
* afterwards, and pretending it was a local activate.
|
|
|
|
* This way is cleaner and logically correct.
|
|
|
|
*/
|
|
|
|
p->timestamp = p->timestamp - rq_src->timestamp_last_tick
|
|
|
|
+ rq_dest->timestamp_last_tick;
|
|
|
|
deactivate_task(p, rq_src);
|
2006-07-10 05:43:51 -06:00
|
|
|
__activate_task(p, rq_dest);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (TASK_PREEMPTS_CURR(p, rq_dest))
|
|
|
|
resched_task(rq_dest->curr);
|
|
|
|
}
|
2006-06-27 03:54:32 -06:00
|
|
|
ret = 1;
|
2005-04-16 16:20:36 -06:00
|
|
|
out:
|
|
|
|
double_rq_unlock(rq_src, rq_dest);
|
2006-06-27 03:54:32 -06:00
|
|
|
return ret;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* migration_thread - this is a highprio system thread that performs
|
|
|
|
* thread migration by bumping thread off CPU then 'pushing' onto
|
|
|
|
* another runqueue.
|
|
|
|
*/
|
2005-09-10 01:26:11 -06:00
|
|
|
static int migration_thread(void *data)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
int cpu = (long)data;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
rq = cpu_rq(cpu);
|
|
|
|
BUG_ON(rq->migration_thread != current);
|
|
|
|
|
|
|
|
set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
while (!kthread_should_stop()) {
|
2006-07-03 01:25:42 -06:00
|
|
|
struct migration_req *req;
|
2005-04-16 16:20:36 -06:00
|
|
|
struct list_head *head;
|
|
|
|
|
2005-06-25 00:13:50 -06:00
|
|
|
try_to_freeze();
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
spin_lock_irq(&rq->lock);
|
|
|
|
|
|
|
|
if (cpu_is_offline(cpu)) {
|
|
|
|
spin_unlock_irq(&rq->lock);
|
|
|
|
goto wait_to_die;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (rq->active_balance) {
|
|
|
|
active_load_balance(rq, cpu);
|
|
|
|
rq->active_balance = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
head = &rq->migration_queue;
|
|
|
|
|
|
|
|
if (list_empty(head)) {
|
|
|
|
spin_unlock_irq(&rq->lock);
|
|
|
|
schedule();
|
|
|
|
set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
continue;
|
|
|
|
}
|
2006-07-03 01:25:42 -06:00
|
|
|
req = list_entry(head->next, struct migration_req, list);
|
2005-04-16 16:20:36 -06:00
|
|
|
list_del_init(head->next);
|
|
|
|
|
2005-06-25 15:57:27 -06:00
|
|
|
spin_unlock(&rq->lock);
|
|
|
|
__migrate_task(req->task, cpu, req->dest_cpu);
|
|
|
|
local_irq_enable();
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
complete(&req->done);
|
|
|
|
}
|
|
|
|
__set_current_state(TASK_RUNNING);
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
wait_to_die:
|
|
|
|
/* Wait for kthread_stop */
|
|
|
|
set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
while (!kthread_should_stop()) {
|
|
|
|
schedule();
|
|
|
|
set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
}
|
|
|
|
__set_current_state(TASK_RUNNING);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_HOTPLUG_CPU
|
|
|
|
/* Figure out where task on dead CPU should go, use force if neccessary. */
|
2006-07-03 01:25:40 -06:00
|
|
|
static void move_task_off_dead_cpu(int dead_cpu, struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-06-27 03:54:32 -06:00
|
|
|
unsigned long flags;
|
2005-04-16 16:20:36 -06:00
|
|
|
cpumask_t mask;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
|
|
|
int dest_cpu;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:32 -06:00
|
|
|
restart:
|
2005-04-16 16:20:36 -06:00
|
|
|
/* On same node? */
|
|
|
|
mask = node_to_cpumask(cpu_to_node(dead_cpu));
|
2006-07-03 01:25:40 -06:00
|
|
|
cpus_and(mask, mask, p->cpus_allowed);
|
2005-04-16 16:20:36 -06:00
|
|
|
dest_cpu = any_online_cpu(mask);
|
|
|
|
|
|
|
|
/* On any allowed CPU? */
|
|
|
|
if (dest_cpu == NR_CPUS)
|
2006-07-03 01:25:40 -06:00
|
|
|
dest_cpu = any_online_cpu(p->cpus_allowed);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* No more Mr. Nice Guy. */
|
|
|
|
if (dest_cpu == NR_CPUS) {
|
2006-07-03 01:25:40 -06:00
|
|
|
rq = task_rq_lock(p, &flags);
|
|
|
|
cpus_setall(p->cpus_allowed);
|
|
|
|
dest_cpu = any_online_cpu(p->cpus_allowed);
|
2006-06-27 03:54:32 -06:00
|
|
|
task_rq_unlock(rq, &flags);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Don't tell them about moving exiting tasks or
|
|
|
|
* kernel threads (both mm NULL), since they never
|
|
|
|
* leave kernel.
|
|
|
|
*/
|
2006-07-03 01:25:40 -06:00
|
|
|
if (p->mm && printk_ratelimit())
|
2005-04-16 16:20:36 -06:00
|
|
|
printk(KERN_INFO "process %d (%s) no "
|
|
|
|
"longer affine to cpu%d\n",
|
2006-07-03 01:25:40 -06:00
|
|
|
p->pid, p->comm, dead_cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2006-07-03 01:25:40 -06:00
|
|
|
if (!__migrate_task(p, dead_cpu, dest_cpu))
|
2006-06-27 03:54:32 -06:00
|
|
|
goto restart;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* While a dead CPU has no uninterruptible tasks queued at this point,
|
|
|
|
* it might still have a nonzero ->nr_uninterruptible counter, because
|
|
|
|
* for performance reasons the counter is not stricly tracking tasks to
|
|
|
|
* their home CPUs. So we just add the counter to another CPU's counter,
|
|
|
|
* to keep the global sum constant after CPU-down:
|
|
|
|
*/
|
2006-07-03 01:25:42 -06:00
|
|
|
static void migrate_nr_uninterruptible(struct rq *rq_src)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq_dest = cpu_rq(any_online_cpu(CPU_MASK_ALL));
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
local_irq_save(flags);
|
|
|
|
double_rq_lock(rq_src, rq_dest);
|
|
|
|
rq_dest->nr_uninterruptible += rq_src->nr_uninterruptible;
|
|
|
|
rq_src->nr_uninterruptible = 0;
|
|
|
|
double_rq_unlock(rq_src, rq_dest);
|
|
|
|
local_irq_restore(flags);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Run through task list and migrate tasks from the dead cpu. */
|
|
|
|
static void migrate_live_tasks(int src_cpu)
|
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
struct task_struct *p, *t;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
write_lock_irq(&tasklist_lock);
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
do_each_thread(t, p) {
|
|
|
|
if (p == current)
|
2005-04-16 16:20:36 -06:00
|
|
|
continue;
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
if (task_cpu(p) == src_cpu)
|
|
|
|
move_task_off_dead_cpu(src_cpu, p);
|
|
|
|
} while_each_thread(t, p);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
write_unlock_irq(&tasklist_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Schedules idle task to be the next runnable task on current CPU.
|
|
|
|
* It does so by boosting its priority to highest possible and adding it to
|
2006-07-03 01:25:40 -06:00
|
|
|
* the _front_ of the runqueue. Used by CPU offline code.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
|
|
|
void sched_idle_next(void)
|
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
int this_cpu = smp_processor_id();
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(this_cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
struct task_struct *p = rq->idle;
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
/* cpu has to be offline */
|
2006-07-03 01:25:40 -06:00
|
|
|
BUG_ON(cpu_online(this_cpu));
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
/*
|
|
|
|
* Strictly not necessary since rest of the CPUs are stopped by now
|
|
|
|
* and interrupts disabled on the current cpu.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
|
|
|
spin_lock_irqsave(&rq->lock, flags);
|
|
|
|
|
|
|
|
__setscheduler(p, SCHED_FIFO, MAX_RT_PRIO-1);
|
2006-07-03 01:25:40 -06:00
|
|
|
|
|
|
|
/* Add idle task to the _front_ of its priority queue: */
|
2005-04-16 16:20:36 -06:00
|
|
|
__activate_idle_task(p, rq);
|
|
|
|
|
|
|
|
spin_unlock_irqrestore(&rq->lock, flags);
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
/*
|
|
|
|
* Ensures that the idle task is using init_mm right before its cpu goes
|
2005-04-16 16:20:36 -06:00
|
|
|
* offline.
|
|
|
|
*/
|
|
|
|
void idle_task_exit(void)
|
|
|
|
{
|
|
|
|
struct mm_struct *mm = current->active_mm;
|
|
|
|
|
|
|
|
BUG_ON(cpu_online(smp_processor_id()));
|
|
|
|
|
|
|
|
if (mm != &init_mm)
|
|
|
|
switch_mm(mm, &init_mm, current);
|
|
|
|
mmdrop(mm);
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:41 -06:00
|
|
|
static void migrate_dead(unsigned int dead_cpu, struct task_struct *p)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(dead_cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Must be exiting, otherwise would be on tasklist. */
|
2006-07-03 01:25:40 -06:00
|
|
|
BUG_ON(p->exit_state != EXIT_ZOMBIE && p->exit_state != EXIT_DEAD);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Cannot have done final schedule yet: would have vanished. */
|
2006-07-03 01:25:40 -06:00
|
|
|
BUG_ON(p->flags & PF_DEAD);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
get_task_struct(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Drop lock around migration; if someone else moves it,
|
|
|
|
* that's OK. No task can be added to this CPU, so iteration is
|
|
|
|
* fine.
|
|
|
|
*/
|
|
|
|
spin_unlock_irq(&rq->lock);
|
2006-07-03 01:25:40 -06:00
|
|
|
move_task_off_dead_cpu(dead_cpu, p);
|
2005-04-16 16:20:36 -06:00
|
|
|
spin_lock_irq(&rq->lock);
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
put_task_struct(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* release_task() removes task from tasklist, so we won't find dead tasks. */
|
|
|
|
static void migrate_dead_tasks(unsigned int dead_cpu)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(dead_cpu);
|
2006-07-03 01:25:40 -06:00
|
|
|
unsigned int arr, i;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
for (arr = 0; arr < 2; arr++) {
|
|
|
|
for (i = 0; i < MAX_PRIO; i++) {
|
|
|
|
struct list_head *list = &rq->arrays[arr].queue[i];
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
while (!list_empty(list))
|
2006-07-03 01:25:41 -06:00
|
|
|
migrate_dead(dead_cpu, list_entry(list->next,
|
|
|
|
struct task_struct, run_list));
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#endif /* CONFIG_HOTPLUG_CPU */
|
|
|
|
|
|
|
|
/*
|
|
|
|
* migration_call - callback that gets triggered when a CPU is added.
|
|
|
|
* Here we can start up the necessary migration thread for the new CPU.
|
|
|
|
*/
|
2006-07-03 01:25:40 -06:00
|
|
|
static int __cpuinit
|
|
|
|
migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct task_struct *p;
|
2006-07-03 01:25:40 -06:00
|
|
|
int cpu = (long)hcpu;
|
2005-04-16 16:20:36 -06:00
|
|
|
unsigned long flags;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
switch (action) {
|
|
|
|
case CPU_UP_PREPARE:
|
|
|
|
p = kthread_create(migration_thread, hcpu, "migration/%d",cpu);
|
|
|
|
if (IS_ERR(p))
|
|
|
|
return NOTIFY_BAD;
|
|
|
|
p->flags |= PF_NOFREEZE;
|
|
|
|
kthread_bind(p, cpu);
|
|
|
|
/* Must be high prio: stop_machine expects to yield to it. */
|
|
|
|
rq = task_rq_lock(p, &flags);
|
|
|
|
__setscheduler(p, SCHED_FIFO, MAX_RT_PRIO-1);
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
cpu_rq(cpu)->migration_thread = p;
|
|
|
|
break;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
case CPU_ONLINE:
|
|
|
|
/* Strictly unneccessary, as first user will wake it. */
|
|
|
|
wake_up_process(cpu_rq(cpu)->migration_thread);
|
|
|
|
break;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_HOTPLUG_CPU
|
|
|
|
case CPU_UP_CANCELED:
|
2006-06-25 06:49:10 -06:00
|
|
|
if (!cpu_rq(cpu)->migration_thread)
|
|
|
|
break;
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Unbind it from offline cpu so it can run. Fall thru. */
|
2005-11-07 01:58:38 -07:00
|
|
|
kthread_bind(cpu_rq(cpu)->migration_thread,
|
|
|
|
any_online_cpu(cpu_online_map));
|
2005-04-16 16:20:36 -06:00
|
|
|
kthread_stop(cpu_rq(cpu)->migration_thread);
|
|
|
|
cpu_rq(cpu)->migration_thread = NULL;
|
|
|
|
break;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
case CPU_DEAD:
|
|
|
|
migrate_live_tasks(cpu);
|
|
|
|
rq = cpu_rq(cpu);
|
|
|
|
kthread_stop(rq->migration_thread);
|
|
|
|
rq->migration_thread = NULL;
|
|
|
|
/* Idle task back to normal (off runqueue, low prio) */
|
|
|
|
rq = task_rq_lock(rq->idle, &flags);
|
|
|
|
deactivate_task(rq->idle, rq);
|
|
|
|
rq->idle->static_prio = MAX_PRIO;
|
|
|
|
__setscheduler(rq->idle, SCHED_NORMAL, 0);
|
|
|
|
migrate_dead_tasks(cpu);
|
|
|
|
task_rq_unlock(rq, &flags);
|
|
|
|
migrate_nr_uninterruptible(rq);
|
|
|
|
BUG_ON(rq->nr_running != 0);
|
|
|
|
|
|
|
|
/* No need to migrate the tasks: it was best-effort if
|
|
|
|
* they didn't do lock_cpu_hotplug(). Just wake up
|
|
|
|
* the requestors. */
|
|
|
|
spin_lock_irq(&rq->lock);
|
|
|
|
while (!list_empty(&rq->migration_queue)) {
|
2006-07-03 01:25:42 -06:00
|
|
|
struct migration_req *req;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
req = list_entry(rq->migration_queue.next,
|
2006-07-03 01:25:42 -06:00
|
|
|
struct migration_req, list);
|
2005-04-16 16:20:36 -06:00
|
|
|
list_del_init(&req->list);
|
|
|
|
complete(&req->done);
|
|
|
|
}
|
|
|
|
spin_unlock_irq(&rq->lock);
|
|
|
|
break;
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
return NOTIFY_OK;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Register at highest priority so that task migration (migrate_all_tasks)
|
|
|
|
* happens before everything else.
|
|
|
|
*/
|
2006-06-27 03:54:10 -06:00
|
|
|
static struct notifier_block __cpuinitdata migration_notifier = {
|
2005-04-16 16:20:36 -06:00
|
|
|
.notifier_call = migration_call,
|
|
|
|
.priority = 10
|
|
|
|
};
|
|
|
|
|
|
|
|
int __init migration_init(void)
|
|
|
|
{
|
|
|
|
void *cpu = (void *)(long)smp_processor_id();
|
2006-07-03 01:25:40 -06:00
|
|
|
|
|
|
|
/* Start one for the boot CPU: */
|
2005-04-16 16:20:36 -06:00
|
|
|
migration_call(&migration_notifier, CPU_UP_PREPARE, cpu);
|
|
|
|
migration_call(&migration_notifier, CPU_ONLINE, cpu);
|
|
|
|
register_cpu_notifier(&migration_notifier);
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
2005-06-25 15:57:33 -06:00
|
|
|
#undef SCHED_DOMAIN_DEBUG
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef SCHED_DOMAIN_DEBUG
|
|
|
|
static void sched_domain_debug(struct sched_domain *sd, int cpu)
|
|
|
|
{
|
|
|
|
int level = 0;
|
|
|
|
|
2005-06-25 15:57:24 -06:00
|
|
|
if (!sd) {
|
|
|
|
printk(KERN_DEBUG "CPU%d attaching NULL sched-domain.\n", cpu);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
printk(KERN_DEBUG "CPU%d attaching sched-domain:\n", cpu);
|
|
|
|
|
|
|
|
do {
|
|
|
|
int i;
|
|
|
|
char str[NR_CPUS];
|
|
|
|
struct sched_group *group = sd->groups;
|
|
|
|
cpumask_t groupmask;
|
|
|
|
|
|
|
|
cpumask_scnprintf(str, NR_CPUS, sd->span);
|
|
|
|
cpus_clear(groupmask);
|
|
|
|
|
|
|
|
printk(KERN_DEBUG);
|
|
|
|
for (i = 0; i < level + 1; i++)
|
|
|
|
printk(" ");
|
|
|
|
printk("domain %d: ", level);
|
|
|
|
|
|
|
|
if (!(sd->flags & SD_LOAD_BALANCE)) {
|
|
|
|
printk("does not load-balance\n");
|
|
|
|
if (sd->parent)
|
|
|
|
printk(KERN_ERR "ERROR: !SD_LOAD_BALANCE domain has parent");
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
printk("span %s\n", str);
|
|
|
|
|
|
|
|
if (!cpu_isset(cpu, sd->span))
|
|
|
|
printk(KERN_ERR "ERROR: domain->span does not contain CPU%d\n", cpu);
|
|
|
|
if (!cpu_isset(cpu, group->cpumask))
|
|
|
|
printk(KERN_ERR "ERROR: domain->groups does not contain CPU%d\n", cpu);
|
|
|
|
|
|
|
|
printk(KERN_DEBUG);
|
|
|
|
for (i = 0; i < level + 2; i++)
|
|
|
|
printk(" ");
|
|
|
|
printk("groups:");
|
|
|
|
do {
|
|
|
|
if (!group) {
|
|
|
|
printk("\n");
|
|
|
|
printk(KERN_ERR "ERROR: group is NULL\n");
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!group->cpu_power) {
|
|
|
|
printk("\n");
|
|
|
|
printk(KERN_ERR "ERROR: domain->cpu_power not set\n");
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!cpus_weight(group->cpumask)) {
|
|
|
|
printk("\n");
|
|
|
|
printk(KERN_ERR "ERROR: empty group\n");
|
|
|
|
}
|
|
|
|
|
|
|
|
if (cpus_intersects(groupmask, group->cpumask)) {
|
|
|
|
printk("\n");
|
|
|
|
printk(KERN_ERR "ERROR: repeated CPUs\n");
|
|
|
|
}
|
|
|
|
|
|
|
|
cpus_or(groupmask, groupmask, group->cpumask);
|
|
|
|
|
|
|
|
cpumask_scnprintf(str, NR_CPUS, group->cpumask);
|
|
|
|
printk(" %s", str);
|
|
|
|
|
|
|
|
group = group->next;
|
|
|
|
} while (group != sd->groups);
|
|
|
|
printk("\n");
|
|
|
|
|
|
|
|
if (!cpus_equal(sd->span, groupmask))
|
|
|
|
printk(KERN_ERR "ERROR: groups don't span domain->span\n");
|
|
|
|
|
|
|
|
level++;
|
|
|
|
sd = sd->parent;
|
|
|
|
|
|
|
|
if (sd) {
|
|
|
|
if (!cpus_subset(groupmask, sd->span))
|
|
|
|
printk(KERN_ERR "ERROR: parent span is not a superset of domain->span\n");
|
|
|
|
}
|
|
|
|
|
|
|
|
} while (sd);
|
|
|
|
}
|
|
|
|
#else
|
2006-07-03 01:25:40 -06:00
|
|
|
# define sched_domain_debug(sd, cpu) do { } while (0)
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
|
|
|
|
2005-06-25 15:57:33 -06:00
|
|
|
static int sd_degenerate(struct sched_domain *sd)
|
2005-06-25 15:57:25 -06:00
|
|
|
{
|
|
|
|
if (cpus_weight(sd->span) == 1)
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
/* Following flags need at least 2 groups */
|
|
|
|
if (sd->flags & (SD_LOAD_BALANCE |
|
|
|
|
SD_BALANCE_NEWIDLE |
|
|
|
|
SD_BALANCE_FORK |
|
|
|
|
SD_BALANCE_EXEC)) {
|
|
|
|
if (sd->groups != sd->groups->next)
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Following flags don't use groups */
|
|
|
|
if (sd->flags & (SD_WAKE_IDLE |
|
|
|
|
SD_WAKE_AFFINE |
|
|
|
|
SD_WAKE_BALANCE))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
static int
|
|
|
|
sd_parent_degenerate(struct sched_domain *sd, struct sched_domain *parent)
|
2005-06-25 15:57:25 -06:00
|
|
|
{
|
|
|
|
unsigned long cflags = sd->flags, pflags = parent->flags;
|
|
|
|
|
|
|
|
if (sd_degenerate(parent))
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
if (!cpus_equal(sd->span, parent->span))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* Does parent contain flags not in child? */
|
|
|
|
/* WAKE_BALANCE is a subset of WAKE_AFFINE */
|
|
|
|
if (cflags & SD_WAKE_AFFINE)
|
|
|
|
pflags &= ~SD_WAKE_BALANCE;
|
|
|
|
/* Flags needing groups don't count if only 1 group in parent */
|
|
|
|
if (parent->groups == parent->groups->next) {
|
|
|
|
pflags &= ~(SD_LOAD_BALANCE |
|
|
|
|
SD_BALANCE_NEWIDLE |
|
|
|
|
SD_BALANCE_FORK |
|
|
|
|
SD_BALANCE_EXEC);
|
|
|
|
}
|
|
|
|
if (~cflags & pflags)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* Attach the domain 'sd' to 'cpu' as its base domain. Callers must
|
|
|
|
* hold the hotplug lock.
|
|
|
|
*/
|
2005-09-06 16:18:14 -06:00
|
|
|
static void cpu_attach_domain(struct sched_domain *sd, int cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq = cpu_rq(cpu);
|
2005-06-25 15:57:25 -06:00
|
|
|
struct sched_domain *tmp;
|
|
|
|
|
|
|
|
/* Remove the sched domains which do not contribute to scheduling. */
|
|
|
|
for (tmp = sd; tmp; tmp = tmp->parent) {
|
|
|
|
struct sched_domain *parent = tmp->parent;
|
|
|
|
if (!parent)
|
|
|
|
break;
|
|
|
|
if (sd_parent_degenerate(tmp, parent))
|
|
|
|
tmp->parent = parent->parent;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (sd && sd_degenerate(sd))
|
|
|
|
sd = sd->parent;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
sched_domain_debug(sd, cpu);
|
|
|
|
|
2005-06-25 15:57:27 -06:00
|
|
|
rcu_assign_pointer(rq->sd, sd);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
/* cpus with isolated domains */
|
2005-09-06 16:18:14 -06:00
|
|
|
static cpumask_t __devinitdata cpu_isolated_map = CPU_MASK_NONE;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/* Setup the mask of cpus configured for isolated domains */
|
|
|
|
static int __init isolated_cpu_setup(char *str)
|
|
|
|
{
|
|
|
|
int ints[NR_CPUS], i;
|
|
|
|
|
|
|
|
str = get_options(str, ARRAY_SIZE(ints), ints);
|
|
|
|
cpus_clear(cpu_isolated_map);
|
|
|
|
for (i = 1; i <= ints[0]; i++)
|
|
|
|
if (ints[i] < NR_CPUS)
|
|
|
|
cpu_set(ints[i], cpu_isolated_map);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
__setup ("isolcpus=", isolated_cpu_setup);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* init_sched_build_groups takes an array of groups, the cpumask we wish
|
|
|
|
* to span, and a pointer to a function which identifies what group a CPU
|
|
|
|
* belongs to. The return value of group_fn must be a valid index into the
|
|
|
|
* groups[] array, and must be >= 0 and < NR_CPUS (due to the fact that we
|
|
|
|
* keep track of groups covered with a cpumask_t).
|
|
|
|
*
|
|
|
|
* init_sched_build_groups will build a circular linked list of the groups
|
|
|
|
* covered by the given span, and will set each group's ->cpumask correctly,
|
|
|
|
* and ->cpu_power to 0.
|
|
|
|
*/
|
2005-09-06 16:18:14 -06:00
|
|
|
static void init_sched_build_groups(struct sched_group groups[], cpumask_t span,
|
|
|
|
int (*group_fn)(int cpu))
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
struct sched_group *first = NULL, *last = NULL;
|
|
|
|
cpumask_t covered = CPU_MASK_NONE;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for_each_cpu_mask(i, span) {
|
|
|
|
int group = group_fn(i);
|
|
|
|
struct sched_group *sg = &groups[group];
|
|
|
|
int j;
|
|
|
|
|
|
|
|
if (cpu_isset(i, covered))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
sg->cpumask = CPU_MASK_NONE;
|
|
|
|
sg->cpu_power = 0;
|
|
|
|
|
|
|
|
for_each_cpu_mask(j, span) {
|
|
|
|
if (group_fn(j) != group)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
cpu_set(j, covered);
|
|
|
|
cpu_set(j, sg->cpumask);
|
|
|
|
}
|
|
|
|
if (!first)
|
|
|
|
first = sg;
|
|
|
|
if (last)
|
|
|
|
last->next = sg;
|
|
|
|
last = sg;
|
|
|
|
}
|
|
|
|
last->next = first;
|
|
|
|
}
|
|
|
|
|
2005-09-06 16:18:14 -06:00
|
|
|
#define SD_NODES_PER_DOMAIN 16
|
2005-04-16 16:20:36 -06:00
|
|
|
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
/*
|
|
|
|
* Self-tuning task migration cost measurement between source and target CPUs.
|
|
|
|
*
|
|
|
|
* This is done by measuring the cost of manipulating buffers of varying
|
|
|
|
* sizes. For a given buffer-size here are the steps that are taken:
|
|
|
|
*
|
|
|
|
* 1) the source CPU reads+dirties a shared buffer
|
|
|
|
* 2) the target CPU reads+dirties the same shared buffer
|
|
|
|
*
|
|
|
|
* We measure how long they take, in the following 4 scenarios:
|
|
|
|
*
|
|
|
|
* - source: CPU1, target: CPU2 | cost1
|
|
|
|
* - source: CPU2, target: CPU1 | cost2
|
|
|
|
* - source: CPU1, target: CPU1 | cost3
|
|
|
|
* - source: CPU2, target: CPU2 | cost4
|
|
|
|
*
|
|
|
|
* We then calculate the cost3+cost4-cost1-cost2 difference - this is
|
|
|
|
* the cost of migration.
|
|
|
|
*
|
|
|
|
* We then start off from a small buffer-size and iterate up to larger
|
|
|
|
* buffer sizes, in 5% steps - measuring each buffer-size separately, and
|
|
|
|
* doing a maximum search for the cost. (The maximum cost for a migration
|
|
|
|
* normally occurs when the working set size is around the effective cache
|
|
|
|
* size.)
|
|
|
|
*/
|
|
|
|
#define SEARCH_SCOPE 2
|
|
|
|
#define MIN_CACHE_SIZE (64*1024U)
|
|
|
|
#define DEFAULT_CACHE_SIZE (5*1024*1024U)
|
2006-01-30 12:24:38 -07:00
|
|
|
#define ITERATIONS 1
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
#define SIZE_THRESH 130
|
|
|
|
#define COST_THRESH 130
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The migration cost is a function of 'domain distance'. Domain
|
|
|
|
* distance is the number of steps a CPU has to iterate down its
|
|
|
|
* domain tree to share a domain with the other CPU. The farther
|
|
|
|
* two CPUs are from each other, the larger the distance gets.
|
|
|
|
*
|
|
|
|
* Note that we use the distance only to cache measurement results,
|
|
|
|
* the distance value is not used numerically otherwise. When two
|
|
|
|
* CPUs have the same distance it is assumed that the migration
|
|
|
|
* cost is the same. (this is a simplification but quite practical)
|
|
|
|
*/
|
|
|
|
#define MAX_DOMAIN_DISTANCE 32
|
|
|
|
|
|
|
|
static unsigned long long migration_cost[MAX_DOMAIN_DISTANCE] =
|
2006-02-17 14:52:44 -07:00
|
|
|
{ [ 0 ... MAX_DOMAIN_DISTANCE-1 ] =
|
|
|
|
/*
|
|
|
|
* Architectures may override the migration cost and thus avoid
|
|
|
|
* boot-time calibration. Unit is nanoseconds. Mostly useful for
|
|
|
|
* virtualized hardware:
|
|
|
|
*/
|
|
|
|
#ifdef CONFIG_DEFAULT_MIGRATION_COST
|
|
|
|
CONFIG_DEFAULT_MIGRATION_COST
|
|
|
|
#else
|
|
|
|
-1LL
|
|
|
|
#endif
|
|
|
|
};
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Allow override of migration cost - in units of microseconds.
|
|
|
|
* E.g. migration_cost=1000,2000,3000 will set up a level-1 cost
|
|
|
|
* of 1 msec, level-2 cost of 2 msecs and level3 cost of 3 msecs:
|
|
|
|
*/
|
|
|
|
static int __init migration_cost_setup(char *str)
|
|
|
|
{
|
|
|
|
int ints[MAX_DOMAIN_DISTANCE+1], i;
|
|
|
|
|
|
|
|
str = get_options(str, ARRAY_SIZE(ints), ints);
|
|
|
|
|
|
|
|
printk("#ints: %d\n", ints[0]);
|
|
|
|
for (i = 1; i <= ints[0]; i++) {
|
|
|
|
migration_cost[i-1] = (unsigned long long)ints[i]*1000;
|
|
|
|
printk("migration_cost[%d]: %Ld\n", i-1, migration_cost[i-1]);
|
|
|
|
}
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
__setup ("migration_cost=", migration_cost_setup);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Global multiplier (divisor) for migration-cutoff values,
|
|
|
|
* in percentiles. E.g. use a value of 150 to get 1.5 times
|
|
|
|
* longer cache-hot cutoff times.
|
|
|
|
*
|
|
|
|
* (We scale it from 100 to 128 to long long handling easier.)
|
|
|
|
*/
|
|
|
|
|
|
|
|
#define MIGRATION_FACTOR_SCALE 128
|
|
|
|
|
|
|
|
static unsigned int migration_factor = MIGRATION_FACTOR_SCALE;
|
|
|
|
|
|
|
|
static int __init setup_migration_factor(char *str)
|
|
|
|
{
|
|
|
|
get_option(&str, &migration_factor);
|
|
|
|
migration_factor = migration_factor * MIGRATION_FACTOR_SCALE / 100;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
__setup("migration_factor=", setup_migration_factor);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Estimated distance of two CPUs, measured via the number of domains
|
|
|
|
* we have to pass for the two CPUs to be in the same span:
|
|
|
|
*/
|
|
|
|
static unsigned long domain_distance(int cpu1, int cpu2)
|
|
|
|
{
|
|
|
|
unsigned long distance = 0;
|
|
|
|
struct sched_domain *sd;
|
|
|
|
|
|
|
|
for_each_domain(cpu1, sd) {
|
|
|
|
WARN_ON(!cpu_isset(cpu1, sd->span));
|
|
|
|
if (cpu_isset(cpu2, sd->span))
|
|
|
|
return distance;
|
|
|
|
distance++;
|
|
|
|
}
|
|
|
|
if (distance >= MAX_DOMAIN_DISTANCE) {
|
|
|
|
WARN_ON(1);
|
|
|
|
distance = MAX_DOMAIN_DISTANCE-1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return distance;
|
|
|
|
}
|
|
|
|
|
|
|
|
static unsigned int migration_debug;
|
|
|
|
|
|
|
|
static int __init setup_migration_debug(char *str)
|
|
|
|
{
|
|
|
|
get_option(&str, &migration_debug);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
__setup("migration_debug=", setup_migration_debug);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Maximum cache-size that the scheduler should try to measure.
|
|
|
|
* Architectures with larger caches should tune this up during
|
|
|
|
* bootup. Gets used in the domain-setup code (i.e. during SMP
|
|
|
|
* bootup).
|
|
|
|
*/
|
|
|
|
unsigned int max_cache_size;
|
|
|
|
|
|
|
|
static int __init setup_max_cache_size(char *str)
|
|
|
|
{
|
|
|
|
get_option(&str, &max_cache_size);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
__setup("max_cache_size=", setup_max_cache_size);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Dirty a big buffer in a hard-to-predict (for the L2 cache) way. This
|
|
|
|
* is the operation that is timed, so we try to generate unpredictable
|
|
|
|
* cachemisses that still end up filling the L2 cache:
|
|
|
|
*/
|
|
|
|
static void touch_cache(void *__cache, unsigned long __size)
|
|
|
|
{
|
|
|
|
unsigned long size = __size/sizeof(long), chunk1 = size/3,
|
|
|
|
chunk2 = 2*size/3;
|
|
|
|
unsigned long *cache = __cache;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < size/6; i += 8) {
|
|
|
|
switch (i % 6) {
|
|
|
|
case 0: cache[i]++;
|
|
|
|
case 1: cache[size-1-i]++;
|
|
|
|
case 2: cache[chunk1-i]++;
|
|
|
|
case 3: cache[chunk1+i]++;
|
|
|
|
case 4: cache[chunk2-i]++;
|
|
|
|
case 5: cache[chunk2+i]++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Measure the cache-cost of one task migration. Returns in units of nsec.
|
|
|
|
*/
|
2006-07-03 01:25:40 -06:00
|
|
|
static unsigned long long
|
|
|
|
measure_one(void *cache, unsigned long size, int source, int target)
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
{
|
|
|
|
cpumask_t mask, saved_mask;
|
|
|
|
unsigned long long t0, t1, t2, t3, cost;
|
|
|
|
|
|
|
|
saved_mask = current->cpus_allowed;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Flush source caches to RAM and invalidate them:
|
|
|
|
*/
|
|
|
|
sched_cacheflush();
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Migrate to the source CPU:
|
|
|
|
*/
|
|
|
|
mask = cpumask_of_cpu(source);
|
|
|
|
set_cpus_allowed(current, mask);
|
|
|
|
WARN_ON(smp_processor_id() != source);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Dirty the working set:
|
|
|
|
*/
|
|
|
|
t0 = sched_clock();
|
|
|
|
touch_cache(cache, size);
|
|
|
|
t1 = sched_clock();
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Migrate to the target CPU, dirty the L2 cache and access
|
|
|
|
* the shared buffer. (which represents the working set
|
|
|
|
* of a migrated task.)
|
|
|
|
*/
|
|
|
|
mask = cpumask_of_cpu(target);
|
|
|
|
set_cpus_allowed(current, mask);
|
|
|
|
WARN_ON(smp_processor_id() != target);
|
|
|
|
|
|
|
|
t2 = sched_clock();
|
|
|
|
touch_cache(cache, size);
|
|
|
|
t3 = sched_clock();
|
|
|
|
|
|
|
|
cost = t1-t0 + t3-t2;
|
|
|
|
|
|
|
|
if (migration_debug >= 2)
|
|
|
|
printk("[%d->%d]: %8Ld %8Ld %8Ld => %10Ld.\n",
|
|
|
|
source, target, t1-t0, t1-t0, t3-t2, cost);
|
|
|
|
/*
|
|
|
|
* Flush target caches to RAM and invalidate them:
|
|
|
|
*/
|
|
|
|
sched_cacheflush();
|
|
|
|
|
|
|
|
set_cpus_allowed(current, saved_mask);
|
|
|
|
|
|
|
|
return cost;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Measure a series of task migrations and return the average
|
|
|
|
* result. Since this code runs early during bootup the system
|
|
|
|
* is 'undisturbed' and the average latency makes sense.
|
|
|
|
*
|
|
|
|
* The algorithm in essence auto-detects the relevant cache-size,
|
|
|
|
* so it will properly detect different cachesizes for different
|
|
|
|
* cache-hierarchies, depending on how the CPUs are connected.
|
|
|
|
*
|
|
|
|
* Architectures can prime the upper limit of the search range via
|
|
|
|
* max_cache_size, otherwise the search range defaults to 20MB...64K.
|
|
|
|
*/
|
|
|
|
static unsigned long long
|
|
|
|
measure_cost(int cpu1, int cpu2, void *cache, unsigned int size)
|
|
|
|
{
|
|
|
|
unsigned long long cost1, cost2;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Measure the migration cost of 'size' bytes, over an
|
|
|
|
* average of 10 runs:
|
|
|
|
*
|
|
|
|
* (We perturb the cache size by a small (0..4k)
|
|
|
|
* value to compensate size/alignment related artifacts.
|
|
|
|
* We also subtract the cost of the operation done on
|
|
|
|
* the same CPU.)
|
|
|
|
*/
|
|
|
|
cost1 = 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* dry run, to make sure we start off cache-cold on cpu1,
|
|
|
|
* and to get any vmalloc pagefaults in advance:
|
|
|
|
*/
|
|
|
|
measure_one(cache, size, cpu1, cpu2);
|
|
|
|
for (i = 0; i < ITERATIONS; i++)
|
|
|
|
cost1 += measure_one(cache, size - i*1024, cpu1, cpu2);
|
|
|
|
|
|
|
|
measure_one(cache, size, cpu2, cpu1);
|
|
|
|
for (i = 0; i < ITERATIONS; i++)
|
|
|
|
cost1 += measure_one(cache, size - i*1024, cpu2, cpu1);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* (We measure the non-migrating [cached] cost on both
|
|
|
|
* cpu1 and cpu2, to handle CPUs with different speeds)
|
|
|
|
*/
|
|
|
|
cost2 = 0;
|
|
|
|
|
|
|
|
measure_one(cache, size, cpu1, cpu1);
|
|
|
|
for (i = 0; i < ITERATIONS; i++)
|
|
|
|
cost2 += measure_one(cache, size - i*1024, cpu1, cpu1);
|
|
|
|
|
|
|
|
measure_one(cache, size, cpu2, cpu2);
|
|
|
|
for (i = 0; i < ITERATIONS; i++)
|
|
|
|
cost2 += measure_one(cache, size - i*1024, cpu2, cpu2);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Get the per-iteration migration cost:
|
|
|
|
*/
|
|
|
|
do_div(cost1, 2*ITERATIONS);
|
|
|
|
do_div(cost2, 2*ITERATIONS);
|
|
|
|
|
|
|
|
return cost1 - cost2;
|
|
|
|
}
|
|
|
|
|
|
|
|
static unsigned long long measure_migration_cost(int cpu1, int cpu2)
|
|
|
|
{
|
|
|
|
unsigned long long max_cost = 0, fluct = 0, avg_fluct = 0;
|
|
|
|
unsigned int max_size, size, size_found = 0;
|
|
|
|
long long cost = 0, prev_cost;
|
|
|
|
void *cache;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Search from max_cache_size*5 down to 64K - the real relevant
|
|
|
|
* cachesize has to lie somewhere inbetween.
|
|
|
|
*/
|
|
|
|
if (max_cache_size) {
|
|
|
|
max_size = max(max_cache_size * SEARCH_SCOPE, MIN_CACHE_SIZE);
|
|
|
|
size = max(max_cache_size / SEARCH_SCOPE, MIN_CACHE_SIZE);
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Since we have no estimation about the relevant
|
|
|
|
* search range
|
|
|
|
*/
|
|
|
|
max_size = DEFAULT_CACHE_SIZE * SEARCH_SCOPE;
|
|
|
|
size = MIN_CACHE_SIZE;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!cpu_online(cpu1) || !cpu_online(cpu2)) {
|
|
|
|
printk("cpu %d and %d not both online!\n", cpu1, cpu2);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Allocate the working set:
|
|
|
|
*/
|
|
|
|
cache = vmalloc(max_size);
|
|
|
|
if (!cache) {
|
|
|
|
printk("could not vmalloc %d bytes for cache!\n", 2*max_size);
|
2006-07-10 05:43:52 -06:00
|
|
|
return 1000000; /* return 1 msec on very small boxen */
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
while (size <= max_size) {
|
|
|
|
prev_cost = cost;
|
|
|
|
cost = measure_cost(cpu1, cpu2, cache, size);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Update the max:
|
|
|
|
*/
|
|
|
|
if (cost > 0) {
|
|
|
|
if (max_cost < cost) {
|
|
|
|
max_cost = cost;
|
|
|
|
size_found = size;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* Calculate average fluctuation, we use this to prevent
|
|
|
|
* noise from triggering an early break out of the loop:
|
|
|
|
*/
|
|
|
|
fluct = abs(cost - prev_cost);
|
|
|
|
avg_fluct = (avg_fluct + fluct)/2;
|
|
|
|
|
|
|
|
if (migration_debug)
|
|
|
|
printk("-> [%d][%d][%7d] %3ld.%ld [%3ld.%ld] (%ld): (%8Ld %8Ld)\n",
|
|
|
|
cpu1, cpu2, size,
|
|
|
|
(long)cost / 1000000,
|
|
|
|
((long)cost / 100000) % 10,
|
|
|
|
(long)max_cost / 1000000,
|
|
|
|
((long)max_cost / 100000) % 10,
|
|
|
|
domain_distance(cpu1, cpu2),
|
|
|
|
cost, avg_fluct);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we iterated at least 20% past the previous maximum,
|
|
|
|
* and the cost has dropped by more than 20% already,
|
|
|
|
* (taking fluctuations into account) then we assume to
|
|
|
|
* have found the maximum and break out of the loop early:
|
|
|
|
*/
|
|
|
|
if (size_found && (size*100 > size_found*SIZE_THRESH))
|
|
|
|
if (cost+avg_fluct <= 0 ||
|
|
|
|
max_cost*100 > (cost+avg_fluct)*COST_THRESH) {
|
|
|
|
|
|
|
|
if (migration_debug)
|
|
|
|
printk("-> found max.\n");
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
/*
|
2006-01-30 12:24:38 -07:00
|
|
|
* Increase the cachesize in 10% steps:
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
*/
|
2006-01-30 12:24:38 -07:00
|
|
|
size = size * 10 / 9;
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
}
|
|
|
|
|
|
|
|
if (migration_debug)
|
|
|
|
printk("[%d][%d] working set size found: %d, cost: %Ld\n",
|
|
|
|
cpu1, cpu2, size_found, max_cost);
|
|
|
|
|
|
|
|
vfree(cache);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* A task is considered 'cache cold' if at least 2 times
|
|
|
|
* the worst-case cost of migration has passed.
|
|
|
|
*
|
|
|
|
* (this limit is only listened to if the load-balancing
|
|
|
|
* situation is 'nice' - if there is a large imbalance we
|
|
|
|
* ignore it for the sake of CPU utilization and
|
|
|
|
* processing fairness.)
|
|
|
|
*/
|
|
|
|
return 2 * max_cost * migration_factor / MIGRATION_FACTOR_SCALE;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void calibrate_migration_costs(const cpumask_t *cpu_map)
|
|
|
|
{
|
|
|
|
int cpu1 = -1, cpu2 = -1, cpu, orig_cpu = raw_smp_processor_id();
|
|
|
|
unsigned long j0, j1, distance, max_distance = 0;
|
|
|
|
struct sched_domain *sd;
|
|
|
|
|
|
|
|
j0 = jiffies;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* First pass - calculate the cacheflush times:
|
|
|
|
*/
|
|
|
|
for_each_cpu_mask(cpu1, *cpu_map) {
|
|
|
|
for_each_cpu_mask(cpu2, *cpu_map) {
|
|
|
|
if (cpu1 == cpu2)
|
|
|
|
continue;
|
|
|
|
distance = domain_distance(cpu1, cpu2);
|
|
|
|
max_distance = max(max_distance, distance);
|
|
|
|
/*
|
|
|
|
* No result cached yet?
|
|
|
|
*/
|
|
|
|
if (migration_cost[distance] == -1LL)
|
|
|
|
migration_cost[distance] =
|
|
|
|
measure_migration_cost(cpu1, cpu2);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* Second pass - update the sched domain hierarchy with
|
|
|
|
* the new cache-hot-time estimations:
|
|
|
|
*/
|
|
|
|
for_each_cpu_mask(cpu, *cpu_map) {
|
|
|
|
distance = 0;
|
|
|
|
for_each_domain(cpu, sd) {
|
|
|
|
sd->cache_hot_time = migration_cost[distance];
|
|
|
|
distance++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/*
|
|
|
|
* Print the matrix:
|
|
|
|
*/
|
|
|
|
if (migration_debug)
|
|
|
|
printk("migration: max_cache_size: %d, cpu: %d MHz:\n",
|
|
|
|
max_cache_size,
|
|
|
|
#ifdef CONFIG_X86
|
|
|
|
cpu_khz/1000
|
|
|
|
#else
|
|
|
|
-1
|
|
|
|
#endif
|
|
|
|
);
|
2006-02-05 00:27:42 -07:00
|
|
|
if (system_state == SYSTEM_BOOTING) {
|
|
|
|
printk("migration_cost=");
|
|
|
|
for (distance = 0; distance <= max_distance; distance++) {
|
|
|
|
if (distance)
|
|
|
|
printk(",");
|
|
|
|
printk("%ld", (long)migration_cost[distance] / 1000);
|
|
|
|
}
|
|
|
|
printk("\n");
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
}
|
|
|
|
j1 = jiffies;
|
|
|
|
if (migration_debug)
|
|
|
|
printk("migration: %ld seconds\n", (j1-j0)/HZ);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Move back to the original CPU. NUMA-Q gets confused
|
|
|
|
* if we migrate to another quad during bootup.
|
|
|
|
*/
|
|
|
|
if (raw_smp_processor_id() != orig_cpu) {
|
|
|
|
cpumask_t mask = cpumask_of_cpu(orig_cpu),
|
|
|
|
saved_mask = current->cpus_allowed;
|
|
|
|
|
|
|
|
set_cpus_allowed(current, mask);
|
|
|
|
set_cpus_allowed(current, saved_mask);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-09-06 16:18:14 -06:00
|
|
|
#ifdef CONFIG_NUMA
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
|
2005-09-06 16:18:14 -06:00
|
|
|
/**
|
|
|
|
* find_next_best_node - find the next node to include in a sched_domain
|
|
|
|
* @node: node whose sched_domain we're building
|
|
|
|
* @used_nodes: nodes already in the sched_domain
|
|
|
|
*
|
|
|
|
* Find the next node to include in a given scheduling domain. Simply
|
|
|
|
* finds the closest node not already in the @used_nodes map.
|
|
|
|
*
|
|
|
|
* Should use nodemask_t.
|
|
|
|
*/
|
|
|
|
static int find_next_best_node(int node, unsigned long *used_nodes)
|
|
|
|
{
|
|
|
|
int i, n, val, min_val, best_node = 0;
|
|
|
|
|
|
|
|
min_val = INT_MAX;
|
|
|
|
|
|
|
|
for (i = 0; i < MAX_NUMNODES; i++) {
|
|
|
|
/* Start at @node */
|
|
|
|
n = (node + i) % MAX_NUMNODES;
|
|
|
|
|
|
|
|
if (!nr_cpus_node(n))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/* Skip already used nodes */
|
|
|
|
if (test_bit(n, used_nodes))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/* Simple min distance search */
|
|
|
|
val = node_distance(node, n);
|
|
|
|
|
|
|
|
if (val < min_val) {
|
|
|
|
min_val = val;
|
|
|
|
best_node = n;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
set_bit(best_node, used_nodes);
|
|
|
|
return best_node;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* sched_domain_node_span - get a cpumask for a node's sched_domain
|
|
|
|
* @node: node whose cpumask we're constructing
|
|
|
|
* @size: number of nodes to include in this span
|
|
|
|
*
|
|
|
|
* Given a node, construct a good cpumask for its sched_domain to span. It
|
|
|
|
* should be one that prevents unnecessary balancing, but also spreads tasks
|
|
|
|
* out optimally.
|
|
|
|
*/
|
|
|
|
static cpumask_t sched_domain_node_span(int node)
|
|
|
|
{
|
|
|
|
DECLARE_BITMAP(used_nodes, MAX_NUMNODES);
|
2006-07-03 01:25:40 -06:00
|
|
|
cpumask_t span, nodemask;
|
|
|
|
int i;
|
2005-09-06 16:18:14 -06:00
|
|
|
|
|
|
|
cpus_clear(span);
|
|
|
|
bitmap_zero(used_nodes, MAX_NUMNODES);
|
|
|
|
|
|
|
|
nodemask = node_to_cpumask(node);
|
|
|
|
cpus_or(span, span, nodemask);
|
|
|
|
set_bit(node, used_nodes);
|
|
|
|
|
|
|
|
for (i = 1; i < SD_NODES_PER_DOMAIN; i++) {
|
|
|
|
int next_node = find_next_best_node(node, used_nodes);
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-09-06 16:18:14 -06:00
|
|
|
nodemask = node_to_cpumask(next_node);
|
|
|
|
cpus_or(span, span, nodemask);
|
|
|
|
}
|
|
|
|
|
|
|
|
return span;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
int sched_smt_power_savings = 0, sched_mc_power_savings = 0;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-09-06 16:18:14 -06:00
|
|
|
/*
|
2006-07-03 01:25:40 -06:00
|
|
|
* SMT sched-domains:
|
2005-09-06 16:18:14 -06:00
|
|
|
*/
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_SCHED_SMT
|
|
|
|
static DEFINE_PER_CPU(struct sched_domain, cpu_domains);
|
|
|
|
static struct sched_group sched_group_cpus[NR_CPUS];
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-06-25 15:57:33 -06:00
|
|
|
static int cpu_to_cpu_group(int cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
return cpu;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2006-07-03 01:25:40 -06:00
|
|
|
/*
|
|
|
|
* multi-core sched-domains:
|
|
|
|
*/
|
2006-03-27 02:15:22 -07:00
|
|
|
#ifdef CONFIG_SCHED_MC
|
|
|
|
static DEFINE_PER_CPU(struct sched_domain, core_domains);
|
2006-06-27 03:54:41 -06:00
|
|
|
static struct sched_group *sched_group_core_bycpu[NR_CPUS];
|
2006-03-27 02:15:22 -07:00
|
|
|
#endif
|
|
|
|
|
|
|
|
#if defined(CONFIG_SCHED_MC) && defined(CONFIG_SCHED_SMT)
|
|
|
|
static int cpu_to_core_group(int cpu)
|
|
|
|
{
|
|
|
|
return first_cpu(cpu_sibling_map[cpu]);
|
|
|
|
}
|
|
|
|
#elif defined(CONFIG_SCHED_MC)
|
|
|
|
static int cpu_to_core_group(int cpu)
|
|
|
|
{
|
|
|
|
return cpu;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
static DEFINE_PER_CPU(struct sched_domain, phys_domains);
|
2006-06-27 03:54:41 -06:00
|
|
|
static struct sched_group *sched_group_phys_bycpu[NR_CPUS];
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-06-25 15:57:33 -06:00
|
|
|
static int cpu_to_phys_group(int cpu)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
#ifdef CONFIG_SCHED_MC
|
2006-03-27 02:15:22 -07:00
|
|
|
cpumask_t mask = cpu_coregroup_map(cpu);
|
|
|
|
return first_cpu(mask);
|
|
|
|
#elif defined(CONFIG_SCHED_SMT)
|
2005-04-16 16:20:36 -06:00
|
|
|
return first_cpu(cpu_sibling_map[cpu]);
|
|
|
|
#else
|
|
|
|
return cpu;
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
/*
|
2005-09-06 16:18:14 -06:00
|
|
|
* The init_sched_build_groups can't handle what we want to do with node
|
|
|
|
* groups, so roll our own. Now each node has its own list of groups which
|
|
|
|
* gets dynamically allocated.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2005-09-06 16:18:14 -06:00
|
|
|
static DEFINE_PER_CPU(struct sched_domain, node_domains);
|
2005-09-06 16:18:14 -06:00
|
|
|
static struct sched_group **sched_group_nodes_bycpu[NR_CPUS];
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-09-06 16:18:14 -06:00
|
|
|
static DEFINE_PER_CPU(struct sched_domain, allnodes_domains);
|
2005-09-06 16:18:14 -06:00
|
|
|
static struct sched_group *sched_group_allnodes_bycpu[NR_CPUS];
|
2005-09-06 16:18:14 -06:00
|
|
|
|
|
|
|
static int cpu_to_allnodes_group(int cpu)
|
|
|
|
{
|
|
|
|
return cpu_to_node(cpu);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2006-03-27 02:15:23 -07:00
|
|
|
static void init_numa_sched_groups_power(struct sched_group *group_head)
|
|
|
|
{
|
|
|
|
struct sched_group *sg = group_head;
|
|
|
|
int j;
|
|
|
|
|
|
|
|
if (!sg)
|
|
|
|
return;
|
|
|
|
next_sg:
|
|
|
|
for_each_cpu_mask(j, sg->cpumask) {
|
|
|
|
struct sched_domain *sd;
|
|
|
|
|
|
|
|
sd = &per_cpu(phys_domains, j);
|
|
|
|
if (j != first_cpu(sd->groups->cpumask)) {
|
|
|
|
/*
|
|
|
|
* Only add "power" once for each
|
|
|
|
* physical package.
|
|
|
|
*/
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
sg->cpu_power += sd->groups->cpu_power;
|
|
|
|
}
|
|
|
|
sg = sg->next;
|
|
|
|
if (sg != group_head)
|
|
|
|
goto next_sg;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
|
|
|
|
2006-06-27 03:54:38 -06:00
|
|
|
/* Free memory allocated for various sched_group structures */
|
|
|
|
static void free_sched_groups(const cpumask_t *cpu_map)
|
|
|
|
{
|
2006-06-27 03:54:41 -06:00
|
|
|
int cpu;
|
2006-06-27 03:54:38 -06:00
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for_each_cpu_mask(cpu, *cpu_map) {
|
|
|
|
struct sched_group *sched_group_allnodes
|
|
|
|
= sched_group_allnodes_bycpu[cpu];
|
|
|
|
struct sched_group **sched_group_nodes
|
|
|
|
= sched_group_nodes_bycpu[cpu];
|
|
|
|
|
|
|
|
if (sched_group_allnodes) {
|
|
|
|
kfree(sched_group_allnodes);
|
|
|
|
sched_group_allnodes_bycpu[cpu] = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!sched_group_nodes)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
for (i = 0; i < MAX_NUMNODES; i++) {
|
|
|
|
cpumask_t nodemask = node_to_cpumask(i);
|
|
|
|
struct sched_group *oldsg, *sg = sched_group_nodes[i];
|
|
|
|
|
|
|
|
cpus_and(nodemask, nodemask, *cpu_map);
|
|
|
|
if (cpus_empty(nodemask))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
if (sg == NULL)
|
|
|
|
continue;
|
|
|
|
sg = sg->next;
|
|
|
|
next_sg:
|
|
|
|
oldsg = sg;
|
|
|
|
sg = sg->next;
|
|
|
|
kfree(oldsg);
|
|
|
|
if (oldsg != sched_group_nodes[i])
|
|
|
|
goto next_sg;
|
|
|
|
}
|
|
|
|
kfree(sched_group_nodes);
|
|
|
|
sched_group_nodes_bycpu[cpu] = NULL;
|
|
|
|
}
|
|
|
|
#endif
|
2006-06-27 03:54:41 -06:00
|
|
|
for_each_cpu_mask(cpu, *cpu_map) {
|
|
|
|
if (sched_group_phys_bycpu[cpu]) {
|
|
|
|
kfree(sched_group_phys_bycpu[cpu]);
|
|
|
|
sched_group_phys_bycpu[cpu] = NULL;
|
|
|
|
}
|
|
|
|
#ifdef CONFIG_SCHED_MC
|
|
|
|
if (sched_group_core_bycpu[cpu]) {
|
|
|
|
kfree(sched_group_core_bycpu[cpu]);
|
|
|
|
sched_group_core_bycpu[cpu] = NULL;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
}
|
2006-06-27 03:54:38 -06:00
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
2005-06-25 15:57:33 -06:00
|
|
|
* Build sched domains for a given set of cpus and attach the sched domains
|
|
|
|
* to the individual cpus
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2006-06-27 03:54:38 -06:00
|
|
|
static int build_sched_domains(const cpumask_t *cpu_map)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
|
|
|
int i;
|
2006-06-27 03:54:41 -06:00
|
|
|
struct sched_group *sched_group_phys = NULL;
|
|
|
|
#ifdef CONFIG_SCHED_MC
|
|
|
|
struct sched_group *sched_group_core = NULL;
|
|
|
|
#endif
|
2005-09-06 16:18:14 -06:00
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
struct sched_group **sched_group_nodes = NULL;
|
|
|
|
struct sched_group *sched_group_allnodes = NULL;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Allocate the per-node list of sched groups
|
|
|
|
*/
|
2006-06-27 03:54:38 -06:00
|
|
|
sched_group_nodes = kzalloc(sizeof(struct sched_group*)*MAX_NUMNODES,
|
2006-06-27 03:54:39 -06:00
|
|
|
GFP_KERNEL);
|
2005-09-06 16:18:14 -06:00
|
|
|
if (!sched_group_nodes) {
|
|
|
|
printk(KERN_WARNING "Can not alloc sched group node list\n");
|
2006-06-27 03:54:38 -06:00
|
|
|
return -ENOMEM;
|
2005-09-06 16:18:14 -06:00
|
|
|
}
|
|
|
|
sched_group_nodes_bycpu[first_cpu(*cpu_map)] = sched_group_nodes;
|
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
/*
|
2005-06-25 15:57:33 -06:00
|
|
|
* Set up domains for cpus specified by the cpu_map.
|
2005-04-16 16:20:36 -06:00
|
|
|
*/
|
2005-06-25 15:57:33 -06:00
|
|
|
for_each_cpu_mask(i, *cpu_map) {
|
2005-04-16 16:20:36 -06:00
|
|
|
int group;
|
|
|
|
struct sched_domain *sd = NULL, *p;
|
|
|
|
cpumask_t nodemask = node_to_cpumask(cpu_to_node(i));
|
|
|
|
|
2005-06-25 15:57:33 -06:00
|
|
|
cpus_and(nodemask, nodemask, *cpu_map);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
2005-09-06 16:18:14 -06:00
|
|
|
if (cpus_weight(*cpu_map)
|
2005-09-06 16:18:14 -06:00
|
|
|
> SD_NODES_PER_DOMAIN*cpus_weight(nodemask)) {
|
2005-09-06 16:18:14 -06:00
|
|
|
if (!sched_group_allnodes) {
|
|
|
|
sched_group_allnodes
|
|
|
|
= kmalloc(sizeof(struct sched_group)
|
|
|
|
* MAX_NUMNODES,
|
|
|
|
GFP_KERNEL);
|
|
|
|
if (!sched_group_allnodes) {
|
|
|
|
printk(KERN_WARNING
|
|
|
|
"Can not alloc allnodes sched group\n");
|
2006-06-27 03:54:38 -06:00
|
|
|
goto error;
|
2005-09-06 16:18:14 -06:00
|
|
|
}
|
|
|
|
sched_group_allnodes_bycpu[i]
|
|
|
|
= sched_group_allnodes;
|
|
|
|
}
|
2005-09-06 16:18:14 -06:00
|
|
|
sd = &per_cpu(allnodes_domains, i);
|
|
|
|
*sd = SD_ALLNODES_INIT;
|
|
|
|
sd->span = *cpu_map;
|
|
|
|
group = cpu_to_allnodes_group(i);
|
|
|
|
sd->groups = &sched_group_allnodes[group];
|
|
|
|
p = sd;
|
|
|
|
} else
|
|
|
|
p = NULL;
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
sd = &per_cpu(node_domains, i);
|
|
|
|
*sd = SD_NODE_INIT;
|
2005-09-06 16:18:14 -06:00
|
|
|
sd->span = sched_domain_node_span(cpu_to_node(i));
|
|
|
|
sd->parent = p;
|
|
|
|
cpus_and(sd->span, sd->span, *cpu_map);
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
|
|
|
|
2006-06-27 03:54:41 -06:00
|
|
|
if (!sched_group_phys) {
|
|
|
|
sched_group_phys
|
|
|
|
= kmalloc(sizeof(struct sched_group) * NR_CPUS,
|
|
|
|
GFP_KERNEL);
|
|
|
|
if (!sched_group_phys) {
|
|
|
|
printk (KERN_WARNING "Can not alloc phys sched"
|
|
|
|
"group\n");
|
|
|
|
goto error;
|
|
|
|
}
|
|
|
|
sched_group_phys_bycpu[i] = sched_group_phys;
|
|
|
|
}
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
p = sd;
|
|
|
|
sd = &per_cpu(phys_domains, i);
|
|
|
|
group = cpu_to_phys_group(i);
|
|
|
|
*sd = SD_CPU_INIT;
|
|
|
|
sd->span = nodemask;
|
|
|
|
sd->parent = p;
|
|
|
|
sd->groups = &sched_group_phys[group];
|
|
|
|
|
2006-03-27 02:15:22 -07:00
|
|
|
#ifdef CONFIG_SCHED_MC
|
2006-06-27 03:54:41 -06:00
|
|
|
if (!sched_group_core) {
|
|
|
|
sched_group_core
|
|
|
|
= kmalloc(sizeof(struct sched_group) * NR_CPUS,
|
|
|
|
GFP_KERNEL);
|
|
|
|
if (!sched_group_core) {
|
|
|
|
printk (KERN_WARNING "Can not alloc core sched"
|
|
|
|
"group\n");
|
|
|
|
goto error;
|
|
|
|
}
|
|
|
|
sched_group_core_bycpu[i] = sched_group_core;
|
|
|
|
}
|
|
|
|
|
2006-03-27 02:15:22 -07:00
|
|
|
p = sd;
|
|
|
|
sd = &per_cpu(core_domains, i);
|
|
|
|
group = cpu_to_core_group(i);
|
|
|
|
*sd = SD_MC_INIT;
|
|
|
|
sd->span = cpu_coregroup_map(i);
|
|
|
|
cpus_and(sd->span, sd->span, *cpu_map);
|
|
|
|
sd->parent = p;
|
|
|
|
sd->groups = &sched_group_core[group];
|
|
|
|
#endif
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_SCHED_SMT
|
|
|
|
p = sd;
|
|
|
|
sd = &per_cpu(cpu_domains, i);
|
|
|
|
group = cpu_to_cpu_group(i);
|
|
|
|
*sd = SD_SIBLING_INIT;
|
|
|
|
sd->span = cpu_sibling_map[i];
|
2005-06-25 15:57:33 -06:00
|
|
|
cpus_and(sd->span, sd->span, *cpu_map);
|
2005-04-16 16:20:36 -06:00
|
|
|
sd->parent = p;
|
|
|
|
sd->groups = &sched_group_cpus[group];
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_SCHED_SMT
|
|
|
|
/* Set up CPU (sibling) groups */
|
2005-09-06 16:18:14 -06:00
|
|
|
for_each_cpu_mask(i, *cpu_map) {
|
2005-04-16 16:20:36 -06:00
|
|
|
cpumask_t this_sibling_map = cpu_sibling_map[i];
|
2005-06-25 15:57:33 -06:00
|
|
|
cpus_and(this_sibling_map, this_sibling_map, *cpu_map);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (i != first_cpu(this_sibling_map))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
init_sched_build_groups(sched_group_cpus, this_sibling_map,
|
|
|
|
&cpu_to_cpu_group);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2006-03-27 02:15:22 -07:00
|
|
|
#ifdef CONFIG_SCHED_MC
|
|
|
|
/* Set up multi-core groups */
|
|
|
|
for_each_cpu_mask(i, *cpu_map) {
|
|
|
|
cpumask_t this_core_map = cpu_coregroup_map(i);
|
|
|
|
cpus_and(this_core_map, this_core_map, *cpu_map);
|
|
|
|
if (i != first_cpu(this_core_map))
|
|
|
|
continue;
|
|
|
|
init_sched_build_groups(sched_group_core, this_core_map,
|
|
|
|
&cpu_to_core_group);
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Set up physical groups */
|
|
|
|
for (i = 0; i < MAX_NUMNODES; i++) {
|
|
|
|
cpumask_t nodemask = node_to_cpumask(i);
|
|
|
|
|
2005-06-25 15:57:33 -06:00
|
|
|
cpus_and(nodemask, nodemask, *cpu_map);
|
2005-04-16 16:20:36 -06:00
|
|
|
if (cpus_empty(nodemask))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
init_sched_build_groups(sched_group_phys, nodemask,
|
|
|
|
&cpu_to_phys_group);
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
/* Set up node groups */
|
2005-09-06 16:18:14 -06:00
|
|
|
if (sched_group_allnodes)
|
|
|
|
init_sched_build_groups(sched_group_allnodes, *cpu_map,
|
|
|
|
&cpu_to_allnodes_group);
|
2005-09-06 16:18:14 -06:00
|
|
|
|
|
|
|
for (i = 0; i < MAX_NUMNODES; i++) {
|
|
|
|
/* Set up node groups */
|
|
|
|
struct sched_group *sg, *prev;
|
|
|
|
cpumask_t nodemask = node_to_cpumask(i);
|
|
|
|
cpumask_t domainspan;
|
|
|
|
cpumask_t covered = CPU_MASK_NONE;
|
|
|
|
int j;
|
|
|
|
|
|
|
|
cpus_and(nodemask, nodemask, *cpu_map);
|
2005-09-06 16:18:14 -06:00
|
|
|
if (cpus_empty(nodemask)) {
|
|
|
|
sched_group_nodes[i] = NULL;
|
2005-09-06 16:18:14 -06:00
|
|
|
continue;
|
2005-09-06 16:18:14 -06:00
|
|
|
}
|
2005-09-06 16:18:14 -06:00
|
|
|
|
|
|
|
domainspan = sched_domain_node_span(i);
|
|
|
|
cpus_and(domainspan, domainspan, *cpu_map);
|
|
|
|
|
2006-06-27 03:54:40 -06:00
|
|
|
sg = kmalloc_node(sizeof(struct sched_group), GFP_KERNEL, i);
|
2006-06-27 03:54:38 -06:00
|
|
|
if (!sg) {
|
|
|
|
printk(KERN_WARNING "Can not alloc domain group for "
|
|
|
|
"node %d\n", i);
|
|
|
|
goto error;
|
|
|
|
}
|
2005-09-06 16:18:14 -06:00
|
|
|
sched_group_nodes[i] = sg;
|
|
|
|
for_each_cpu_mask(j, nodemask) {
|
|
|
|
struct sched_domain *sd;
|
|
|
|
sd = &per_cpu(node_domains, j);
|
|
|
|
sd->groups = sg;
|
|
|
|
}
|
|
|
|
sg->cpu_power = 0;
|
|
|
|
sg->cpumask = nodemask;
|
2006-06-27 03:54:38 -06:00
|
|
|
sg->next = sg;
|
2005-09-06 16:18:14 -06:00
|
|
|
cpus_or(covered, covered, nodemask);
|
|
|
|
prev = sg;
|
|
|
|
|
|
|
|
for (j = 0; j < MAX_NUMNODES; j++) {
|
|
|
|
cpumask_t tmp, notcovered;
|
|
|
|
int n = (i + j) % MAX_NUMNODES;
|
|
|
|
|
|
|
|
cpus_complement(notcovered, covered);
|
|
|
|
cpus_and(tmp, notcovered, *cpu_map);
|
|
|
|
cpus_and(tmp, tmp, domainspan);
|
|
|
|
if (cpus_empty(tmp))
|
|
|
|
break;
|
|
|
|
|
|
|
|
nodemask = node_to_cpumask(n);
|
|
|
|
cpus_and(tmp, tmp, nodemask);
|
|
|
|
if (cpus_empty(tmp))
|
|
|
|
continue;
|
|
|
|
|
2006-06-27 03:54:40 -06:00
|
|
|
sg = kmalloc_node(sizeof(struct sched_group),
|
|
|
|
GFP_KERNEL, i);
|
2005-09-06 16:18:14 -06:00
|
|
|
if (!sg) {
|
|
|
|
printk(KERN_WARNING
|
|
|
|
"Can not alloc domain group for node %d\n", j);
|
2006-06-27 03:54:38 -06:00
|
|
|
goto error;
|
2005-09-06 16:18:14 -06:00
|
|
|
}
|
|
|
|
sg->cpu_power = 0;
|
|
|
|
sg->cpumask = tmp;
|
2006-06-27 03:54:38 -06:00
|
|
|
sg->next = prev->next;
|
2005-09-06 16:18:14 -06:00
|
|
|
cpus_or(covered, covered, tmp);
|
|
|
|
prev->next = sg;
|
|
|
|
prev = sg;
|
|
|
|
}
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
|
|
|
|
|
|
|
/* Calculate CPU power for physical packages and nodes */
|
2006-06-27 03:54:42 -06:00
|
|
|
#ifdef CONFIG_SCHED_SMT
|
2005-06-25 15:57:33 -06:00
|
|
|
for_each_cpu_mask(i, *cpu_map) {
|
2005-04-16 16:20:36 -06:00
|
|
|
struct sched_domain *sd;
|
|
|
|
sd = &per_cpu(cpu_domains, i);
|
2006-06-27 03:54:42 -06:00
|
|
|
sd->groups->cpu_power = SCHED_LOAD_SCALE;
|
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
#endif
|
2006-03-27 02:15:22 -07:00
|
|
|
#ifdef CONFIG_SCHED_MC
|
2006-06-27 03:54:42 -06:00
|
|
|
for_each_cpu_mask(i, *cpu_map) {
|
|
|
|
int power;
|
|
|
|
struct sched_domain *sd;
|
2006-03-27 02:15:22 -07:00
|
|
|
sd = &per_cpu(core_domains, i);
|
2006-06-27 03:54:42 -06:00
|
|
|
if (sched_smt_power_savings)
|
|
|
|
power = SCHED_LOAD_SCALE * cpus_weight(sd->groups->cpumask);
|
|
|
|
else
|
|
|
|
power = SCHED_LOAD_SCALE + (cpus_weight(sd->groups->cpumask)-1)
|
2006-03-27 02:15:22 -07:00
|
|
|
* SCHED_LOAD_SCALE / 10;
|
|
|
|
sd->groups->cpu_power = power;
|
2006-06-27 03:54:42 -06:00
|
|
|
}
|
|
|
|
#endif
|
2006-03-27 02:15:22 -07:00
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
for_each_cpu_mask(i, *cpu_map) {
|
|
|
|
struct sched_domain *sd;
|
|
|
|
#ifdef CONFIG_SCHED_MC
|
2006-03-27 02:15:22 -07:00
|
|
|
sd = &per_cpu(phys_domains, i);
|
2006-06-27 03:54:42 -06:00
|
|
|
if (i != first_cpu(sd->groups->cpumask))
|
|
|
|
continue;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
sd->groups->cpu_power = 0;
|
|
|
|
if (sched_mc_power_savings || sched_smt_power_savings) {
|
|
|
|
int j;
|
|
|
|
|
|
|
|
for_each_cpu_mask(j, sd->groups->cpumask) {
|
|
|
|
struct sched_domain *sd1;
|
|
|
|
sd1 = &per_cpu(core_domains, j);
|
|
|
|
/*
|
|
|
|
* for each core we will add once
|
|
|
|
* to the group in physical domain
|
|
|
|
*/
|
|
|
|
if (j != first_cpu(sd1->groups->cpumask))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
if (sched_smt_power_savings)
|
|
|
|
sd->groups->cpu_power += sd1->groups->cpu_power;
|
|
|
|
else
|
|
|
|
sd->groups->cpu_power += SCHED_LOAD_SCALE;
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
/*
|
|
|
|
* This has to be < 2 * SCHED_LOAD_SCALE
|
|
|
|
* Lets keep it SCHED_LOAD_SCALE, so that
|
|
|
|
* while calculating NUMA group's cpu_power
|
|
|
|
* we can simply do
|
|
|
|
* numa_group->cpu_power += phys_group->cpu_power;
|
|
|
|
*
|
|
|
|
* See "only add power once for each physical pkg"
|
|
|
|
* comment below
|
|
|
|
*/
|
|
|
|
sd->groups->cpu_power = SCHED_LOAD_SCALE;
|
2006-03-27 02:15:22 -07:00
|
|
|
#else
|
2006-06-27 03:54:42 -06:00
|
|
|
int power;
|
2005-04-16 16:20:36 -06:00
|
|
|
sd = &per_cpu(phys_domains, i);
|
2006-06-27 03:54:42 -06:00
|
|
|
if (sched_smt_power_savings)
|
|
|
|
power = SCHED_LOAD_SCALE * cpus_weight(sd->groups->cpumask);
|
|
|
|
else
|
|
|
|
power = SCHED_LOAD_SCALE;
|
2005-04-16 16:20:36 -06:00
|
|
|
sd->groups->cpu_power = power;
|
2006-03-27 02:15:22 -07:00
|
|
|
#endif
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
|
2005-09-06 16:18:14 -06:00
|
|
|
#ifdef CONFIG_NUMA
|
2006-03-27 02:15:23 -07:00
|
|
|
for (i = 0; i < MAX_NUMNODES; i++)
|
|
|
|
init_numa_sched_groups_power(sched_group_nodes[i]);
|
2005-09-06 16:18:14 -06:00
|
|
|
|
2006-07-30 04:02:59 -06:00
|
|
|
if (sched_group_allnodes) {
|
|
|
|
int group = cpu_to_allnodes_group(first_cpu(*cpu_map));
|
|
|
|
struct sched_group *sg = &sched_group_allnodes[group];
|
|
|
|
|
|
|
|
init_numa_sched_groups_power(sg);
|
|
|
|
}
|
2005-09-06 16:18:14 -06:00
|
|
|
#endif
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/* Attach the domains */
|
2005-06-25 15:57:33 -06:00
|
|
|
for_each_cpu_mask(i, *cpu_map) {
|
2005-04-16 16:20:36 -06:00
|
|
|
struct sched_domain *sd;
|
|
|
|
#ifdef CONFIG_SCHED_SMT
|
|
|
|
sd = &per_cpu(cpu_domains, i);
|
2006-03-27 02:15:22 -07:00
|
|
|
#elif defined(CONFIG_SCHED_MC)
|
|
|
|
sd = &per_cpu(core_domains, i);
|
2005-04-16 16:20:36 -06:00
|
|
|
#else
|
|
|
|
sd = &per_cpu(phys_domains, i);
|
|
|
|
#endif
|
|
|
|
cpu_attach_domain(sd, i);
|
|
|
|
}
|
[PATCH] scheduler cache-hot-autodetect
)
From: Ingo Molnar <mingo@elte.hu>
This is the latest version of the scheduler cache-hot-auto-tune patch.
The first problem was that detection time scaled with O(N^2), which is
unacceptable on larger SMP and NUMA systems. To solve this:
- I've added a 'domain distance' function, which is used to cache
measurement results. Each distance is only measured once. This means
that e.g. on NUMA distances of 0, 1 and 2 might be measured, on HT
distances 0 and 1, and on SMP distance 0 is measured. The code walks
the domain tree to determine the distance, so it automatically follows
whatever hierarchy an architecture sets up. This cuts down on the boot
time significantly and removes the O(N^2) limit. The only assumption
is that migration costs can be expressed as a function of domain
distance - this covers the overwhelming majority of existing systems,
and is a good guess even for more assymetric systems.
[ People hacking systems that have assymetries that break this
assumption (e.g. different CPU speeds) should experiment a bit with
the cpu_distance() function. Adding a ->migration_distance factor to
the domain structure would be one possible solution - but lets first
see the problem systems, if they exist at all. Lets not overdesign. ]
Another problem was that only a single cache-size was used for measuring
the cost of migration, and most architectures didnt set that variable
up. Furthermore, a single cache-size does not fit NUMA hierarchies with
L3 caches and does not fit HT setups, where different CPUs will often
have different 'effective cache sizes'. To solve this problem:
- Instead of relying on a single cache-size provided by the platform and
sticking to it, the code now auto-detects the 'effective migration
cost' between two measured CPUs, via iterating through a wide range of
cachesizes. The code searches for the maximum migration cost, which
occurs when the working set of the test-workload falls just below the
'effective cache size'. I.e. real-life optimized search is done for
the maximum migration cost, between two real CPUs.
This, amongst other things, has the positive effect hat if e.g. two
CPUs share a L2/L3 cache, a different (and accurate) migration cost
will be found than between two CPUs on the same system that dont share
any caches.
(The reliable measurement of migration costs is tricky - see the source
for details.)
Furthermore i've added various boot-time options to override/tune
migration behavior.
Firstly, there's a blanket override for autodetection:
migration_cost=1000,2000,3000
will override the depth 0/1/2 values with 1msec/2msec/3msec values.
Secondly, there's a global factor that can be used to increase (or
decrease) the autodetected values:
migration_factor=120
will increase the autodetected values by 20%. This option is useful to
tune things in a workload-dependent way - e.g. if a workload is
cache-insensitive then CPU utilization can be maximized by specifying
migration_factor=0.
I've tested the autodetection code quite extensively on x86, on 3
P3/Xeon/2MB, and the autodetected values look pretty good:
Dual Celeron (128K L2 cache):
---------------------
migration cost matrix (max_cache_size: 131072, cpu: 467 MHz):
---------------------
[00] [01]
[00]: - 1.7(1)
[01]: 1.7(1) -
---------------------
cacheflush times [2]: 0.0 (0) 1.7 (1784008)
---------------------
Here the slow memory subsystem dominates system performance, and even
though caches are small, the migration cost is 1.7 msecs.
Dual HT P4 (512K L2 cache):
---------------------
migration cost matrix (max_cache_size: 524288, cpu: 2379 MHz):
---------------------
[00] [01] [02] [03]
[00]: - 0.4(1) 0.0(0) 0.4(1)
[01]: 0.4(1) - 0.4(1) 0.0(0)
[02]: 0.0(0) 0.4(1) - 0.4(1)
[03]: 0.4(1) 0.0(0) 0.4(1) -
---------------------
cacheflush times [2]: 0.0 (33900) 0.4 (448514)
---------------------
Here it can be seen that there is no migration cost between two HT
siblings (CPU#0/2 and CPU#1/3 are separate physical CPUs). A fast memory
system makes inter-physical-CPU migration pretty cheap: 0.4 msecs.
8-way P3/Xeon [2MB L2 cache]:
---------------------
migration cost matrix (max_cache_size: 2097152, cpu: 700 MHz):
---------------------
[00] [01] [02] [03] [04] [05] [06] [07]
[00]: - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[01]: 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[02]: 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[03]: 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1) 19.2(1)
[04]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1) 19.2(1)
[05]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1) 19.2(1)
[06]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) - 19.2(1)
[07]: 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) 19.2(1) -
---------------------
cacheflush times [2]: 0.0 (0) 19.2 (19281756)
---------------------
This one has huge caches and a relatively slow memory subsystem - so the
migration cost is 19 msecs.
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Ashok Raj <ashok.raj@intel.com>
Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Cc: <wilder@us.ibm.com>
Signed-off-by: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-01-12 02:05:30 -07:00
|
|
|
/*
|
|
|
|
* Tune cache-hot values:
|
|
|
|
*/
|
|
|
|
calibrate_migration_costs(cpu_map);
|
2006-06-27 03:54:38 -06:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
error:
|
|
|
|
free_sched_groups(cpu_map);
|
|
|
|
return -ENOMEM;
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
2005-06-25 15:57:33 -06:00
|
|
|
/*
|
|
|
|
* Set up scheduler domains and groups. Callers must hold the hotplug lock.
|
|
|
|
*/
|
2006-06-27 03:54:38 -06:00
|
|
|
static int arch_init_sched_domains(const cpumask_t *cpu_map)
|
2005-06-25 15:57:33 -06:00
|
|
|
{
|
|
|
|
cpumask_t cpu_default_map;
|
2006-06-27 03:54:38 -06:00
|
|
|
int err;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-06-25 15:57:33 -06:00
|
|
|
/*
|
|
|
|
* Setup mask for cpus without special case scheduling requirements.
|
|
|
|
* For now this just excludes isolated cpus, but could be used to
|
|
|
|
* exclude other special cases in the future.
|
|
|
|
*/
|
|
|
|
cpus_andnot(cpu_default_map, *cpu_map, cpu_isolated_map);
|
|
|
|
|
2006-06-27 03:54:38 -06:00
|
|
|
err = build_sched_domains(&cpu_default_map);
|
|
|
|
|
|
|
|
return err;
|
2005-06-25 15:57:33 -06:00
|
|
|
}
|
|
|
|
|
|
|
|
static void arch_destroy_sched_domains(const cpumask_t *cpu_map)
|
2005-04-16 16:20:36 -06:00
|
|
|
{
|
2006-06-27 03:54:38 -06:00
|
|
|
free_sched_groups(cpu_map);
|
2005-09-06 16:18:14 -06:00
|
|
|
}
|
2005-04-16 16:20:36 -06:00
|
|
|
|
2005-06-25 15:57:33 -06:00
|
|
|
/*
|
|
|
|
* Detach sched domains from a group of cpus specified in cpu_map
|
|
|
|
* These cpus will now be attached to the NULL domain
|
|
|
|
*/
|
2006-01-14 14:20:43 -07:00
|
|
|
static void detach_destroy_domains(const cpumask_t *cpu_map)
|
2005-06-25 15:57:33 -06:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for_each_cpu_mask(i, *cpu_map)
|
|
|
|
cpu_attach_domain(NULL, i);
|
|
|
|
synchronize_sched();
|
|
|
|
arch_destroy_sched_domains(cpu_map);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Partition sched domains as specified by the cpumasks below.
|
|
|
|
* This attaches all cpus from the cpumasks to the NULL domain,
|
|
|
|
* waits for a RCU quiescent period, recalculates sched
|
|
|
|
* domain information and then attaches them back to the
|
|
|
|
* correct sched domains
|
|
|
|
* Call with hotplug lock held
|
|
|
|
*/
|
2006-06-27 03:54:38 -06:00
|
|
|
int partition_sched_domains(cpumask_t *partition1, cpumask_t *partition2)
|
2005-06-25 15:57:33 -06:00
|
|
|
{
|
|
|
|
cpumask_t change_map;
|
2006-06-27 03:54:38 -06:00
|
|
|
int err = 0;
|
2005-06-25 15:57:33 -06:00
|
|
|
|
|
|
|
cpus_and(*partition1, *partition1, cpu_online_map);
|
|
|
|
cpus_and(*partition2, *partition2, cpu_online_map);
|
|
|
|
cpus_or(change_map, *partition1, *partition2);
|
|
|
|
|
|
|
|
/* Detach sched domains from all of the affected cpus */
|
|
|
|
detach_destroy_domains(&change_map);
|
|
|
|
if (!cpus_empty(*partition1))
|
2006-06-27 03:54:38 -06:00
|
|
|
err = build_sched_domains(partition1);
|
|
|
|
if (!err && !cpus_empty(*partition2))
|
|
|
|
err = build_sched_domains(partition2);
|
|
|
|
|
|
|
|
return err;
|
2005-06-25 15:57:33 -06:00
|
|
|
}
|
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
#if defined(CONFIG_SCHED_MC) || defined(CONFIG_SCHED_SMT)
|
|
|
|
int arch_reinit_sched_domains(void)
|
|
|
|
{
|
|
|
|
int err;
|
|
|
|
|
|
|
|
lock_cpu_hotplug();
|
|
|
|
detach_destroy_domains(&cpu_online_map);
|
|
|
|
err = arch_init_sched_domains(&cpu_online_map);
|
|
|
|
unlock_cpu_hotplug();
|
|
|
|
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
static ssize_t sched_power_savings_store(const char *buf, size_t count, int smt)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (buf[0] != '0' && buf[0] != '1')
|
|
|
|
return -EINVAL;
|
|
|
|
|
|
|
|
if (smt)
|
|
|
|
sched_smt_power_savings = (buf[0] == '1');
|
|
|
|
else
|
|
|
|
sched_mc_power_savings = (buf[0] == '1');
|
|
|
|
|
|
|
|
ret = arch_reinit_sched_domains();
|
|
|
|
|
|
|
|
return ret ? ret : count;
|
|
|
|
}
|
|
|
|
|
|
|
|
int sched_create_sysfs_power_savings_entries(struct sysdev_class *cls)
|
|
|
|
{
|
|
|
|
int err = 0;
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2006-06-27 03:54:42 -06:00
|
|
|
#ifdef CONFIG_SCHED_SMT
|
|
|
|
if (smt_capable())
|
|
|
|
err = sysfs_create_file(&cls->kset.kobj,
|
|
|
|
&attr_sched_smt_power_savings.attr);
|
|
|
|
#endif
|
|
|
|
#ifdef CONFIG_SCHED_MC
|
|
|
|
if (!err && mc_capable())
|
|
|
|
err = sysfs_create_file(&cls->kset.kobj,
|
|
|
|
&attr_sched_mc_power_savings.attr);
|
|
|
|
#endif
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#ifdef CONFIG_SCHED_MC
|
|
|
|
static ssize_t sched_mc_power_savings_show(struct sys_device *dev, char *page)
|
|
|
|
{
|
|
|
|
return sprintf(page, "%u\n", sched_mc_power_savings);
|
|
|
|
}
|
2006-07-03 01:25:40 -06:00
|
|
|
static ssize_t sched_mc_power_savings_store(struct sys_device *dev,
|
|
|
|
const char *buf, size_t count)
|
2006-06-27 03:54:42 -06:00
|
|
|
{
|
|
|
|
return sched_power_savings_store(buf, count, 0);
|
|
|
|
}
|
|
|
|
SYSDEV_ATTR(sched_mc_power_savings, 0644, sched_mc_power_savings_show,
|
|
|
|
sched_mc_power_savings_store);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#ifdef CONFIG_SCHED_SMT
|
|
|
|
static ssize_t sched_smt_power_savings_show(struct sys_device *dev, char *page)
|
|
|
|
{
|
|
|
|
return sprintf(page, "%u\n", sched_smt_power_savings);
|
|
|
|
}
|
2006-07-03 01:25:40 -06:00
|
|
|
static ssize_t sched_smt_power_savings_store(struct sys_device *dev,
|
|
|
|
const char *buf, size_t count)
|
2006-06-27 03:54:42 -06:00
|
|
|
{
|
|
|
|
return sched_power_savings_store(buf, count, 1);
|
|
|
|
}
|
|
|
|
SYSDEV_ATTR(sched_smt_power_savings, 0644, sched_smt_power_savings_show,
|
|
|
|
sched_smt_power_savings_store);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
#ifdef CONFIG_HOTPLUG_CPU
|
|
|
|
/*
|
|
|
|
* Force a reinitialization of the sched domains hierarchy. The domains
|
|
|
|
* and groups cannot be updated in place without racing with the balancing
|
2005-06-25 15:57:24 -06:00
|
|
|
* code, so we temporarily attach all running cpus to the NULL domain
|
2005-04-16 16:20:36 -06:00
|
|
|
* which will prevent rebalancing while the sched domains are recalculated.
|
|
|
|
*/
|
|
|
|
static int update_sched_domains(struct notifier_block *nfb,
|
|
|
|
unsigned long action, void *hcpu)
|
|
|
|
{
|
|
|
|
switch (action) {
|
|
|
|
case CPU_UP_PREPARE:
|
|
|
|
case CPU_DOWN_PREPARE:
|
2005-06-25 15:57:33 -06:00
|
|
|
detach_destroy_domains(&cpu_online_map);
|
2005-04-16 16:20:36 -06:00
|
|
|
return NOTIFY_OK;
|
|
|
|
|
|
|
|
case CPU_UP_CANCELED:
|
|
|
|
case CPU_DOWN_FAILED:
|
|
|
|
case CPU_ONLINE:
|
|
|
|
case CPU_DEAD:
|
|
|
|
/*
|
|
|
|
* Fall through and re-initialise the domains.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
return NOTIFY_DONE;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* The hotplug lock is already held by cpu_up/cpu_down */
|
2005-06-25 15:57:33 -06:00
|
|
|
arch_init_sched_domains(&cpu_online_map);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
return NOTIFY_OK;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
void __init sched_init_smp(void)
|
|
|
|
{
|
|
|
|
lock_cpu_hotplug();
|
2005-06-25 15:57:33 -06:00
|
|
|
arch_init_sched_domains(&cpu_online_map);
|
2005-04-16 16:20:36 -06:00
|
|
|
unlock_cpu_hotplug();
|
|
|
|
/* XXX: Theoretical race here - CPU may be hotplugged now */
|
|
|
|
hotcpu_notifier(update_sched_domains, 0);
|
|
|
|
}
|
|
|
|
#else
|
|
|
|
void __init sched_init_smp(void)
|
|
|
|
{
|
|
|
|
}
|
|
|
|
#endif /* CONFIG_SMP */
|
|
|
|
|
|
|
|
int in_sched_functions(unsigned long addr)
|
|
|
|
{
|
|
|
|
/* Linker adds these: start and end of __sched functions */
|
|
|
|
extern char __sched_text_start[], __sched_text_end[];
|
2006-07-03 01:25:40 -06:00
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
return in_lock_functions(addr) ||
|
|
|
|
(addr >= (unsigned long)__sched_text_start
|
|
|
|
&& addr < (unsigned long)__sched_text_end);
|
|
|
|
}
|
|
|
|
|
|
|
|
void __init sched_init(void)
|
|
|
|
{
|
|
|
|
int i, j, k;
|
|
|
|
|
2006-03-28 02:56:37 -07:00
|
|
|
for_each_possible_cpu(i) {
|
2006-07-03 01:25:42 -06:00
|
|
|
struct prio_array *array;
|
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
rq = cpu_rq(i);
|
|
|
|
spin_lock_init(&rq->lock);
|
2006-07-03 01:25:10 -06:00
|
|
|
lockdep_set_class(&rq->lock, &rq->rq_lock_key);
|
2005-06-25 15:57:13 -06:00
|
|
|
rq->nr_running = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
rq->active = rq->arrays;
|
|
|
|
rq->expired = rq->arrays + 1;
|
|
|
|
rq->best_expired_prio = MAX_PRIO;
|
|
|
|
|
|
|
|
#ifdef CONFIG_SMP
|
2005-06-25 15:57:24 -06:00
|
|
|
rq->sd = NULL;
|
2005-06-25 15:57:13 -06:00
|
|
|
for (j = 1; j < 3; j++)
|
|
|
|
rq->cpu_load[j] = 0;
|
2005-04-16 16:20:36 -06:00
|
|
|
rq->active_balance = 0;
|
|
|
|
rq->push_cpu = 0;
|
|
|
|
rq->migration_thread = NULL;
|
|
|
|
INIT_LIST_HEAD(&rq->migration_queue);
|
|
|
|
#endif
|
|
|
|
atomic_set(&rq->nr_iowait, 0);
|
|
|
|
|
|
|
|
for (j = 0; j < 2; j++) {
|
|
|
|
array = rq->arrays + j;
|
|
|
|
for (k = 0; k < MAX_PRIO; k++) {
|
|
|
|
INIT_LIST_HEAD(array->queue + k);
|
|
|
|
__clear_bit(k, array->bitmap);
|
|
|
|
}
|
|
|
|
// delimiter for bitsearch
|
|
|
|
__set_bit(MAX_PRIO, array->bitmap);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[PATCH] sched: implement smpnice
Problem:
The introduction of separate run queues per CPU has brought with it "nice"
enforcement problems that are best described by a simple example.
For the sake of argument suppose that on a single CPU machine with a
nice==19 hard spinner and a nice==0 hard spinner running that the nice==0
task gets 95% of the CPU and the nice==19 task gets 5% of the CPU. Now
suppose that there is a system with 2 CPUs and 2 nice==19 hard spinners and
2 nice==0 hard spinners running. The user of this system would be entitled
to expect that the nice==0 tasks each get 95% of a CPU and the nice==19
tasks only get 5% each. However, whether this expectation is met is pretty
much down to luck as there are four equally likely distributions of the
tasks to the CPUs that the load balancing code will consider to be balanced
with loads of 2.0 for each CPU. Two of these distributions involve one
nice==0 and one nice==19 task per CPU and in these circumstances the users
expectations will be met. The other two distributions both involve both
nice==0 tasks being on one CPU and both nice==19 being on the other CPU and
each task will get 50% of a CPU and the user's expectations will not be
met.
Solution:
The solution to this problem that is implemented in the attached patch is
to use weighted loads when determining if the system is balanced and, when
an imbalance is detected, to move an amount of weighted load between run
queues (as opposed to a number of tasks) to restore the balance. Once
again, the easiest way to explain why both of these measures are necessary
is to use a simple example. Suppose that (in a slight variation of the
above example) that we have a two CPU system with 4 nice==0 and 4 nice=19
hard spinning tasks running and that the 4 nice==0 tasks are on one CPU and
the 4 nice==19 tasks are on the other CPU. The weighted loads for the two
CPUs would be 4.0 and 0.2 respectively and the load balancing code would
move 2 tasks resulting in one CPU with a load of 2.0 and the other with
load of 2.2. If this was considered to be a big enough imbalance to
justify moving a task and that task was moved using the current
move_tasks() then it would move the highest priority task that it found and
this would result in one CPU with a load of 3.0 and the other with a load
of 1.2 which would result in the movement of a task in the opposite
direction and so on -- infinite loop. If, on the other hand, an amount of
load to be moved is calculated from the imbalance (in this case 0.1) and
move_tasks() skips tasks until it find ones whose contributions to the
weighted load are less than this amount it would move two of the nice==19
tasks resulting in a system with 2 nice==0 and 2 nice=19 on each CPU with
loads of 2.1 for each CPU.
One of the advantages of this mechanism is that on a system where all tasks
have nice==0 the load balancing calculations would be mathematically
identical to the current load balancing code.
Notes:
struct task_struct:
has a new field load_weight which (in a trade off of space for speed)
stores the contribution that this task makes to a CPU's weighted load when
it is runnable.
struct runqueue:
has a new field raw_weighted_load which is the sum of the load_weight
values for the currently runnable tasks on this run queue. This field
always needs to be updated when nr_running is updated so two new inline
functions inc_nr_running() and dec_nr_running() have been created to make
sure that this happens. This also offers a convenient way to optimize away
this part of the smpnice mechanism when CONFIG_SMP is not defined.
int try_to_wake_up():
in this function the value SCHED_LOAD_BALANCE is used to represent the load
contribution of a single task in various calculations in the code that
decides which CPU to put the waking task on. While this would be a valid
on a system where the nice values for the runnable tasks were distributed
evenly around zero it will lead to anomalous load balancing if the
distribution is skewed in either direction. To overcome this problem
SCHED_LOAD_SCALE has been replaced by the load_weight for the relevant task
or by the average load_weight per task for the queue in question (as
appropriate).
int move_tasks():
The modifications to this function were complicated by the fact that
active_load_balance() uses it to move exactly one task without checking
whether an imbalance actually exists. This precluded the simple
overloading of max_nr_move with max_load_move and necessitated the addition
of the latter as an extra argument to the function. The internal
implementation is then modified to move up to max_nr_move tasks and
max_load_move of weighted load. This slightly complicates the code where
move_tasks() is called and if ever active_load_balance() is changed to not
use move_tasks() the implementation of move_tasks() should be simplified
accordingly.
struct sched_group *find_busiest_group():
Similar to try_to_wake_up(), there are places in this function where
SCHED_LOAD_SCALE is used to represent the load contribution of a single
task and the same issues are created. A similar solution is adopted except
that it is now the average per task contribution to a group's load (as
opposed to a run queue) that is required. As this value is not directly
available from the group it is calculated on the fly as the queues in the
groups are visited when determining the busiest group.
A key change to this function is that it is no longer to scale down
*imbalance on exit as move_tasks() uses the load in its scaled form.
void set_user_nice():
has been modified to update the task's load_weight field when it's nice
value and also to ensure that its run queue's raw_weighted_load field is
updated if it was runnable.
From: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
With smpnice, sched groups with highest priority tasks can mask the imbalance
between the other sched groups with in the same domain. This patch fixes some
of the listed down scenarios by not considering the sched groups which are
lightly loaded.
a) on a simple 4-way MP system, if we have one high priority and 4 normal
priority tasks, with smpnice we would like to see the high priority task
scheduled on one cpu, two other cpus getting one normal task each and the
fourth cpu getting the remaining two normal tasks. but with current
smpnice extra normal priority task keeps jumping from one cpu to another
cpu having the normal priority task. This is because of the
busiest_has_loaded_cpus, nr_loaded_cpus logic.. We are not including the
cpu with high priority task in max_load calculations but including that in
total and avg_load calcuations.. leading to max_load < avg_load and load
balance between cpus running normal priority tasks(2 Vs 1) will always show
imbalanace as one normal priority and the extra normal priority task will
keep moving from one cpu to another cpu having normal priority task..
b) 4-way system with HT (8 logical processors). Package-P0 T0 has a
highest priority task, T1 is idle. Package-P1 Both T0 and T1 have 1 normal
priority task each.. P2 and P3 are idle. With this patch, one of the
normal priority tasks on P1 will be moved to P2 or P3..
c) With the current weighted smp nice calculations, it doesn't always make
sense to look at the highest weighted runqueue in the busy group..
Consider a load balance scenario on a DP with HT system, with Package-0
containing one high priority and one low priority, Package-1 containing one
low priority(with other thread being idle).. Package-1 thinks that it need
to take the low priority thread from Package-0. And find_busiest_queue()
returns the cpu thread with highest priority task.. And ultimately(with
help of active load balance) we move high priority task to Package-1. And
same continues with Package-0 now, moving high priority task from package-1
to package-0.. Even without the presence of active load balance, load
balance will fail to balance the above scenario.. Fix find_busiest_queue
to use "imbalance" when it is lightly loaded.
[kernel@kolivas.org: sched: store weighted load on up]
[kernel@kolivas.org: sched: add discrete weighted cpu load function]
[suresh.b.siddha@intel.com: sched: remove dead code]
Signed-off-by: Peter Williams <pwil3058@bigpond.com.au>
Cc: "Siddha, Suresh B" <suresh.b.siddha@intel.com>
Cc: "Chen, Kenneth W" <kenneth.w.chen@intel.com>
Acked-by: Ingo Molnar <mingo@elte.hu>
Cc: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Con Kolivas <kernel@kolivas.org>
Cc: John Hawkes <hawkes@sgi.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
2006-06-27 03:54:34 -06:00
|
|
|
set_load_weight(&init_task);
|
2006-07-30 04:03:52 -06:00
|
|
|
|
|
|
|
#ifdef CONFIG_RT_MUTEXES
|
|
|
|
plist_head_init(&init_task.pi_waiters, &init_task.pi_lock);
|
|
|
|
#endif
|
|
|
|
|
2005-04-16 16:20:36 -06:00
|
|
|
/*
|
|
|
|
* The boot idle thread does lazy MMU switching as well:
|
|
|
|
*/
|
|
|
|
atomic_inc(&init_mm.mm_count);
|
|
|
|
enter_lazy_tlb(&init_mm, current);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Make us the idle thread. Technically, schedule() should not be
|
|
|
|
* called from this thread, however somewhere below it might be,
|
|
|
|
* but because we are the idle thread, we just pick up running again
|
|
|
|
* when this runqueue becomes "idle".
|
|
|
|
*/
|
|
|
|
init_idle(current, smp_processor_id());
|
|
|
|
}
|
|
|
|
|
|
|
|
#ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
|
|
|
|
void __might_sleep(char *file, int line)
|
|
|
|
{
|
2006-07-03 01:25:40 -06:00
|
|
|
#ifdef in_atomic
|
2005-04-16 16:20:36 -06:00
|
|
|
static unsigned long prev_jiffy; /* ratelimiting */
|
|
|
|
|
|
|
|
if ((in_atomic() || irqs_disabled()) &&
|
|
|
|
system_state == SYSTEM_RUNNING && !oops_in_progress) {
|
|
|
|
if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy)
|
|
|
|
return;
|
|
|
|
prev_jiffy = jiffies;
|
2006-03-23 04:00:54 -07:00
|
|
|
printk(KERN_ERR "BUG: sleeping function called from invalid"
|
2005-04-16 16:20:36 -06:00
|
|
|
" context at %s:%d\n", file, line);
|
|
|
|
printk("in_atomic():%d, irqs_disabled():%d\n",
|
|
|
|
in_atomic(), irqs_disabled());
|
|
|
|
dump_stack();
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(__might_sleep);
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#ifdef CONFIG_MAGIC_SYSRQ
|
|
|
|
void normalize_rt_tasks(void)
|
|
|
|
{
|
2006-07-03 01:25:42 -06:00
|
|
|
struct prio_array *array;
|
2005-04-16 16:20:36 -06:00
|
|
|
struct task_struct *p;
|
|
|
|
unsigned long flags;
|
2006-07-03 01:25:42 -06:00
|
|
|
struct rq *rq;
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
read_lock_irq(&tasklist_lock);
|
2006-06-27 03:54:28 -06:00
|
|
|
for_each_process(p) {
|
2005-04-16 16:20:36 -06:00
|
|
|
if (!rt_task(p))
|
|
|
|
continue;
|
|
|
|
|
2006-06-27 03:54:51 -06:00
|
|
|
spin_lock_irqsave(&p->pi_lock, flags);
|
|
|
|
rq = __task_rq_lock(p);
|
2005-04-16 16:20:36 -06:00
|
|
|
|
|
|
|
array = p->array;
|
|
|
|
if (array)
|
|
|
|
deactivate_task(p, task_rq(p));
|
|
|
|
__setscheduler(p, SCHED_NORMAL, 0);
|
|
|
|
if (array) {
|
|
|
|
__activate_task(p, task_rq(p));
|
|
|
|
resched_task(rq->curr);
|
|
|
|
}
|
|
|
|
|
2006-06-27 03:54:51 -06:00
|
|
|
__task_rq_unlock(rq);
|
|
|
|
spin_unlock_irqrestore(&p->pi_lock, flags);
|
2005-04-16 16:20:36 -06:00
|
|
|
}
|
|
|
|
read_unlock_irq(&tasklist_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
#endif /* CONFIG_MAGIC_SYSRQ */
|
2005-09-12 08:59:21 -06:00
|
|
|
|
|
|
|
#ifdef CONFIG_IA64
|
|
|
|
/*
|
|
|
|
* These functions are only useful for the IA64 MCA handling.
|
|
|
|
*
|
|
|
|
* They can only be called when the whole system has been
|
|
|
|
* stopped - every CPU needs to be quiescent, and no scheduling
|
|
|
|
* activity can take place. Using them for anything else would
|
|
|
|
* be a serious bug, and as a result, they aren't even visible
|
|
|
|
* under any other configuration.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/**
|
|
|
|
* curr_task - return the current task for a given cpu.
|
|
|
|
* @cpu: the processor in question.
|
|
|
|
*
|
|
|
|
* ONLY VALID WHEN THE WHOLE SYSTEM IS STOPPED!
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
struct task_struct *curr_task(int cpu)
|
2005-09-12 08:59:21 -06:00
|
|
|
{
|
|
|
|
return cpu_curr(cpu);
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* set_curr_task - set the current task for a given cpu.
|
|
|
|
* @cpu: the processor in question.
|
|
|
|
* @p: the task pointer to set.
|
|
|
|
*
|
|
|
|
* Description: This function must only be used when non-maskable interrupts
|
|
|
|
* are serviced on a separate stack. It allows the architecture to switch the
|
|
|
|
* notion of the current task on a cpu in a non-blocking manner. This function
|
|
|
|
* must be called with all CPU's synchronized, and interrupts disabled, the
|
|
|
|
* and caller must save the original value of the current task (see
|
|
|
|
* curr_task() above) and restore that value before reenabling interrupts and
|
|
|
|
* re-starting the system.
|
|
|
|
*
|
|
|
|
* ONLY VALID WHEN THE WHOLE SYSTEM IS STOPPED!
|
|
|
|
*/
|
2006-07-03 01:25:41 -06:00
|
|
|
void set_curr_task(int cpu, struct task_struct *p)
|
2005-09-12 08:59:21 -06:00
|
|
|
{
|
|
|
|
cpu_curr(cpu) = p;
|
|
|
|
}
|
|
|
|
|
|
|
|
#endif
|