314cdbefd1
Introduce ticket lock spinlocks for x86 which are FIFO. The implementation is described in the comments. The straight-line lock/unlock instruction sequence is slightly slower than the dec based locks on modern x86 CPUs, however the difference is quite small on Core2 and Opteron when working out of cache, and becomes almost insignificant even on P4 when the lock misses cache. trylock is more significantly slower, but they are relatively rare. On an 8 core (2 socket) Opteron, spinlock unfairness is extremely noticable, with a userspace test having a difference of up to 2x runtime per thread, and some threads are starved or "unfairly" granted the lock up to 1 000 000 (!) times. After this patch, all threads appear to finish at exactly the same time. The memory ordering of the lock does conform to x86 standards, and the implementation has been reviewed by Intel and AMD engineers. The algorithm also tells us how many CPUs are contending the lock, so lockbreak becomes trivial and we no longer have to waste 4 bytes per spinlock for it. After this, we can no longer spin on any locks with preempt enabled and cannot reenable interrupts when spinning on an irq safe lock, because at that point we have already taken a ticket and the would deadlock if the same CPU tries to take the lock again. These are questionable anyway: if the lock happens to be called under a preempt or interrupt disabled section, then it will just have the same latency problems. The real fix is to keep critical sections short, and ensure locks are reasonably fair (which this patch does). Signed-off-by: Nick Piggin <npiggin@suse.de> Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Ingo Molnar <mingo@elte.hu>
21 lines
362 B
C
21 lines
362 B
C
#ifndef __ASM_SPINLOCK_TYPES_H
|
|
#define __ASM_SPINLOCK_TYPES_H
|
|
|
|
#ifndef __LINUX_SPINLOCK_TYPES_H
|
|
# error "please don't include this file directly"
|
|
#endif
|
|
|
|
typedef struct {
|
|
unsigned int slock;
|
|
} raw_spinlock_t;
|
|
|
|
#define __RAW_SPIN_LOCK_UNLOCKED { 0 }
|
|
|
|
typedef struct {
|
|
unsigned int lock;
|
|
} raw_rwlock_t;
|
|
|
|
#define __RAW_RW_LOCK_UNLOCKED { RW_LOCK_BIAS }
|
|
|
|
#endif
|