Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit c86ad14d authored by Linus Torvalds's avatar Linus Torvalds
Browse files

Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 

Pull locking updates from Ingo Molnar:
 "The locking tree was busier in this cycle than the usual pattern - a
  couple of major projects happened to coincide.

  The main changes are:

   - implement the atomic_fetch_{add,sub,and,or,xor}() API natively
     across all SMP architectures (Peter Zijlstra)

   - add atomic_fetch_{inc/dec}() as well, using the generic primitives
     (Davidlohr Bueso)

   - optimize various aspects of rwsems (Jason Low, Davidlohr Bueso,
     Waiman Long)

   - optimize smp_cond_load_acquire() on arm64 and implement LSE based
     atomic{,64}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}()
     on arm64 (Will Deacon)

   - introduce smp_acquire__after_ctrl_dep() and fix various barrier
     mis-uses and bugs (Peter Zijlstra)

   - after discovering ancient spin_unlock_wait() barrier bugs in its
     implementation and usage, strengthen its semantics and update/fix
     usage sites (Peter Zijlstra)

   - optimize mutex_trylock() fastpath (Peter Zijlstra)

   - ... misc fixes and cleanups"

* 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (67 commits)
  locking/atomic: Introduce inc/dec variants for the atomic_fetch_$op() API
  locking/barriers, arch/arm64: Implement LDXR+WFE based smp_cond_load_acquire()
  locking/static_keys: Fix non static symbol Sparse warning
  locking/qspinlock: Use __this_cpu_dec() instead of full-blown this_cpu_dec()
  locking/atomic, arch/tile: Fix tilepro build
  locking/atomic, arch/m68k: Remove comment
  locking/atomic, arch/arc: Fix build
  locking/Documentation: Clarify limited control-dependency scope
  locking/atomic, arch/rwsem: Employ atomic_long_fetch_add()
  locking/atomic, arch/qrwlock: Employ atomic_fetch_add_acquire()
  locking/atomic, arch/mips: Convert to _relaxed atomics
  locking/atomic, arch/alpha: Convert to _relaxed atomics
  locking/atomic: Remove the deprecated atomic_{set,clear}_mask() functions
  locking/atomic: Remove linux/atomic.h:atomic_fetch_or()
  locking/atomic: Implement atomic{,64,_long}_fetch_{add,sub,and,andnot,or,xor}{,_relaxed,_acquire,_release}()
  locking/atomic: Fix atomic64_relaxed() bits
  locking/atomic, arch/xtensa: Implement atomic_fetch_{add,sub,and,or,xor}()
  locking/atomic, arch/x86: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
  locking/atomic, arch/tile: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
  locking/atomic, arch/sparc: Implement atomic{,64}_fetch_{add,sub,and,or,xor}()
  ...
parents a2303849 f0662863

Documentation/memory-barriers.txt

+41 −0
Original line number Diff line number Diff line
@@ -806,6 +806,41 @@ out-guess your code. More generally, although READ_ONCE() does force 
the compiler to actually emit code for a given load, it does not force

the compiler to use the results.



In addition, control dependencies apply only to the then-clause and

else-clause of the if-statement in question.  In particular, it does

not necessarily apply to code following the if-statement:



	q = READ_ONCE(a);

	if (q) {

		WRITE_ONCE(b, p);

	} else {

		WRITE_ONCE(b, r);

	}

	WRITE_ONCE(c, 1);  /* BUG: No ordering against the read from "a". */



It is tempting to argue that there in fact is ordering because the

compiler cannot reorder volatile accesses and also cannot reorder

the writes to "b" with the condition.  Unfortunately for this line

of reasoning, the compiler might compile the two writes to "b" as

conditional-move instructions, as in this fanciful pseudo-assembly

language:



	ld r1,a

	ld r2,p

	ld r3,r

	cmp r1,$0

	cmov,ne r4,r2

	cmov,eq r4,r3

	st r4,b

	st $1,c



A weakly ordered CPU would have no dependency of any sort between the load

from "a" and the store to "c".  The control dependencies would extend

only to the pair of cmov instructions and the store depending on them.

In short, control dependencies apply only to the stores in the then-clause

and else-clause of the if-statement in question (including functions

invoked by those two clauses), not to code following that if-statement.



Finally, control dependencies do -not- provide transitivity.  This is

demonstrated by two related examples, with the initial values of

x and y both being zero:
@@ -869,6 +904,12 @@ In summary: 
      atomic{,64}_read() can help to preserve your control dependency.

      Please see the COMPILER BARRIER section for more information.



  (*) Control dependencies apply only to the then-clause and else-clause

      of the if-statement containing the control dependency, including

      any functions that these two clauses call.  Control dependencies

      do -not- apply to code following the if-statement containing the

      control dependency.



  (*) Control dependencies pair normally with other types of barriers.



  (*) Control dependencies do -not- provide transitivity.  If you

MAINTAINERS

+12 −4
Original line number Diff line number Diff line
@@ -7024,15 +7024,23 @@ Q: http://patchwork.linuxtv.org/project/linux-media/list/ 
S:	Maintained

F:	drivers/media/usb/dvb-usb-v2/lmedm04*



LOCKDEP AND LOCKSTAT

LOCKING PRIMITIVES

M:	Peter Zijlstra <peterz@infradead.org>

M:	Ingo Molnar <mingo@redhat.com>

L:	linux-kernel@vger.kernel.org

T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core/locking

T:	git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking/core

S:	Maintained

F:	Documentation/locking/lockdep*.txt

F:	Documentation/locking/lockstat.txt

F:	Documentation/locking/

F:	include/linux/lockdep.h

F:	include/linux/spinlock*.h

F:	arch/*/include/asm/spinlock*.h

F:	include/linux/rwlock*.h

F:	include/linux/mutex*.h

F:	arch/*/include/asm/mutex*.h

F:	include/linux/rwsem*.h

F:	arch/*/include/asm/rwsem.h

F:	include/linux/seqlock.h

F:	lib/locking*.[ch]

F:	kernel/locking/



LOGICAL DISK MANAGER SUPPORT (LDM, Windows 2000/XP/Vista Dynamic Disks)

arch/alpha/include/asm/atomic.h

+72 −15
Original line number Diff line number Diff line
@@ -46,10 +46,9 @@ static __inline__ void atomic_##op(int i, atomic_t * v) \ 
}									\



#define ATOMIC_OP_RETURN(op, asm_op)					\

static inline int atomic_##op##_return(int i, atomic_t *v)		\

static inline int atomic_##op##_return_relaxed(int i, atomic_t *v)	\

{									\

	long temp, result;						\

	smp_mb();							\

	__asm__ __volatile__(						\

	"1:	ldl_l %0,%1\n"						\

	"	" #asm_op " %0,%3,%2\n"					\
@@ -61,7 +60,23 @@ static inline int atomic_##op##_return(int i, atomic_t *v) \ 
	".previous"							\

	:"=&r" (temp), "=m" (v->counter), "=&r" (result)		\

	:"Ir" (i), "m" (v->counter) : "memory");			\

	smp_mb();							\

	return result;							\

}



#define ATOMIC_FETCH_OP(op, asm_op)					\

static inline int atomic_fetch_##op##_relaxed(int i, atomic_t *v)	\

{									\

	long temp, result;						\

	__asm__ __volatile__(						\

	"1:	ldl_l %2,%1\n"						\

	"	" #asm_op " %2,%3,%0\n"					\

	"	stl_c %0,%1\n"						\

	"	beq %0,2f\n"						\

	".subsection 2\n"						\

	"2:	br 1b\n"						\

	".previous"							\

	:"=&r" (temp), "=m" (v->counter), "=&r" (result)		\

	:"Ir" (i), "m" (v->counter) : "memory");			\

	return result;							\

}
@@ -82,10 +97,9 @@ static __inline__ void atomic64_##op(long i, atomic64_t * v) \ 
}									\



#define ATOMIC64_OP_RETURN(op, asm_op)					\

static __inline__ long atomic64_##op##_return(long i, atomic64_t * v)	\

static __inline__ long atomic64_##op##_return_relaxed(long i, atomic64_t * v)	\

{									\

	long temp, result;						\

	smp_mb();							\

	__asm__ __volatile__(						\

	"1:	ldq_l %0,%1\n"						\

	"	" #asm_op " %0,%3,%2\n"					\
@@ -97,34 +111,77 @@ static __inline__ long atomic64_##op##_return(long i, atomic64_t * v) \ 
	".previous"							\

	:"=&r" (temp), "=m" (v->counter), "=&r" (result)		\

	:"Ir" (i), "m" (v->counter) : "memory");			\

	smp_mb();							\

	return result;							\

}



#define ATOMIC64_FETCH_OP(op, asm_op)					\

static __inline__ long atomic64_fetch_##op##_relaxed(long i, atomic64_t * v)	\

{									\

	long temp, result;						\

	__asm__ __volatile__(						\

	"1:	ldq_l %2,%1\n"						\

	"	" #asm_op " %2,%3,%0\n"					\

	"	stq_c %0,%1\n"						\

	"	beq %0,2f\n"						\

	".subsection 2\n"						\

	"2:	br 1b\n"						\

	".previous"							\

	:"=&r" (temp), "=m" (v->counter), "=&r" (result)		\

	:"Ir" (i), "m" (v->counter) : "memory");			\

	return result;							\

}



#define ATOMIC_OPS(op)							\

	ATOMIC_OP(op, op##l)						\

	ATOMIC_OP_RETURN(op, op##l)					\

	ATOMIC_FETCH_OP(op, op##l)					\

	ATOMIC64_OP(op, op##q)						\

	ATOMIC64_OP_RETURN(op, op##q)

	ATOMIC64_OP_RETURN(op, op##q)					\

	ATOMIC64_FETCH_OP(op, op##q)



ATOMIC_OPS(add)

ATOMIC_OPS(sub)



#define atomic_add_return_relaxed	atomic_add_return_relaxed

#define atomic_sub_return_relaxed	atomic_sub_return_relaxed

#define atomic_fetch_add_relaxed	atomic_fetch_add_relaxed

#define atomic_fetch_sub_relaxed	atomic_fetch_sub_relaxed



#define atomic64_add_return_relaxed	atomic64_add_return_relaxed

#define atomic64_sub_return_relaxed	atomic64_sub_return_relaxed

#define atomic64_fetch_add_relaxed	atomic64_fetch_add_relaxed

#define atomic64_fetch_sub_relaxed	atomic64_fetch_sub_relaxed



#define atomic_andnot atomic_andnot

#define atomic64_andnot atomic64_andnot



ATOMIC_OP(and, and)

ATOMIC_OP(andnot, bic)

ATOMIC_OP(or, bis)

ATOMIC_OP(xor, xor)

ATOMIC64_OP(and, and)

ATOMIC64_OP(andnot, bic)

ATOMIC64_OP(or, bis)

ATOMIC64_OP(xor, xor)

#undef ATOMIC_OPS

#define ATOMIC_OPS(op, asm)						\

	ATOMIC_OP(op, asm)						\

	ATOMIC_FETCH_OP(op, asm)					\

	ATOMIC64_OP(op, asm)						\

	ATOMIC64_FETCH_OP(op, asm)



ATOMIC_OPS(and, and)

ATOMIC_OPS(andnot, bic)

ATOMIC_OPS(or, bis)

ATOMIC_OPS(xor, xor)



#define atomic_fetch_and_relaxed	atomic_fetch_and_relaxed

#define atomic_fetch_andnot_relaxed	atomic_fetch_andnot_relaxed

#define atomic_fetch_or_relaxed		atomic_fetch_or_relaxed

#define atomic_fetch_xor_relaxed	atomic_fetch_xor_relaxed



#define atomic64_fetch_and_relaxed	atomic64_fetch_and_relaxed

#define atomic64_fetch_andnot_relaxed	atomic64_fetch_andnot_relaxed

#define atomic64_fetch_or_relaxed	atomic64_fetch_or_relaxed

#define atomic64_fetch_xor_relaxed	atomic64_fetch_xor_relaxed



#undef ATOMIC_OPS

#undef ATOMIC64_FETCH_OP

#undef ATOMIC64_OP_RETURN

#undef ATOMIC64_OP

#undef ATOMIC_FETCH_OP

#undef ATOMIC_OP_RETURN

#undef ATOMIC_OP

arch/alpha/include/asm/rwsem.h

+13 −55
Original line number Diff line number Diff line
@@ -25,8 +25,8 @@ static inline void __down_read(struct rw_semaphore *sem) 
{

	long oldcount;

#ifndef	CONFIG_SMP

	oldcount = sem->count;

	sem->count += RWSEM_ACTIVE_READ_BIAS;

	oldcount = sem->count.counter;

	sem->count.counter += RWSEM_ACTIVE_READ_BIAS;

#else

	long temp;

	__asm__ __volatile__(
@@ -52,13 +52,13 @@ static inline int __down_read_trylock(struct rw_semaphore *sem) 
{

	long old, new, res;



	res = sem->count;

	res = atomic_long_read(&sem->count);

	do {

		new = res + RWSEM_ACTIVE_READ_BIAS;

		if (new <= 0)

			break;

		old = res;

		res = cmpxchg(&sem->count, old, new);

		res = atomic_long_cmpxchg(&sem->count, old, new);

	} while (res != old);

	return res >= 0 ? 1 : 0;

}
@@ -67,8 +67,8 @@ static inline long ___down_write(struct rw_semaphore *sem) 
{

	long oldcount;

#ifndef	CONFIG_SMP

	oldcount = sem->count;

	sem->count += RWSEM_ACTIVE_WRITE_BIAS;

	oldcount = sem->count.counter;

	sem->count.counter += RWSEM_ACTIVE_WRITE_BIAS;

#else

	long temp;

	__asm__ __volatile__(
@@ -106,7 +106,7 @@ static inline int __down_write_killable(struct rw_semaphore *sem) 
 */

static inline int __down_write_trylock(struct rw_semaphore *sem)

{

	long ret = cmpxchg(&sem->count, RWSEM_UNLOCKED_VALUE,

	long ret = atomic_long_cmpxchg(&sem->count, RWSEM_UNLOCKED_VALUE,

			   RWSEM_ACTIVE_WRITE_BIAS);

	if (ret == RWSEM_UNLOCKED_VALUE)

		return 1;
@@ -117,8 +117,8 @@ static inline void __up_read(struct rw_semaphore *sem) 
{

	long oldcount;

#ifndef	CONFIG_SMP

	oldcount = sem->count;

	sem->count -= RWSEM_ACTIVE_READ_BIAS;

	oldcount = sem->count.counter;

	sem->count.counter -= RWSEM_ACTIVE_READ_BIAS;

#else

	long temp;

	__asm__ __volatile__(
@@ -142,8 +142,8 @@ static inline void __up_write(struct rw_semaphore *sem) 
{

	long count;

#ifndef	CONFIG_SMP

	sem->count -= RWSEM_ACTIVE_WRITE_BIAS;

	count = sem->count;

	sem->count.counter -= RWSEM_ACTIVE_WRITE_BIAS;

	count = sem->count.counter;

#else

	long temp;

	__asm__ __volatile__(
@@ -171,8 +171,8 @@ static inline void __downgrade_write(struct rw_semaphore *sem) 
{

	long oldcount;

#ifndef	CONFIG_SMP

	oldcount = sem->count;

	sem->count -= RWSEM_WAITING_BIAS;

	oldcount = sem->count.counter;

	sem->count.counter -= RWSEM_WAITING_BIAS;

#else

	long temp;

	__asm__ __volatile__(
@@ -191,47 +191,5 @@ static inline void __downgrade_write(struct rw_semaphore *sem) 
		rwsem_downgrade_wake(sem);

}



static inline void rwsem_atomic_add(long val, struct rw_semaphore *sem)

{

#ifndef	CONFIG_SMP

	sem->count += val;

#else

	long temp;

	__asm__ __volatile__(

	"1:	ldq_l	%0,%1\n"

	"	addq	%0,%2,%0\n"

	"	stq_c	%0,%1\n"

	"	beq	%0,2f\n"

	".subsection 2\n"

	"2:	br	1b\n"

	".previous"

	:"=&r" (temp), "=m" (sem->count)

	:"Ir" (val), "m" (sem->count));

#endif

}



static inline long rwsem_atomic_update(long val, struct rw_semaphore *sem)

{

#ifndef	CONFIG_SMP

	sem->count += val;

	return sem->count;

#else

	long ret, temp;

	__asm__ __volatile__(

	"1:	ldq_l	%0,%1\n"

	"	addq 	%0,%3,%2\n"

	"	addq	%0,%3,%0\n"

	"	stq_c	%2,%1\n"

	"	beq	%2,2f\n"

	".subsection 2\n"

	"2:	br	1b\n"

	".previous"

	:"=&r" (ret), "=m" (sem->count), "=&r" (temp)

	:"Ir" (val), "m" (sem->count));



	return ret;

#endif

}



#endif /* __KERNEL__ */

#endif /* _ALPHA_RWSEM_H */

arch/alpha/include/asm/spinlock.h

+7 −2
Original line number Diff line number Diff line
@@ -3,6 +3,8 @@ 


#include <linux/kernel.h>

#include <asm/current.h>

#include <asm/barrier.h>

#include <asm/processor.h>



/*

 * Simple spin lock operations.  There are two variants, one clears IRQ's
@@ -13,8 +15,11 @@ 


#define arch_spin_lock_flags(lock, flags) arch_spin_lock(lock)

#define arch_spin_is_locked(x)	((x)->lock != 0)

#define arch_spin_unlock_wait(x) \

		do { cpu_relax(); } while ((x)->lock)



static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)

{

	smp_cond_load_acquire(&lock->lock, !VAL);

}



static inline int arch_spin_value_unlocked(arch_spinlock_t lock)

{