Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit d63a9788 authored by Linus Torvalds's avatar Linus Torvalds
Browse files

Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 

Pull locking changes from Ingo Molnar:
 "The main changes in this cycle were:

   - More gradual enhancements to atomic ops: new atomic*_read_ctrl()
     ops, synchronize atomic_{read,set}() ordering requirements between
     architectures, add atomic_long_t bitops.  (Peter Zijlstra)

   - Add _{relaxed|acquire|release}() variants for inc/dec atomics and
     use them in various locking primitives: mutex, rtmutex, mcs, rwsem.
     This enables weakly ordered architectures (such as arm64) to make
     use of more locking related optimizations.  (Davidlohr Bueso)

   - Implement atomic[64]_{inc,dec}_relaxed() on ARM.  (Will Deacon)

   - Futex kernel data cache footprint micro-optimization.  (Rasmus
     Villemoes)

   - pvqspinlock runtime overhead micro-optimization.  (Waiman Long)

   - misc smaller fixlets"

* 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  ARM, locking/atomics: Implement _relaxed variants of atomic[64]_{inc,dec}
  locking/rwsem: Use acquire/release semantics
  locking/mcs: Use acquire/release semantics
  locking/rtmutex: Use acquire/release semantics
  locking/mutex: Use acquire/release semantics
  locking/asm-generic: Add _{relaxed|acquire|release}() variants for inc/dec atomics
  atomic: Implement atomic_read_ctrl()
  atomic, arch: Audit atomic_{read,set}()
  atomic: Add atomic_long_t bitops
  futex: Force hot variables into a single cache line
  locking/pvqspinlock: Kick the PV CPU unconditionally when _Q_SLOW_VAL
  locking/osq: Relax atomic semantics
  locking/qrwlock: Rename ->lock to ->wait_lock
  locking/Documentation/lockstat: Fix typo - lokcing -> locking
  locking/atomics, cmpxchg: Privatize the inclusion of asm/cmpxchg.h
parents 28142286 6e490b01

Documentation/atomic_ops.txt

+4 −0
Original line number Diff line number Diff line
@@ -542,6 +542,10 @@ The routines xchg() and cmpxchg() must provide the same exact 
memory-barrier semantics as the atomic and bit operations returning

values.



Note: If someone wants to use xchg(), cmpxchg() and their variants,

linux/atomic.h should be included rather than asm/cmpxchg.h, unless

the code is in arch/* and can take care of itself.



Spinlocks and rwlocks have memory barrier expectations as well.

The rule to follow is simple:

Documentation/locking/lockstat.txt

+1 −1
Original line number Diff line number Diff line
@@ -12,7 +12,7 @@ Because things like lock contention can severely impact performance. 
- HOW



Lockdep already has hooks in the lock functions and maps lock instances to

lock classes. We build on that (see Documentation/lokcing/lockdep-design.txt).

lock classes. We build on that (see Documentation/locking/lockdep-design.txt).

The graph below shows the relation between the lock functions and the various

hooks therein.

Documentation/memory-barriers.txt

+9 −8
Original line number Diff line number Diff line
@@ -637,7 +637,8 @@ as follows: 
	b = p;  /* BUG: Compiler and CPU can both reorder!!! */



Finally, the READ_ONCE_CTRL() includes an smp_read_barrier_depends()

that DEC Alpha needs in order to respect control depedencies.

that DEC Alpha needs in order to respect control depedencies. Alternatively

use one of atomic{,64}_read_ctrl().



So don't leave out the READ_ONCE_CTRL().
@@ -796,9 +797,9 @@ site: https://www.cl.cam.ac.uk/~pes20/ppcmem/index.html. 


In summary:



  (*) Control dependencies must be headed by READ_ONCE_CTRL().

      Or, as a much less preferable alternative, interpose

      smp_read_barrier_depends() between a READ_ONCE() and the

  (*) Control dependencies must be headed by READ_ONCE_CTRL(),

      atomic{,64}_read_ctrl(). Or, as a much less preferable alternative,

      interpose smp_read_barrier_depends() between a READ_ONCE() and the

      control-dependent write.



  (*) Control dependencies can order prior loads against later stores.
@@ -820,10 +821,10 @@ In summary: 
      and WRITE_ONCE() can help to preserve the needed conditional.



  (*) Control dependencies require that the compiler avoid reordering the

      dependency into nonexistence.  Careful use of READ_ONCE_CTRL()

      or smp_read_barrier_depends() can help to preserve your control

      dependency.  Please see the Compiler Barrier section for more

      information.

      dependency into nonexistence.  Careful use of READ_ONCE_CTRL(),

      atomic{,64}_read_ctrl() or smp_read_barrier_depends() can help to

      preserve your control dependency.  Please see the Compiler Barrier

      section for more information.



  (*) Control dependencies pair normally with other types of barriers.

arch/alpha/include/asm/atomic.h

+4 −4
Original line number Diff line number Diff line
@@ -17,11 +17,11 @@ 
#define ATOMIC_INIT(i)		{ (i) }

#define ATOMIC64_INIT(i)	{ (i) }



#define atomic_read(v)		ACCESS_ONCE((v)->counter)

#define atomic64_read(v)	ACCESS_ONCE((v)->counter)

#define atomic_read(v)		READ_ONCE((v)->counter)

#define atomic64_read(v)	READ_ONCE((v)->counter)



#define atomic_set(v,i)		((v)->counter = (i))

#define atomic64_set(v,i)	((v)->counter = (i))

#define atomic_set(v,i)		WRITE_ONCE((v)->counter, (i))

#define atomic64_set(v,i)	WRITE_ONCE((v)->counter, (i))



/*

 * To get proper branch prediction for the main line, we must branch

arch/arc/include/asm/atomic.h

+4 −4
Original line number Diff line number Diff line
@@ -17,11 +17,11 @@ 
#include <asm/barrier.h>

#include <asm/smp.h>



#define atomic_read(v)  ((v)->counter)

#define atomic_read(v)  READ_ONCE((v)->counter)



#ifdef CONFIG_ARC_HAS_LLSC



#define atomic_set(v, i) (((v)->counter) = (i))

#define atomic_set(v, i) WRITE_ONCE(((v)->counter), (i))



#ifdef CONFIG_ARC_STAR_9000923308
@@ -107,7 +107,7 @@ static inline int atomic_##op##_return(int i, atomic_t *v) \ 
#ifndef CONFIG_SMP



 /* violating atomic_xxx API locking protocol in UP for optimization sake */

#define atomic_set(v, i) (((v)->counter) = (i))

#define atomic_set(v, i) WRITE_ONCE(((v)->counter), (i))



#else
@@ -125,7 +125,7 @@ static inline void atomic_set(atomic_t *v, int i) 
	unsigned long flags;



	atomic_ops_lock(flags);

	v->counter = i;

	WRITE_ONCE(v->counter, i);

	atomic_ops_unlock(flags);

}