Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 6428671b authored by Peter Zijlstra's avatar Peter Zijlstra Committed by Ingo Molnar
Browse files

locking/mutex: Optimize mutex_trylock() fast-path 



A while back Viro posted a number of 'interesting' mutex_is_locked()
users on IRC, one of those was RCU.

RCU seems to use mutex_is_locked() to avoid doing mutex_trylock(), the
regular load before modify pattern.

While the use isn't wrong per se, its curious in that its needed at all,
mutex_trylock() should be good enough on its own to avoid the pointless
cacheline bounces.

So fix those and remove the mutex_is_locked() (ab)use from RCU.

Reported-by: default avatarAl Viro <viro@ZenIV.linux.org.uk>
Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: default avatarPaul McKenney <paulmck@linux.vnet.ibm.com>
Acked-by: default avatarDavidlohr Bueso <dave@stgolabs.net>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Waiman Long <Waiman.Long@hpe.com>
Link: http://lkml.kernel.org/r/20160601185815.GW3190@twins.programming.kicks-ass.net


Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
parent ddd0fa73

arch/ia64/include/asm/mutex.h

+1 −1
Original line number Diff line number Diff line
@@ -82,7 +82,7 @@ __mutex_fastpath_unlock(atomic_t *count, void (*fail_fn)(atomic_t *)) 
static inline int

__mutex_fastpath_trylock(atomic_t *count, int (*fail_fn)(atomic_t *))

{

	if (cmpxchg_acq(count, 1, 0) == 1)

	if (atomic_read(count) == 1 && cmpxchg_acq(count, 1, 0) == 1)

		return 1;

	return 0;

}

arch/powerpc/include/asm/mutex.h

+1 −1
Original line number Diff line number Diff line
@@ -124,7 +124,7 @@ __mutex_fastpath_unlock(atomic_t *count, void (*fail_fn)(atomic_t *)) 
static inline int

__mutex_fastpath_trylock(atomic_t *count, int (*fail_fn)(atomic_t *))

{

	if (likely(__mutex_cmpxchg_lock(count, 1, 0) == 1))

	if (likely(atomic_read(count) == 1 && __mutex_cmpxchg_lock(count, 1, 0) == 1))

		return 1;

	return 0;

}

arch/x86/include/asm/mutex_32.h

+1 −1
Original line number Diff line number Diff line
@@ -101,7 +101,7 @@ static inline int __mutex_fastpath_trylock(atomic_t *count, 
					   int (*fail_fn)(atomic_t *))

{

	/* cmpxchg because it never induces a false contention state. */

	if (likely(atomic_cmpxchg(count, 1, 0) == 1))

	if (likely(atomic_read(count) == 1 && atomic_cmpxchg(count, 1, 0) == 1))

		return 1;



	return 0;

arch/x86/include/asm/mutex_64.h

+3 −3
Original line number Diff line number Diff line
@@ -118,9 +118,9 @@ do { \ 
static inline int __mutex_fastpath_trylock(atomic_t *count,

					   int (*fail_fn)(atomic_t *))

{

	if (likely(atomic_cmpxchg(count, 1, 0) == 1))

	if (likely(atomic_read(count) == 1 && atomic_cmpxchg(count, 1, 0) == 1))

		return 1;

	else



	return 0;

}

include/asm-generic/mutex-dec.h

+1 −1
Original line number Diff line number Diff line
@@ -80,7 +80,7 @@ __mutex_fastpath_unlock(atomic_t *count, void (*fail_fn)(atomic_t *)) 
static inline int

__mutex_fastpath_trylock(atomic_t *count, int (*fail_fn)(atomic_t *))

{

	if (likely(atomic_cmpxchg_acquire(count, 1, 0) == 1))

	if (likely(atomic_read(count) == 1 && atomic_cmpxchg_acquire(count, 1, 0) == 1))

		return 1;

	return 0;

}