Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit d89e588c authored by Peter Zijlstra's avatar Peter Zijlstra Committed by Ingo Molnar
Browse files

locking: Introduce smp_mb__after_spinlock() 



Since its inception, our understanding of ACQUIRE, esp. as applied to
spinlocks, has changed somewhat. Also, I wonder if, with a simple
change, we cannot make it provide more.

The problem with the comment is that the STORE done by spin_lock isn't
itself ordered by the ACQUIRE, and therefore a later LOAD can pass over
it and cross with any prior STORE, rendering the default WMB
insufficient (pointed out by Alan).

Now, this is only really a problem on PowerPC and ARM64, both of
which already defined smp_mb__before_spinlock() as a smp_mb().

At the same time, we can get a much stronger construct if we place
that same barrier _inside_ the spin_lock(). In that case we upgrade
the RCpc spinlock to an RCsc.  That would make all schedule() calls
fully transitive against one another.

Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: default avatarWill Deacon <will.deacon@arm.com>
Cc: Alan Stern <stern@rowland.harvard.edu>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Paul McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
parent ff7a5fb0

arch/arm64/include/asm/spinlock.h

+2 −0
Original line number Diff line number Diff line
@@ -367,5 +367,7 @@ static inline int arch_read_trylock(arch_rwlock_t *rw) 
 * smp_mb__before_spinlock() can restore the required ordering.

 */

#define smp_mb__before_spinlock()	smp_mb()

/* See include/linux/spinlock.h */

#define smp_mb__after_spinlock()	smp_mb()



#endif /* __ASM_SPINLOCK_H */

arch/powerpc/include/asm/spinlock.h

+3 −0
Original line number Diff line number Diff line
@@ -342,5 +342,8 @@ static inline void arch_write_unlock(arch_rwlock_t *rw) 
#define arch_read_relax(lock)	__rw_yield(lock)

#define arch_write_relax(lock)	__rw_yield(lock)



/* See include/linux/spinlock.h */

#define smp_mb__after_spinlock()   smp_mb()



#endif /* __KERNEL__ */

#endif /* __ASM_SPINLOCK_H */

include/linux/atomic.h

+3 −0
Original line number Diff line number Diff line
@@ -38,6 +38,9 @@ 
 * Besides, if an arch has a special barrier for acquire/release, it could

 * implement its own __atomic_op_* and use the same framework for building

 * variants

 *

 * If an architecture overrides __atomic_op_acquire() it will probably want

 * to define smp_mb__after_spinlock().

 */

#ifndef __atomic_op_acquire

#define __atomic_op_acquire(op, args...)				\

include/linux/spinlock.h

+36 −0
Original line number Diff line number Diff line
@@ -130,6 +130,42 @@ do { \ 
#define smp_mb__before_spinlock()	smp_wmb()

#endif



/*

 * This barrier must provide two things:

 *

 *   - it must guarantee a STORE before the spin_lock() is ordered against a

 *     LOAD after it, see the comments at its two usage sites.

 *

 *   - it must ensure the critical section is RCsc.

 *

 * The latter is important for cases where we observe values written by other

 * CPUs in spin-loops, without barriers, while being subject to scheduling.

 *

 * CPU0			CPU1			CPU2

 *

 *			for (;;) {

 *			  if (READ_ONCE(X))

 *			    break;

 *			}

 * X=1

 *			<sched-out>

 *						<sched-in>

 *						r = X;

 *

 * without transitivity it could be that CPU1 observes X!=0 breaks the loop,

 * we get migrated and CPU2 sees X==0.

 *

 * Since most load-store architectures implement ACQUIRE with an smp_mb() after

 * the LL/SC loop, they need no further barriers. Similarly all our TSO

 * architectures imply an smp_mb() for each atomic instruction and equally don't

 * need more.

 *

 * Architectures that can implement ACQUIRE better need to take care.

 */

#ifndef smp_mb__after_spinlock

#define smp_mb__after_spinlock()	do { } while (0)

#endif



/**

 * raw_spin_unlock_wait - wait until the spinlock gets unlocked

 * @lock: the spinlock in question.

kernel/sched/core.c

+2 −2
Original line number Diff line number Diff line
@@ -1967,8 +1967,8 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) 
	 * reordered with p->state check below. This pairs with mb() in

	 * set_current_state() the waiting thread does.

	 */

	smp_mb__before_spinlock();

	raw_spin_lock_irqsave(&p->pi_lock, flags);

	smp_mb__after_spinlock();

	if (!(p->state & state))

		goto out;
@@ -3281,8 +3281,8 @@ static void __sched notrace __schedule(bool preempt) 
	 * can't be reordered with __set_current_state(TASK_INTERRUPTIBLE)

	 * done by the caller to avoid the race with signal_wake_up().

	 */

	smp_mb__before_spinlock();

	rq_lock(rq, &rf);

	smp_mb__after_spinlock();



	/* Promote REQ to ACT */

	rq->clock_update_flags <<= 1;