Donate to e Foundation | Murena handsets with /e/OS | Own a part of Murena! Learn more

Commit 424acaae authored by Michel Lespinasse's avatar Michel Lespinasse Committed by Linus Torvalds
Browse files

rwsem: wake queued readers when writer blocks on active read lock 



This change addresses the following situation:

- Thread A acquires the rwsem for read
- Thread B tries to acquire the rwsem for write, notices there is already
  an active owner for the rwsem.
- Thread C tries to acquire the rwsem for read, notices that thread B already
  tried to acquire it.
- Thread C grabs the spinlock and queues itself on the wait queue.
- Thread B grabs the spinlock and queues itself behind C. At this point A is
  the only remaining active owner on the rwsem.

In this situation thread B could notice that it was the last active writer
on the rwsem, and decide to wake C to let it proceed in parallel with A
since they both only want the rwsem for read.

Signed-off-by: default avatarMichel Lespinasse <walken@google.com>
Acked-by: default avatarDavid Howells <dhowells@redhat.com>
Cc: Mike Waychison <mikew@google.com>
Cc: Suleiman Souhlal <suleiman@google.com>
Cc: Ying Han <yinghan@google.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
parent fd41b334

lib/rwsem.c

+15 −4
Original line number Original line Diff line number Diff line
@@ -67,6 +67,9 @@ __rwsem_do_wake(struct rw_semaphore *sem, int wake_type) 
		goto readers_only;

		goto readers_only;





	if (wake_type == RWSEM_WAKE_READ_OWNED)

	if (wake_type == RWSEM_WAKE_READ_OWNED)

		/* Another active reader was observed, so wakeup is not

		 * likely to succeed. Save the atomic op.

		 */

		goto out;

		goto out;





	/* There's a writer at the front of the queue - try to grant it the

	/* There's a writer at the front of the queue - try to grant it the
@@ -111,8 +114,8 @@ __rwsem_do_wake(struct rw_semaphore *sem, int wake_type) 
	 * count adjustment pretty soon.

	 * count adjustment pretty soon.

	 */

	 */

	if (wake_type == RWSEM_WAKE_ANY &&

	if (wake_type == RWSEM_WAKE_ANY &&

	    (rwsem_atomic_update(0, sem) & RWSEM_ACTIVE_MASK))

	    rwsem_atomic_update(0, sem) < RWSEM_WAITING_BIAS)

		/* Someone grabbed the sem already */

		/* Someone grabbed the sem for write already */

		goto out;

		goto out;





	/* Grant an infinite number of read locks to the readers at the front

	/* Grant an infinite number of read locks to the readers at the front
@@ -187,9 +190,17 @@ rwsem_down_failed_common(struct rw_semaphore *sem, 
	/* we're now waiting on the lock, but no longer actively locking */

	/* we're now waiting on the lock, but no longer actively locking */

	count = rwsem_atomic_update(adjustment, sem);

	count = rwsem_atomic_update(adjustment, sem);





	/* if there are no active locks, wake the front queued process(es) up */

	/* If there are no active locks, wake the front queued process(es) up.

	if (!(count & RWSEM_ACTIVE_MASK))

	 *

	 * Alternatively, if we're called from a failed down_write(), there

	 * were already threads queued before us and there are no active

	 * writers, the lock must be read owned; so we try to wake any read

	 * locks that were queued ahead of us. */

	if (count == RWSEM_WAITING_BIAS)

		sem = __rwsem_do_wake(sem, RWSEM_WAKE_NO_ACTIVE);

		sem = __rwsem_do_wake(sem, RWSEM_WAKE_NO_ACTIVE);

	else if (count > RWSEM_WAITING_BIAS &&

		 adjustment == -RWSEM_ACTIVE_WRITE_BIAS)

		sem = __rwsem_do_wake(sem, RWSEM_WAKE_READ_OWNED);





	spin_unlock_irq(&sem->wait_lock);

	spin_unlock_irq(&sem->wait_lock);