Commit 1a993670 authored Jan 30, 2015 by Davidlohr Bueso Committed by Ingo Molnar Feb 18, 2015

locking/rwsem: Check for active lock before bailing on spinning



37e95624 ("locking/rwsem: Allow conservative optimistic
spinning when readers have lock") forced the default for
optimistic spinning to be disabled if the lock owner was
nil, which makes much sense for readers. However, while
it is not our priority, we can make some optimizations
for write-mostly workloads. We can bail the spinning step
and still be conservative if there are any active tasks,
otherwise there's really no reason not to spin, as the
semaphore is most likely unlocked.

This patch recovers most of a Unixbench 'execl' benchmark
throughput by sleeping less and making better average system
usage:

  before:
  CPU     %user     %nice   %system   %iowait    %steal     %idle
  all      0.60      0.00      8.02      0.00      0.00     91.38

  after:
  CPU     %user     %nice   %system   %iowait    %steal     %idle
  all      1.22      0.00     70.18      0.00      0.00     28.60

Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Jason Low <jason.low2@hp.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Michel Lespinasse <walken@google.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Link: http://lkml.kernel.org/r/1422609267-15102-6-git-send-email-dave@stgolabs.net


Signed-off-by: Ingo Molnar <mingo@kernel.org>

parent b3fd4f03

kernel/locking/rwsem-xadd.c

+17 −10

Original line number	Diff line number	Diff line
		@@ -298,23 +298,30 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
		static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
		{
		struct task_struct *owner;
		bool on_cpu = false;
		bool ret = true;

		if (need_resched())
		return false;

		rcu_read_lock();
		owner = ACCESS_ONCE(sem->owner);
		if (owner)
		on_cpu = owner->on_cpu;
		rcu_read_unlock();

		if (!owner) {
		long count = ACCESS_ONCE(sem->count);
		/*
		* If sem->owner is not set, yet we have just recently entered the
		* slowpath, then there is a possibility reader(s) may have the lock.
		* To be safe, avoid spinning in these situations.
		* slowpath with the lock being active, then there is a possibility
		* reader(s) may have the lock. To be safe, bail spinning in these
		* situations.
		*/
		return on_cpu;
		if (count & RWSEM_ACTIVE_MASK)
		ret = false;
		goto done;
		}

		ret = owner->on_cpu;
		done:
		rcu_read_unlock();
		return ret;
		}

		static inline bool owner_running(struct rw_semaphore *sem,