AuxXxilium/linux_dsm_epyc7002
1
0
Fork 0
mirror of https://github.com/AuxXxilium/linux_dsm_epyc7002.git synced 2024-11-25 12:50:51 +07:00
Code Issues Actions Packages Projects Releases Wiki Activity

locking/rwsem: Avoid deceiving lock spinners

Browse Source 
When readers hold the semaphore, the ->owner is nil. As such,
and unlike mutexes, '!owner' does not necessarily imply that
the lock is free. This will cause writers to potentially spin
excessively as they've been mislead to thinking they have a
chance of acquiring the lock, instead of blocking.

This patch therefore enhances the counter check when the owner
is not set by the time we've broken out of the loop. Otherwise
we can return true as a new owner has the lock and thus we want
to continue spinning. While at it, we can make rwsem_spin_on_owner()
less ambiguos and return right away under need_resched conditions.

Signed-off-by: Davidlohr Bueso <dbueso@suse.de>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Jason Low <jason.low2@hp.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Michel Lespinasse <walken@google.com>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Tim Chen <tim.c.chen@linux.intel.com>
Link: http://lkml.kernel.org/r/1422609267-15102-5-git-send-email-dave@stgolabs.net
Signed-off-by: Ingo Molnar <mingo@kernel.org>
This commit is contained in:
Davidlohr Bueso 2015-01-30 01:14:26 -08:00 committed by Ingo Molnar
parent 7a215f89a0
commit b3fd4f03ca
1 changed files with 15 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
kernel/locking/rwsem-xadd.c
View File

@ -337,21 +337,30 @@ static inline bool owner_running(struct rw_semaphore *sem,
static noinline
bool rwsem_spin_on_owner(struct rw_semaphore *sem, struct task_struct *owner)
{
long count;
rcu_read_lock();
while (owner_running(sem, owner)) {
if (need_resched())
break;
/* abort spinning when need_resched */
if (need_resched()) {
rcu_read_unlock();
return false;
}
cpu_relax_lowlatency();
}
rcu_read_unlock();
if (READ_ONCE(sem->owner))
return true; /* new owner, continue spinning */
/*
* We break out the loop above on need_resched() or when the
* owner changed, which is a sign for heavy contention. Return
* success only when sem->owner is NULL.
* When the owner is not set, the lock could be free or
* held by readers. Check the counter to verify the
* state.
*/
return sem->owner == NULL;
count = READ_ONCE(sem->count);
return (count == 0 || count == RWSEM_WAITING_BIAS);
}
static bool rwsem_optimistic_spin(struct rw_semaphore *sem)