License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 21:07:57 +07:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* Implement CPU time clocks for the POSIX clock interface.
|
|
|
|
*/
|
|
|
|
|
2017-02-09 00:51:30 +07:00
|
|
|
#include <linux/sched/signal.h>
|
2017-02-05 17:48:36 +07:00
|
|
|
#include <linux/sched/cputime.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <linux/posix-timers.h>
|
|
|
|
#include <linux/errno.h>
|
2008-05-01 18:34:31 +07:00
|
|
|
#include <linux/math64.h>
|
2016-12-25 02:46:01 +07:00
|
|
|
#include <linux/uaccess.h>
|
2008-09-12 23:54:39 +07:00
|
|
|
#include <linux/kernel_stat.h>
|
2009-08-10 09:52:30 +07:00
|
|
|
#include <trace/events/timer.h>
|
2013-04-18 06:31:13 +07:00
|
|
|
#include <linux/tick.h>
|
|
|
|
#include <linux/workqueue.h>
|
2017-06-07 15:42:31 +07:00
|
|
|
#include <linux/compat.h>
|
2017-12-12 18:10:24 +07:00
|
|
|
#include <linux/sched/deadline.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2017-05-31 04:15:41 +07:00
|
|
|
#include "posix-timers.h"
|
|
|
|
|
2017-05-31 04:15:47 +07:00
|
|
|
static void posix_cpu_timer_rearm(struct k_itimer *timer);
|
|
|
|
|
2019-08-22 02:09:06 +07:00
|
|
|
void posix_cputimers_group_init(struct posix_cputimers *pct, u64 cpu_limit)
|
|
|
|
{
|
|
|
|
posix_cputimers_init(pct);
|
|
|
|
if (cpu_limit != RLIM_INFINITY)
|
2019-08-27 01:22:24 +07:00
|
|
|
pct->bases[CPUCLOCK_PROF].nextevt = cpu_limit * NSEC_PER_SEC;
|
2019-08-22 02:09:06 +07:00
|
|
|
}
|
|
|
|
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
/*
|
2010-03-12 05:04:37 +07:00
|
|
|
* Called after updating RLIMIT_CPU to run cpu timer and update
|
2019-08-27 01:22:24 +07:00
|
|
|
* tsk->signal->posix_cputimers.bases[clock].nextevt expiration cache if
|
|
|
|
* necessary. Needs siglock protection since other code may update the
|
2019-08-22 02:09:06 +07:00
|
|
|
* expiration cache as well.
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
*/
|
2009-08-28 19:05:12 +07:00
|
|
|
void update_rlimit_cpu(struct task_struct *task, unsigned long rlim_new)
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
{
|
2017-01-31 10:09:35 +07:00
|
|
|
u64 nsecs = rlim_new * NSEC_PER_SEC;
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
|
2009-08-28 19:05:12 +07:00
|
|
|
spin_lock_irq(&task->sighand->siglock);
|
2017-01-31 10:09:35 +07:00
|
|
|
set_process_cpu_timer(task, CPUCLOCK_PROF, &nsecs, NULL);
|
2009-08-28 19:05:12 +07:00
|
|
|
spin_unlock_irq(&task->sighand->siglock);
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
}
|
|
|
|
|
2019-08-22 02:08:48 +07:00
|
|
|
/*
|
|
|
|
* Functions for validating access to tasks.
|
|
|
|
*/
|
|
|
|
static struct task_struct *lookup_task(const pid_t pid, bool thread)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
struct task_struct *p;
|
|
|
|
|
2019-08-22 02:08:48 +07:00
|
|
|
if (!pid)
|
|
|
|
return thread ? current : current->group_leader;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-22 02:08:48 +07:00
|
|
|
p = find_task_by_vpid(pid);
|
|
|
|
if (!p || p == current)
|
|
|
|
return p;
|
|
|
|
if (thread)
|
|
|
|
return same_thread_group(p, current) ? p : NULL;
|
|
|
|
if (p == current)
|
|
|
|
return p;
|
|
|
|
return has_group_leader_pid(p) ? p : NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct task_struct *__get_task_for_clock(const clockid_t clock,
|
|
|
|
bool getref)
|
|
|
|
{
|
|
|
|
const bool thread = !!CPUCLOCK_PERTHREAD(clock);
|
|
|
|
const pid_t pid = CPUCLOCK_PID(clock);
|
|
|
|
struct task_struct *p;
|
|
|
|
|
|
|
|
if (CPUCLOCK_WHICH(clock) >= CPUCLOCK_MAX)
|
|
|
|
return NULL;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2010-11-03 23:52:56 +07:00
|
|
|
rcu_read_lock();
|
2019-08-22 02:08:48 +07:00
|
|
|
p = lookup_task(pid, thread);
|
|
|
|
if (p && getref)
|
|
|
|
get_task_struct(p);
|
2010-11-03 23:52:56 +07:00
|
|
|
rcu_read_unlock();
|
2019-08-22 02:08:48 +07:00
|
|
|
return p;
|
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-22 02:08:48 +07:00
|
|
|
static inline struct task_struct *get_task_for_clock(const clockid_t clock)
|
|
|
|
{
|
|
|
|
return __get_task_for_clock(clock, true);
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline int validate_clock_permissions(const clockid_t clock)
|
|
|
|
{
|
|
|
|
return __get_task_for_clock(clock, false) ? 0 : -EINVAL;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Update expiry time from increment, and increase overrun count,
|
|
|
|
* given the current clock sample.
|
|
|
|
*/
|
2017-01-31 10:09:34 +07:00
|
|
|
static void bump_cpu_timer(struct k_itimer *timer, u64 now)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
int i;
|
2017-01-31 10:09:34 +07:00
|
|
|
u64 delta, incr;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-01-11 20:33:17 +07:00
|
|
|
if (!timer->it_interval)
|
2005-04-17 05:20:36 +07:00
|
|
|
return;
|
|
|
|
|
2013-06-28 07:06:42 +07:00
|
|
|
if (now < timer->it.cpu.expires)
|
|
|
|
return;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-01-11 20:33:17 +07:00
|
|
|
incr = timer->it_interval;
|
2013-06-28 07:06:42 +07:00
|
|
|
delta = now + incr - timer->it.cpu.expires;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2013-06-28 07:06:42 +07:00
|
|
|
/* Don't use (incr*2 < delta), incr*2 might overflow. */
|
|
|
|
for (i = 0; incr < delta - incr; i++)
|
|
|
|
incr = incr << 1;
|
|
|
|
|
|
|
|
for (; i >= 0; incr >>= 1, i--) {
|
|
|
|
if (delta < incr)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
timer->it.cpu.expires += incr;
|
2018-06-26 20:21:32 +07:00
|
|
|
timer->it_overrun += 1LL << i;
|
2013-06-28 07:06:42 +07:00
|
|
|
delta -= incr;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
static inline bool expiry_cache_is_zero(const struct posix_cputimers *pct)
|
2013-04-19 21:17:38 +07:00
|
|
|
{
|
2019-08-27 01:22:24 +07:00
|
|
|
return !(pct->bases[CPUCLOCK_PROF].nextevt |
|
|
|
|
pct->bases[CPUCLOCK_VIRT].nextevt |
|
|
|
|
pct->bases[CPUCLOCK_SCHED].nextevt);
|
2013-04-19 21:17:38 +07:00
|
|
|
}
|
|
|
|
|
2011-02-01 20:52:12 +07:00
|
|
|
static int
|
2017-03-27 02:04:15 +07:00
|
|
|
posix_cpu_clock_getres(const clockid_t which_clock, struct timespec64 *tp)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-22 02:08:48 +07:00
|
|
|
int error = validate_clock_permissions(which_clock);
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
if (!error) {
|
|
|
|
tp->tv_sec = 0;
|
|
|
|
tp->tv_nsec = ((NSEC_PER_SEC + HZ - 1) / HZ);
|
|
|
|
if (CPUCLOCK_WHICH(which_clock) == CPUCLOCK_SCHED) {
|
|
|
|
/*
|
|
|
|
* If sched_clock is using a cycle counter, we
|
|
|
|
* don't have any idea of its true resolution
|
|
|
|
* exported, but it is much more than 1s/HZ.
|
|
|
|
*/
|
|
|
|
tp->tv_nsec = 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2011-02-01 20:52:12 +07:00
|
|
|
static int
|
2019-08-22 02:08:48 +07:00
|
|
|
posix_cpu_clock_set(const clockid_t clock, const struct timespec64 *tp)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-22 02:08:48 +07:00
|
|
|
int error = validate_clock_permissions(clock);
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* You can never reset a CPU clock, but we check for other errors
|
|
|
|
* in the call before failing with EPERM.
|
|
|
|
*/
|
2019-08-22 02:08:48 +07:00
|
|
|
return error ? : -EPERM;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2019-08-22 02:09:00 +07:00
|
|
|
* Sample a per-thread clock for the given task. clkid is validated.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2019-08-22 02:09:01 +07:00
|
|
|
static u64 cpu_clock_sample(const clockid_t clkid, struct task_struct *p)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-22 02:09:03 +07:00
|
|
|
u64 utime, stime;
|
|
|
|
|
|
|
|
if (clkid == CPUCLOCK_SCHED)
|
|
|
|
return task_sched_runtime(p);
|
|
|
|
|
|
|
|
task_cputime(p, &utime, &stime);
|
|
|
|
|
2019-08-22 02:09:00 +07:00
|
|
|
switch (clkid) {
|
2005-04-17 05:20:36 +07:00
|
|
|
case CPUCLOCK_PROF:
|
2019-08-22 02:09:03 +07:00
|
|
|
return utime + stime;
|
2005-04-17 05:20:36 +07:00
|
|
|
case CPUCLOCK_VIRT:
|
2019-08-22 02:09:03 +07:00
|
|
|
return utime;
|
2019-08-22 02:09:00 +07:00
|
|
|
default:
|
|
|
|
WARN_ON_ONCE(1);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2019-08-22 02:09:01 +07:00
|
|
|
return 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2019-08-22 02:09:12 +07:00
|
|
|
static inline void store_samples(u64 *samples, u64 stime, u64 utime, u64 rtime)
|
|
|
|
{
|
|
|
|
samples[CPUCLOCK_PROF] = stime + utime;
|
|
|
|
samples[CPUCLOCK_VIRT] = utime;
|
|
|
|
samples[CPUCLOCK_SCHED] = rtime;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void task_sample_cputime(struct task_struct *p, u64 *samples)
|
|
|
|
{
|
|
|
|
u64 stime, utime;
|
|
|
|
|
|
|
|
task_cputime(p, &utime, &stime);
|
|
|
|
store_samples(samples, stime, utime, p->se.sum_exec_runtime);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void proc_sample_cputime_atomic(struct task_cputime_atomic *at,
|
|
|
|
u64 *samples)
|
|
|
|
{
|
|
|
|
u64 stime, utime, rtime;
|
|
|
|
|
|
|
|
utime = atomic64_read(&at->utime);
|
|
|
|
stime = atomic64_read(&at->stime);
|
|
|
|
rtime = atomic64_read(&at->sum_exec_runtime);
|
|
|
|
store_samples(samples, stime, utime, rtime);
|
|
|
|
}
|
|
|
|
|
2015-04-29 03:00:22 +07:00
|
|
|
/*
|
|
|
|
* Set cputime to sum_cputime if sum_cputime > cputime. Use cmpxchg
|
|
|
|
* to avoid race conditions with concurrent updates to cputime.
|
|
|
|
*/
|
|
|
|
static inline void __update_gt_cputime(atomic64_t *cputime, u64 sum_cputime)
|
2009-02-11 17:30:27 +07:00
|
|
|
{
|
2015-04-29 03:00:22 +07:00
|
|
|
u64 curr_cputime;
|
|
|
|
retry:
|
|
|
|
curr_cputime = atomic64_read(cputime);
|
|
|
|
if (sum_cputime > curr_cputime) {
|
|
|
|
if (atomic64_cmpxchg(cputime, curr_cputime, sum_cputime) != curr_cputime)
|
|
|
|
goto retry;
|
|
|
|
}
|
|
|
|
}
|
2009-02-11 17:30:27 +07:00
|
|
|
|
2017-01-31 10:09:34 +07:00
|
|
|
static void update_gt_cputime(struct task_cputime_atomic *cputime_atomic, struct task_cputime *sum)
|
2015-04-29 03:00:22 +07:00
|
|
|
{
|
2015-04-29 03:00:24 +07:00
|
|
|
__update_gt_cputime(&cputime_atomic->utime, sum->utime);
|
|
|
|
__update_gt_cputime(&cputime_atomic->stime, sum->stime);
|
|
|
|
__update_gt_cputime(&cputime_atomic->sum_exec_runtime, sum->sum_exec_runtime);
|
2015-04-29 03:00:22 +07:00
|
|
|
}
|
2009-02-11 17:30:27 +07:00
|
|
|
|
2015-04-29 03:00:24 +07:00
|
|
|
/* Sample task_cputime_atomic values in "atomic_timers", store results in "times". */
|
2017-01-31 10:09:34 +07:00
|
|
|
static inline void sample_cputime_atomic(struct task_cputime *times,
|
2015-04-29 03:00:24 +07:00
|
|
|
struct task_cputime_atomic *atomic_times)
|
2015-04-29 03:00:22 +07:00
|
|
|
{
|
2015-04-29 03:00:24 +07:00
|
|
|
times->utime = atomic64_read(&atomic_times->utime);
|
|
|
|
times->stime = atomic64_read(&atomic_times->stime);
|
|
|
|
times->sum_exec_runtime = atomic64_read(&atomic_times->sum_exec_runtime);
|
2009-02-11 17:30:27 +07:00
|
|
|
}
|
|
|
|
|
2019-08-22 02:08:51 +07:00
|
|
|
/**
|
|
|
|
* thread_group_sample_cputime - Sample cputime for a given task
|
|
|
|
* @tsk: Task for which cputime needs to be started
|
|
|
|
* @iimes: Storage for time samples
|
|
|
|
*
|
|
|
|
* Called from sys_getitimer() to calculate the expiry time of an active
|
|
|
|
* timer. That means group cputime accounting is already active. Called
|
|
|
|
* with task sighand lock held.
|
|
|
|
*
|
|
|
|
* Updates @times with an uptodate sample of the thread group cputimes.
|
|
|
|
*/
|
|
|
|
void thread_group_sample_cputime(struct task_struct *tsk,
|
|
|
|
struct task_cputime *times)
|
|
|
|
{
|
|
|
|
struct thread_group_cputimer *cputimer = &tsk->signal->cputimer;
|
|
|
|
|
|
|
|
WARN_ON_ONCE(!cputimer->running);
|
|
|
|
|
|
|
|
sample_cputime_atomic(times, &cputimer->cputime_atomic);
|
|
|
|
}
|
|
|
|
|
2019-08-22 02:08:54 +07:00
|
|
|
/**
|
|
|
|
* thread_group_start_cputime - Start cputime and return a sample
|
|
|
|
* @tsk: Task for which cputime needs to be started
|
|
|
|
* @iimes: Storage for time samples
|
|
|
|
*
|
|
|
|
* The thread group cputime accouting is avoided when there are no posix
|
|
|
|
* CPU timers armed. Before starting a timer it's required to check whether
|
|
|
|
* the time accounting is active. If not, a full update of the atomic
|
|
|
|
* accounting store needs to be done and the accounting enabled.
|
|
|
|
*
|
|
|
|
* Updates @times with an uptodate sample of the thread group cputimes.
|
|
|
|
*/
|
|
|
|
static void
|
|
|
|
thread_group_start_cputime(struct task_struct *tsk, struct task_cputime *times)
|
2009-02-11 17:30:27 +07:00
|
|
|
{
|
|
|
|
struct thread_group_cputimer *cputimer = &tsk->signal->cputimer;
|
2017-01-31 10:09:34 +07:00
|
|
|
struct task_cputime sum;
|
2009-02-11 17:30:27 +07:00
|
|
|
|
2015-04-29 03:00:22 +07:00
|
|
|
/* Check if cputimer isn't running. This is accessed without locking. */
|
|
|
|
if (!READ_ONCE(cputimer->running)) {
|
2009-02-11 17:30:27 +07:00
|
|
|
/*
|
|
|
|
* The POSIX timer interface allows for absolute time expiry
|
|
|
|
* values through the TIMER_ABSTIME flag, therefore we have
|
2015-04-29 03:00:22 +07:00
|
|
|
* to synchronize the timer to the clock every time we start it.
|
2009-02-11 17:30:27 +07:00
|
|
|
*/
|
2017-01-31 10:09:34 +07:00
|
|
|
thread_group_cputime(tsk, &sum);
|
2015-04-29 03:00:24 +07:00
|
|
|
update_gt_cputime(&cputimer->cputime_atomic, &sum);
|
2015-04-29 03:00:22 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We're setting cputimer->running without a lock. Ensure
|
|
|
|
* this only gets written to in one operation. We set
|
|
|
|
* running after update_gt_cputime() as a small optimization,
|
|
|
|
* but barriers are not required because update_gt_cputime()
|
|
|
|
* can handle concurrent updates.
|
|
|
|
*/
|
2015-10-15 02:07:55 +07:00
|
|
|
WRITE_ONCE(cputimer->running, true);
|
2015-04-29 03:00:22 +07:00
|
|
|
}
|
2015-04-29 03:00:24 +07:00
|
|
|
sample_cputime_atomic(times, &cputimer->cputime_atomic);
|
2009-02-11 17:30:27 +07:00
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
2019-08-22 02:08:55 +07:00
|
|
|
* Sample a process (thread group) clock for the given task clkid. If the
|
|
|
|
* group's cputime accounting is already enabled, read the atomic
|
|
|
|
* store. Otherwise a full update is required. Task's sighand lock must be
|
2019-08-22 02:09:00 +07:00
|
|
|
* held to protect the task traversal on a full update. clkid is already
|
|
|
|
* validated.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2019-08-22 02:09:01 +07:00
|
|
|
static u64 cpu_clock_sample_group(const clockid_t clkid, struct task_struct *p,
|
|
|
|
bool start)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-22 02:08:55 +07:00
|
|
|
struct thread_group_cputimer *cputimer = &p->signal->cputimer;
|
2017-01-31 10:09:34 +07:00
|
|
|
struct task_cputime cputime;
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
|
2019-08-22 02:08:55 +07:00
|
|
|
if (!READ_ONCE(cputimer->running)) {
|
|
|
|
if (start)
|
|
|
|
thread_group_start_cputime(p, &cputime);
|
|
|
|
else
|
|
|
|
thread_group_cputime(p, &cputime);
|
|
|
|
} else {
|
|
|
|
sample_cputime_atomic(&cputime, &cputimer->cputime_atomic);
|
|
|
|
}
|
|
|
|
|
2019-08-22 02:09:00 +07:00
|
|
|
switch (clkid) {
|
2005-04-17 05:20:36 +07:00
|
|
|
case CPUCLOCK_PROF:
|
2019-08-22 02:09:01 +07:00
|
|
|
return cputime.utime + cputime.stime;
|
2005-04-17 05:20:36 +07:00
|
|
|
case CPUCLOCK_VIRT:
|
2019-08-22 02:09:01 +07:00
|
|
|
return cputime.utime;
|
2005-04-17 05:20:36 +07:00
|
|
|
case CPUCLOCK_SCHED:
|
2019-08-22 02:09:01 +07:00
|
|
|
return cputime.sum_exec_runtime;
|
2019-08-22 02:09:00 +07:00
|
|
|
default:
|
|
|
|
WARN_ON_ONCE(1);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2019-08-22 02:09:01 +07:00
|
|
|
return 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2019-08-22 02:08:49 +07:00
|
|
|
static int posix_cpu_clock_get(const clockid_t clock, struct timespec64 *tp)
|
2013-10-11 22:41:11 +07:00
|
|
|
{
|
2019-08-22 02:08:49 +07:00
|
|
|
const clockid_t clkid = CPUCLOCK_WHICH(clock);
|
|
|
|
struct task_struct *tsk;
|
|
|
|
u64 t;
|
2013-10-11 22:41:11 +07:00
|
|
|
|
2019-08-22 02:08:49 +07:00
|
|
|
tsk = get_task_for_clock(clock);
|
|
|
|
if (!tsk)
|
|
|
|
return -EINVAL;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-22 02:08:49 +07:00
|
|
|
if (CPUCLOCK_PERTHREAD(clock))
|
2019-08-22 02:09:01 +07:00
|
|
|
t = cpu_clock_sample(clkid, tsk);
|
2019-08-22 02:08:49 +07:00
|
|
|
else
|
2019-08-22 02:09:01 +07:00
|
|
|
t = cpu_clock_sample_group(clkid, tsk, false);
|
2019-08-22 02:08:49 +07:00
|
|
|
put_task_struct(tsk);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-22 02:08:49 +07:00
|
|
|
*tp = ns_to_timespec64(t);
|
|
|
|
return 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Validate the clockid_t for a new CPU-clock timer, and initialize the timer.
|
2009-11-18 05:14:13 +07:00
|
|
|
* This is called from sys_timer_create() and do_cpu_nanosleep() with the
|
|
|
|
* new timer already all-zeros initialized.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2011-02-01 20:52:12 +07:00
|
|
|
static int posix_cpu_timer_create(struct k_itimer *new_timer)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-22 02:08:50 +07:00
|
|
|
struct task_struct *p = get_task_for_clock(new_timer->it_clock);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-22 02:08:50 +07:00
|
|
|
if (!p)
|
2005-04-17 05:20:36 +07:00
|
|
|
return -EINVAL;
|
|
|
|
|
2017-05-31 04:15:44 +07:00
|
|
|
new_timer->kclock = &clock_posix_cpu;
|
2005-04-17 05:20:36 +07:00
|
|
|
INIT_LIST_HEAD(&new_timer->it.cpu.entry);
|
|
|
|
new_timer->it.cpu.task = p;
|
2019-08-22 02:08:50 +07:00
|
|
|
return 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Clean up a CPU-clock timer that is about to be destroyed.
|
|
|
|
* This is called from timer deletion with the timer already locked.
|
|
|
|
* If we return TIMER_RETRY, it's necessary to release the timer's lock
|
|
|
|
* and try again. (This happens when the timer is in the middle of firing.)
|
|
|
|
*/
|
2011-02-01 20:52:12 +07:00
|
|
|
static int posix_cpu_timer_del(struct k_itimer *timer)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2005-10-23 23:25:39 +07:00
|
|
|
int ret = 0;
|
2013-10-11 22:41:11 +07:00
|
|
|
unsigned long flags;
|
|
|
|
struct sighand_struct *sighand;
|
|
|
|
struct task_struct *p = timer->it.cpu.task;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-19 21:31:46 +07:00
|
|
|
if (WARN_ON_ONCE(!p))
|
|
|
|
return -EINVAL;
|
2005-10-23 23:25:39 +07:00
|
|
|
|
2013-10-11 22:41:11 +07:00
|
|
|
/*
|
|
|
|
* Protect against sighand release/switch in exit/exec and process/
|
|
|
|
* thread timer list entry concurrent read/writes.
|
|
|
|
*/
|
|
|
|
sighand = lock_task_sighand(p, &flags);
|
|
|
|
if (unlikely(sighand == NULL)) {
|
2013-10-11 05:37:39 +07:00
|
|
|
/*
|
|
|
|
* We raced with the reaping of the task.
|
|
|
|
* The deletion should have cleared us off the list.
|
|
|
|
*/
|
2013-10-11 22:58:08 +07:00
|
|
|
WARN_ON_ONCE(!list_empty(&timer->it.cpu.entry));
|
2013-10-11 05:37:39 +07:00
|
|
|
} else {
|
|
|
|
if (timer->it.cpu.firing)
|
|
|
|
ret = TIMER_RETRY;
|
|
|
|
else
|
|
|
|
list_del(&timer->it.cpu.entry);
|
2013-10-11 22:41:11 +07:00
|
|
|
|
|
|
|
unlock_task_sighand(p, &flags);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2013-10-11 05:37:39 +07:00
|
|
|
|
|
|
|
if (!ret)
|
|
|
|
put_task_struct(p);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2005-10-23 23:25:39 +07:00
|
|
|
return ret;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2013-10-11 21:11:43 +07:00
|
|
|
static void cleanup_timers_list(struct list_head *head)
|
2013-06-28 07:06:42 +07:00
|
|
|
{
|
|
|
|
struct cpu_timer_list *timer, *next;
|
|
|
|
|
posix_timers: fix racy timer delta caching on task exit
When a task exits, we perform a caching of the remaining cputime delta
before expiring of its timers.
This is done from the following places:
* When the task is reaped. We iterate through its list of
posix cpu timers and store the remaining timer delta to
the timer struct instead of the absolute value.
(See posix_cpu_timers_exit() / posix_cpu_timers_exit_group() )
* When we call posix_cpu_timer_get() or posix_cpu_timer_schedule().
If the timer's task is considered dying when watched from these
places, the same conversion from absolute to relative expiry time
is performed. Then the given task's reference is released.
(See clear_dead_task() ).
The relevance of this caching is questionable but this is another
and deeper debate.
The big issue here is that these two sources of caching don't mix
up very well together.
More specifically, the caching can easily be done twice, resulting
in a wrong delta as it gets spuriously substracted a second time by
the elapsed clock. This can happen in the following scenario:
1) The task exits and gets reaped: we call posix_cpu_timers_exit()
and the absolute timer expiry values are converted to a relative
delta.
2) timer_gettime() -> posix_cpu_timer_get() is called and relies on
clear_dead_task() because tsk->exit_state == EXIT_DEAD.
The delta gets substracted again by the elapsed clock and we return
a wrong result.
To fix this, just remove the caching done on task reaping time. It
doesn't bring much value on its own. The caching done from
posix_cpu_timer_get/schedule is enough.
And it would also be hard to get it really right: we could make it put and
clear the target task in the timer struct so that readers know if they are
dealing with a relative cached of absolute value. But it would be racy.
The only safe way to do it would be to lock the itimer->it_lock so that we
know nobody reads the cputime expiry value while we modify it and its
target task reference. Doing so would involve some funny workarounds to
avoid circular lock against the sighand lock. There is just no reason to
maintain this.
The user visible effect of this patch can be observed by running the
following code: it creates a subthread that launches a posix cputimer
which expires after 10 seconds. But then the subthread only busy loops for 2
seconds and exits. The parent reaps the subthread and read the timer value.
Its expected value should the be the initial timer's expiration value
minus the cputime elapsed in the subthread. Roughly 10 - 2 = 8 seconds:
#include <sys/time.h>
#include <stdio.h>
#include <unistd.h>
#include <time.h>
#include <pthread.h>
static timer_t id;
static struct itimerspec val = { .it_value.tv_sec = 10, }, new;
static void *thread(void *unused)
{
int err;
struct timeval start, end, diff;
timer_create(CLOCK_THREAD_CPUTIME_ID, NULL, &id);
if (err < 0) {
perror("Can't create timer\n");
return NULL;
}
/* Arm 10 sec timer */
err = timer_settime(id, 0, &val, NULL);
if (err < 0) {
perror("Can't set timer\n");
return NULL;
}
/* Exit after 2 seconds of execution */
gettimeofday(&start, NULL);
do {
gettimeofday(&end, NULL);
timersub(&end, &start, &diff);
} while (diff.tv_sec < 2);
return NULL;
}
int main(int argc, char **argv)
{
pthread_t pthread;
int err;
err = pthread_create(&pthread, NULL, thread, NULL);
if (err) {
perror("Can't create thread\n");
return -1;
}
pthread_join(pthread, NULL);
/* Just wait a little bit to make sure the child got reaped */
sleep(1);
err = timer_gettime(id, &new);
if (err)
perror("Can't get timer value\n");
printf("%d %ld\n", new.it_value.tv_sec, new.it_value.tv_nsec);
return 0;
}
Before the patch:
$ ./posix_cpu_timers
6 2278074
After the patch:
$ ./posix_cpu_timers
8 1158766
Before the patch, the elapsed time got two more seconds spuriously accounted.
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Stanislaw Gruszka <sgruszka@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@gmail.com>
Cc: Olivier Langlois <olivier@trillion01.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2013-06-28 07:06:43 +07:00
|
|
|
list_for_each_entry_safe(timer, next, head, entry)
|
2013-06-28 07:06:42 +07:00
|
|
|
list_del_init(&timer->entry);
|
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
2019-08-19 21:31:45 +07:00
|
|
|
* Clean out CPU timers which are still armed when a thread exits. The
|
|
|
|
* timers are only removed from the list. No other updates are done. The
|
|
|
|
* corresponding posix timers are still accessible, but cannot be rearmed.
|
|
|
|
*
|
2005-04-17 05:20:36 +07:00
|
|
|
* This must be called with the siglock held.
|
|
|
|
*/
|
2019-08-22 02:09:04 +07:00
|
|
|
static void cleanup_timers(struct posix_cputimers *pct)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-27 01:22:24 +07:00
|
|
|
cleanup_timers_list(&pct->bases[CPUCLOCK_PROF].cpu_timers);
|
|
|
|
cleanup_timers_list(&pct->bases[CPUCLOCK_VIRT].cpu_timers);
|
|
|
|
cleanup_timers_list(&pct->bases[CPUCLOCK_SCHED].cpu_timers);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* These are both called with the siglock held, when the current thread
|
|
|
|
* is being reaped. When the final (leader) thread in the group is reaped,
|
|
|
|
* posix_cpu_timers_exit_group will be called after posix_cpu_timers_exit.
|
|
|
|
*/
|
|
|
|
void posix_cpu_timers_exit(struct task_struct *tsk)
|
|
|
|
{
|
2019-08-22 02:09:04 +07:00
|
|
|
cleanup_timers(&tsk->posix_cputimers);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
void posix_cpu_timers_exit_group(struct task_struct *tsk)
|
|
|
|
{
|
2019-08-22 02:09:04 +07:00
|
|
|
cleanup_timers(&tsk->signal->posix_cputimers);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2017-01-31 10:09:34 +07:00
|
|
|
static inline int expires_gt(u64 expires, u64 new_exp)
|
2009-07-29 17:15:28 +07:00
|
|
|
{
|
2011-12-15 20:56:09 +07:00
|
|
|
return expires == 0 || expires > new_exp;
|
2009-07-29 17:15:28 +07:00
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* Insert the timer on the appropriate list before any timers that
|
2013-10-11 23:56:49 +07:00
|
|
|
* expire later. This must be called with the sighand lock held.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2010-03-12 05:04:38 +07:00
|
|
|
static void arm_timer(struct k_itimer *timer)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-22 02:09:08 +07:00
|
|
|
struct cpu_timer_list *const nt = &timer->it.cpu;
|
|
|
|
int clkidx = CPUCLOCK_WHICH(timer->it_clock);
|
2005-04-17 05:20:36 +07:00
|
|
|
struct task_struct *p = timer->it.cpu.task;
|
2019-08-27 01:22:24 +07:00
|
|
|
u64 newexp = timer->it.cpu.expires;
|
|
|
|
struct posix_cputimer_base *base;
|
2005-04-17 05:20:36 +07:00
|
|
|
struct list_head *head, *listpos;
|
|
|
|
struct cpu_timer_list *next;
|
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
if (CPUCLOCK_PERTHREAD(timer->it_clock))
|
|
|
|
base = p->posix_cputimers.bases + clkidx;
|
|
|
|
else
|
|
|
|
base = p->signal->posix_cputimers.bases + clkidx;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
listpos = head = &base->cpu_timers;
|
|
|
|
list_for_each_entry(next,head, entry) {
|
2013-06-28 07:06:42 +07:00
|
|
|
if (nt->expires < next->expires)
|
2010-03-12 05:04:38 +07:00
|
|
|
break;
|
|
|
|
listpos = &next->entry;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
list_add(&nt->entry, listpos);
|
|
|
|
|
2019-08-22 02:09:08 +07:00
|
|
|
if (listpos != head)
|
|
|
|
return;
|
2010-03-12 05:04:38 +07:00
|
|
|
|
2019-08-22 02:09:08 +07:00
|
|
|
/*
|
|
|
|
* We are the new earliest-expiring POSIX 1.b timer, hence
|
|
|
|
* need to update expiration cache. Take into account that
|
|
|
|
* for process timers we share expiration cache with itimers
|
|
|
|
* and RLIMIT_CPU and for thread timers with RLIMIT_RTTIME.
|
|
|
|
*/
|
2019-08-27 01:22:24 +07:00
|
|
|
if (expires_gt(base->nextevt, newexp))
|
|
|
|
base->nextevt = newexp;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-22 02:09:08 +07:00
|
|
|
if (CPUCLOCK_PERTHREAD(timer->it_clock))
|
|
|
|
tick_dep_set_task(p, TICK_DEP_BIT_POSIX_TIMER);
|
|
|
|
else
|
|
|
|
tick_dep_set_signal(p->signal, TICK_DEP_BIT_POSIX_TIMER);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The timer is locked, fire it and arrange for its reload.
|
|
|
|
*/
|
|
|
|
static void cpu_timer_fire(struct k_itimer *timer)
|
|
|
|
{
|
cpu-timers: Change SIGEV_NONE timer implementation
When user sets up a timer without associated signal and process does
not use any other cpu timers and does not exit, tsk->signal->cputimer
is enabled and running forever.
Avoid running the timer for no reason.
I used below program to check patch does not break current user space
visible behavior.
#include <sys/time.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <assert.h>
void consume_cpu(void)
{
int i = 0;
int count = 0;
for(i=0; i<100000000; i++)
count++;
}
int main(void)
{
int i;
struct sigaction act;
struct sigevent evt = { };
timer_t tid;
struct itimerspec spec = { };
evt.sigev_notify = SIGEV_NONE;
assert(timer_create(CLOCK_PROCESS_CPUTIME_ID, &evt, &tid) == 0);
spec.it_value.tv_sec = 10;
assert(timer_settime(tid, 0, &spec, NULL) == 0);
for (i = 0; i < 30; i++) {
consume_cpu();
memset(&spec, 0, sizeof(spec));
assert(timer_gettime(tid, &spec) == 0);
printf("%lu.%09lu\n",
(unsigned long) spec.it_value.tv_sec,
(unsigned long) spec.it_value.tv_nsec);
}
assert(timer_delete(tid) == 0);
return 0;
}
Signed-off-by: Stanislaw Gruszka <sgruszka@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
Cc: Balbir Singh <balbir@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
2010-03-12 05:04:41 +07:00
|
|
|
if ((timer->it_sigev_notify & ~SIGEV_THREAD_ID) == SIGEV_NONE) {
|
|
|
|
/*
|
|
|
|
* User don't want any signal.
|
|
|
|
*/
|
2013-06-28 07:06:42 +07:00
|
|
|
timer->it.cpu.expires = 0;
|
cpu-timers: Change SIGEV_NONE timer implementation
When user sets up a timer without associated signal and process does
not use any other cpu timers and does not exit, tsk->signal->cputimer
is enabled and running forever.
Avoid running the timer for no reason.
I used below program to check patch does not break current user space
visible behavior.
#include <sys/time.h>
#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <assert.h>
void consume_cpu(void)
{
int i = 0;
int count = 0;
for(i=0; i<100000000; i++)
count++;
}
int main(void)
{
int i;
struct sigaction act;
struct sigevent evt = { };
timer_t tid;
struct itimerspec spec = { };
evt.sigev_notify = SIGEV_NONE;
assert(timer_create(CLOCK_PROCESS_CPUTIME_ID, &evt, &tid) == 0);
spec.it_value.tv_sec = 10;
assert(timer_settime(tid, 0, &spec, NULL) == 0);
for (i = 0; i < 30; i++) {
consume_cpu();
memset(&spec, 0, sizeof(spec));
assert(timer_gettime(tid, &spec) == 0);
printf("%lu.%09lu\n",
(unsigned long) spec.it_value.tv_sec,
(unsigned long) spec.it_value.tv_nsec);
}
assert(timer_delete(tid) == 0);
return 0;
}
Signed-off-by: Stanislaw Gruszka <sgruszka@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Hidetoshi Seto <seto.hidetoshi@jp.fujitsu.com>
Cc: Balbir Singh <balbir@in.ibm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
2010-03-12 05:04:41 +07:00
|
|
|
} else if (unlikely(timer->sigq == NULL)) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* This a special case for clock_nanosleep,
|
|
|
|
* not a normal timer from sys_timer_create.
|
|
|
|
*/
|
|
|
|
wake_up_process(timer->it_process);
|
2013-06-28 07:06:42 +07:00
|
|
|
timer->it.cpu.expires = 0;
|
2019-01-11 20:33:17 +07:00
|
|
|
} else if (!timer->it_interval) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* One-shot timer. Clear it as soon as it's fired.
|
|
|
|
*/
|
|
|
|
posix_timer_event(timer, 0);
|
2013-06-28 07:06:42 +07:00
|
|
|
timer->it.cpu.expires = 0;
|
2005-04-17 05:20:36 +07:00
|
|
|
} else if (posix_timer_event(timer, ++timer->it_requeue_pending)) {
|
|
|
|
/*
|
|
|
|
* The signal did not get queued because the signal
|
|
|
|
* was ignored, so we won't get any callback to
|
|
|
|
* reload the timer. But we need to keep it
|
|
|
|
* ticking in case the signal is deliverable next time.
|
|
|
|
*/
|
2017-05-31 04:15:47 +07:00
|
|
|
posix_cpu_timer_rearm(timer);
|
2017-05-31 04:15:42 +07:00
|
|
|
++timer->it_requeue_pending;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Guts of sys_timer_settime for CPU timers.
|
|
|
|
* This is called with the timer locked and interrupts disabled.
|
|
|
|
* If we return TIMER_RETRY, it's necessary to release the timer's lock
|
|
|
|
* and try again. (This happens when the timer is in the middle of firing.)
|
|
|
|
*/
|
2013-10-11 23:56:49 +07:00
|
|
|
static int posix_cpu_timer_set(struct k_itimer *timer, int timer_flags,
|
2017-03-27 02:04:17 +07:00
|
|
|
struct itimerspec64 *new, struct itimerspec64 *old)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-22 02:08:56 +07:00
|
|
|
clockid_t clkid = CPUCLOCK_WHICH(timer->it_clock);
|
2017-01-31 10:09:34 +07:00
|
|
|
u64 old_expires, new_expires, old_incr, val;
|
2019-08-22 02:08:56 +07:00
|
|
|
struct task_struct *p = timer->it.cpu.task;
|
|
|
|
struct sighand_struct *sighand;
|
|
|
|
unsigned long flags;
|
2005-04-17 05:20:36 +07:00
|
|
|
int ret;
|
|
|
|
|
2019-08-19 21:31:46 +07:00
|
|
|
if (WARN_ON_ONCE(!p))
|
|
|
|
return -EINVAL;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2017-06-20 22:37:36 +07:00
|
|
|
/*
|
|
|
|
* Use the to_ktime conversion because that clamps the maximum
|
|
|
|
* value to KTIME_MAX and avoid multiplication overflows.
|
|
|
|
*/
|
|
|
|
new_expires = ktime_to_ns(timespec64_to_ktime(new->it_value));
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/*
|
2013-10-11 23:56:49 +07:00
|
|
|
* Protect against sighand release/switch in exit/exec and p->cpu_timers
|
|
|
|
* and p->signal->cpu_timers read/write in arm_timer()
|
|
|
|
*/
|
|
|
|
sighand = lock_task_sighand(p, &flags);
|
|
|
|
/*
|
|
|
|
* If p has just been reaped, we can no
|
2005-04-17 05:20:36 +07:00
|
|
|
* longer get any information about it at all.
|
|
|
|
*/
|
2013-10-11 23:56:49 +07:00
|
|
|
if (unlikely(sighand == NULL)) {
|
2005-04-17 05:20:36 +07:00
|
|
|
return -ESRCH;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Disarm any old timer after extracting its expiry time.
|
|
|
|
*/
|
2005-10-24 21:29:58 +07:00
|
|
|
|
|
|
|
ret = 0;
|
2019-01-11 20:33:17 +07:00
|
|
|
old_incr = timer->it_interval;
|
2005-04-17 05:20:36 +07:00
|
|
|
old_expires = timer->it.cpu.expires;
|
2005-10-24 21:29:58 +07:00
|
|
|
if (unlikely(timer->it.cpu.firing)) {
|
|
|
|
timer->it.cpu.firing = -1;
|
|
|
|
ret = TIMER_RETRY;
|
|
|
|
} else
|
|
|
|
list_del_init(&timer->it.cpu.entry);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We need to sample the current value to convert the new
|
|
|
|
* value from to relative and absolute, and to convert the
|
|
|
|
* old value from absolute to relative. To set a process
|
|
|
|
* timer, we need a sample to balance the thread expiry
|
|
|
|
* times (in arm_timer). With an absolute time, we must
|
|
|
|
* check if it's already passed. In short, we need a sample.
|
|
|
|
*/
|
2019-08-22 02:09:01 +07:00
|
|
|
if (CPUCLOCK_PERTHREAD(timer->it_clock))
|
|
|
|
val = cpu_clock_sample(clkid, p);
|
|
|
|
else
|
|
|
|
val = cpu_clock_sample_group(clkid, p, true);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
if (old) {
|
2013-06-28 07:06:42 +07:00
|
|
|
if (old_expires == 0) {
|
2005-04-17 05:20:36 +07:00
|
|
|
old->it_value.tv_sec = 0;
|
|
|
|
old->it_value.tv_nsec = 0;
|
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* Update the timer in case it has
|
|
|
|
* overrun already. If it has,
|
|
|
|
* we'll report it as having overrun
|
|
|
|
* and with the next reloaded timer
|
|
|
|
* already ticking, though we are
|
|
|
|
* swallowing that pending
|
|
|
|
* notification here to install the
|
|
|
|
* new setting.
|
|
|
|
*/
|
|
|
|
bump_cpu_timer(timer, val);
|
2013-06-28 07:06:42 +07:00
|
|
|
if (val < timer->it.cpu.expires) {
|
|
|
|
old_expires = timer->it.cpu.expires - val;
|
2017-03-27 02:04:17 +07:00
|
|
|
old->it_value = ns_to_timespec64(old_expires);
|
2005-04-17 05:20:36 +07:00
|
|
|
} else {
|
|
|
|
old->it_value.tv_nsec = 1;
|
|
|
|
old->it_value.tv_sec = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-10-24 21:29:58 +07:00
|
|
|
if (unlikely(ret)) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* We are colliding with the timer actually firing.
|
|
|
|
* Punt after filling in the timer's old value, and
|
|
|
|
* disable this firing since we are already reporting
|
|
|
|
* it as an overrun (thanks to bump_cpu_timer above).
|
|
|
|
*/
|
2013-10-11 23:56:49 +07:00
|
|
|
unlock_task_sighand(p, &flags);
|
2005-04-17 05:20:36 +07:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
2013-10-11 23:56:49 +07:00
|
|
|
if (new_expires != 0 && !(timer_flags & TIMER_ABSTIME)) {
|
2013-06-28 07:06:42 +07:00
|
|
|
new_expires += val;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Install the new expiry time (or zero).
|
|
|
|
* For a timer with no notification action, we don't actually
|
|
|
|
* arm the timer (we'll just fake it for timer_gettime).
|
|
|
|
*/
|
|
|
|
timer->it.cpu.expires = new_expires;
|
2013-06-28 07:06:42 +07:00
|
|
|
if (new_expires != 0 && val < new_expires) {
|
2010-03-12 05:04:38 +07:00
|
|
|
arm_timer(timer);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2013-10-11 23:56:49 +07:00
|
|
|
unlock_task_sighand(p, &flags);
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* Install the new reload setting, and
|
|
|
|
* set up the signal and overrun bookkeeping.
|
|
|
|
*/
|
2019-01-11 20:33:17 +07:00
|
|
|
timer->it_interval = timespec64_to_ktime(new->it_interval);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* This acts as a modification timestamp for the timer,
|
|
|
|
* so any automatic reload attempt will punt on seeing
|
|
|
|
* that we have reset the timer manually.
|
|
|
|
*/
|
|
|
|
timer->it_requeue_pending = (timer->it_requeue_pending + 2) &
|
|
|
|
~REQUEUE_PENDING;
|
|
|
|
timer->it_overrun_last = 0;
|
|
|
|
timer->it_overrun = -1;
|
|
|
|
|
2013-06-28 07:06:42 +07:00
|
|
|
if (new_expires != 0 && !(val < new_expires)) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* The designated time already passed, so we notify
|
|
|
|
* immediately, even if the thread never runs to
|
|
|
|
* accumulate more time on this clock.
|
|
|
|
*/
|
|
|
|
cpu_timer_fire(timer);
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = 0;
|
|
|
|
out:
|
2017-01-31 10:09:34 +07:00
|
|
|
if (old)
|
2017-03-27 02:04:17 +07:00
|
|
|
old->it_interval = ns_to_timespec64(old_incr);
|
2015-07-18 03:25:49 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2017-03-27 02:04:17 +07:00
|
|
|
static void posix_cpu_timer_get(struct k_itimer *timer, struct itimerspec64 *itp)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-22 02:08:57 +07:00
|
|
|
clockid_t clkid = CPUCLOCK_WHICH(timer->it_clock);
|
2005-04-17 05:20:36 +07:00
|
|
|
struct task_struct *p = timer->it.cpu.task;
|
2019-08-19 21:31:46 +07:00
|
|
|
u64 now;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-19 21:31:46 +07:00
|
|
|
if (WARN_ON_ONCE(!p))
|
|
|
|
return;
|
2013-10-11 05:37:39 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* Easy part: convert the reload time.
|
|
|
|
*/
|
2019-01-11 20:33:17 +07:00
|
|
|
itp->it_interval = ktime_to_timespec64(timer->it_interval);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2017-05-31 04:15:51 +07:00
|
|
|
if (!timer->it.cpu.expires)
|
2005-04-17 05:20:36 +07:00
|
|
|
return;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Sample the clock to take the difference with the expiry time.
|
|
|
|
*/
|
|
|
|
if (CPUCLOCK_PERTHREAD(timer->it_clock)) {
|
2019-08-22 02:09:01 +07:00
|
|
|
now = cpu_clock_sample(clkid, p);
|
2005-04-17 05:20:36 +07:00
|
|
|
} else {
|
2013-10-11 23:56:49 +07:00
|
|
|
struct sighand_struct *sighand;
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Protect against sighand release/switch in exit/exec and
|
|
|
|
* also make timer sampling safe if it ends up calling
|
2017-01-31 10:09:34 +07:00
|
|
|
* thread_group_cputime().
|
2013-10-11 23:56:49 +07:00
|
|
|
*/
|
|
|
|
sighand = lock_task_sighand(p, &flags);
|
|
|
|
if (unlikely(sighand == NULL)) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* The process has been reaped.
|
|
|
|
* We can't even collect a sample any more.
|
|
|
|
* Call the timer disarmed, nothing else to do.
|
|
|
|
*/
|
2013-06-28 07:06:42 +07:00
|
|
|
timer->it.cpu.expires = 0;
|
2016-07-08 05:39:11 +07:00
|
|
|
return;
|
2005-04-17 05:20:36 +07:00
|
|
|
} else {
|
2019-08-22 02:09:01 +07:00
|
|
|
now = cpu_clock_sample_group(clkid, p, false);
|
2013-10-11 23:56:49 +07:00
|
|
|
unlock_task_sighand(p, &flags);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-06-28 07:06:42 +07:00
|
|
|
if (now < timer->it.cpu.expires) {
|
2017-03-27 02:04:17 +07:00
|
|
|
itp->it_value = ns_to_timespec64(timer->it.cpu.expires - now);
|
2005-04-17 05:20:36 +07:00
|
|
|
} else {
|
|
|
|
/*
|
|
|
|
* The timer should have expired already, but the firing
|
|
|
|
* hasn't taken place yet. Say it's just about to expire.
|
|
|
|
*/
|
|
|
|
itp->it_value.tv_nsec = 1;
|
|
|
|
itp->it_value.tv_sec = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-06-28 07:06:43 +07:00
|
|
|
static unsigned long long
|
|
|
|
check_timers_list(struct list_head *timers,
|
|
|
|
struct list_head *firing,
|
|
|
|
unsigned long long curr)
|
|
|
|
{
|
|
|
|
int maxfire = 20;
|
|
|
|
|
|
|
|
while (!list_empty(timers)) {
|
|
|
|
struct cpu_timer_list *t;
|
|
|
|
|
|
|
|
t = list_first_entry(timers, struct cpu_timer_list, entry);
|
|
|
|
|
|
|
|
if (!--maxfire || curr < t->expires)
|
|
|
|
return t->expires;
|
|
|
|
|
|
|
|
t->firing = 1;
|
|
|
|
list_move_tail(&t->entry, firing);
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2017-12-12 18:10:24 +07:00
|
|
|
static inline void check_dl_overrun(struct task_struct *tsk)
|
|
|
|
{
|
|
|
|
if (tsk->dl.dl_overrun) {
|
|
|
|
tsk->dl.dl_overrun = 0;
|
|
|
|
__group_send_sig_info(SIGXCPU, SEND_SIG_PRIV, tsk);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* Check for any per-thread CPU timers that have fired and move them off
|
|
|
|
* the tsk->cpu_timers[N] list onto the firing list. Here we update the
|
|
|
|
* tsk->it_*_expires values to reflect the remaining thread CPU timers.
|
|
|
|
*/
|
|
|
|
static void check_thread_timers(struct task_struct *tsk,
|
|
|
|
struct list_head *firing)
|
|
|
|
{
|
2019-08-27 01:22:24 +07:00
|
|
|
struct posix_cputimer_base *base = tsk->posix_cputimers.bases;
|
2010-03-06 04:42:53 +07:00
|
|
|
unsigned long soft;
|
2019-08-27 01:22:24 +07:00
|
|
|
u64 stime, utime;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2017-12-12 18:10:24 +07:00
|
|
|
if (dl_task(tsk))
|
|
|
|
check_dl_overrun(tsk);
|
|
|
|
|
2015-10-15 02:07:54 +07:00
|
|
|
/*
|
2019-08-22 02:09:13 +07:00
|
|
|
* If the expiry cache is zero, then there are no active per thread
|
|
|
|
* CPU timers.
|
2015-10-15 02:07:54 +07:00
|
|
|
*/
|
2019-08-27 01:22:24 +07:00
|
|
|
if (expiry_cache_is_zero(&tsk->posix_cputimers))
|
2015-10-15 02:07:54 +07:00
|
|
|
return;
|
|
|
|
|
2019-08-22 02:09:02 +07:00
|
|
|
task_cputime(tsk, &utime, &stime);
|
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
base->nextevt = check_timers_list(&base->cpu_timers, firing,
|
|
|
|
utime + stime);
|
|
|
|
base++;
|
|
|
|
base->nextevt = check_timers_list(&base->cpu_timers, firing, utime);
|
|
|
|
base++;
|
|
|
|
base->nextevt = check_timers_list(&base->cpu_timers, firing,
|
|
|
|
tsk->se.sum_exec_runtime);
|
2008-01-26 03:08:27 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Check for the special case thread timers.
|
|
|
|
*/
|
2017-07-06 00:25:48 +07:00
|
|
|
soft = task_rlimit(tsk, RLIMIT_RTTIME);
|
2010-03-06 04:42:53 +07:00
|
|
|
if (soft != RLIM_INFINITY) {
|
2017-07-06 00:25:48 +07:00
|
|
|
unsigned long hard = task_rlimit_max(tsk, RLIMIT_RTTIME);
|
2008-01-26 03:08:27 +07:00
|
|
|
|
2008-01-26 03:08:32 +07:00
|
|
|
if (hard != RLIM_INFINITY &&
|
|
|
|
tsk->rt.timeout > DIV_ROUND_UP(hard, USEC_PER_SEC/HZ)) {
|
2008-01-26 03:08:27 +07:00
|
|
|
/*
|
|
|
|
* At the hard limit, we just die.
|
|
|
|
* No need to calculate anything else now.
|
|
|
|
*/
|
2017-05-24 04:27:38 +07:00
|
|
|
if (print_fatal_signals) {
|
|
|
|
pr_info("CPU Watchdog Timeout (hard): %s[%d]\n",
|
|
|
|
tsk->comm, task_pid_nr(tsk));
|
|
|
|
}
|
2008-01-26 03:08:27 +07:00
|
|
|
__group_send_sig_info(SIGKILL, SEND_SIG_PRIV, tsk);
|
|
|
|
return;
|
|
|
|
}
|
2010-03-06 04:42:53 +07:00
|
|
|
if (tsk->rt.timeout > DIV_ROUND_UP(soft, USEC_PER_SEC/HZ)) {
|
2008-01-26 03:08:27 +07:00
|
|
|
/*
|
|
|
|
* At the soft limit, send a SIGXCPU every second.
|
|
|
|
*/
|
2010-03-06 04:42:53 +07:00
|
|
|
if (soft < hard) {
|
|
|
|
soft += USEC_PER_SEC;
|
2017-07-06 00:25:48 +07:00
|
|
|
tsk->signal->rlim[RLIMIT_RTTIME].rlim_cur =
|
|
|
|
soft;
|
2008-01-26 03:08:27 +07:00
|
|
|
}
|
2017-05-24 04:27:38 +07:00
|
|
|
if (print_fatal_signals) {
|
|
|
|
pr_info("RT Watchdog Timeout (soft): %s[%d]\n",
|
|
|
|
tsk->comm, task_pid_nr(tsk));
|
|
|
|
}
|
2008-01-26 03:08:27 +07:00
|
|
|
__group_send_sig_info(SIGXCPU, SEND_SIG_PRIV, tsk);
|
|
|
|
}
|
|
|
|
}
|
2019-08-22 02:09:10 +07:00
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
if (expiry_cache_is_zero(&tsk->posix_cputimers))
|
2015-07-18 03:25:49 +07:00
|
|
|
tick_dep_clear_task(tsk, TICK_DEP_BIT_POSIX_TIMER);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2015-04-29 03:00:22 +07:00
|
|
|
static inline void stop_process_timers(struct signal_struct *sig)
|
2009-02-10 22:37:31 +07:00
|
|
|
{
|
2010-03-12 05:04:31 +07:00
|
|
|
struct thread_group_cputimer *cputimer = &sig->cputimer;
|
2009-02-10 22:37:31 +07:00
|
|
|
|
2015-04-29 03:00:22 +07:00
|
|
|
/* Turn off cputimer->running. This is done without locking. */
|
2015-10-15 02:07:55 +07:00
|
|
|
WRITE_ONCE(cputimer->running, false);
|
2015-07-18 03:25:49 +07:00
|
|
|
tick_dep_clear_signal(sig, TICK_DEP_BIT_POSIX_TIMER);
|
2009-02-10 22:37:31 +07:00
|
|
|
}
|
|
|
|
|
2009-07-29 17:15:26 +07:00
|
|
|
static void check_cpu_itimer(struct task_struct *tsk, struct cpu_itimer *it,
|
2017-01-31 10:09:34 +07:00
|
|
|
u64 *expires, u64 cur_time, int signo)
|
2009-07-29 17:15:26 +07:00
|
|
|
{
|
2011-12-15 20:56:09 +07:00
|
|
|
if (!it->expires)
|
2009-07-29 17:15:26 +07:00
|
|
|
return;
|
|
|
|
|
2017-01-31 10:09:35 +07:00
|
|
|
if (cur_time >= it->expires) {
|
|
|
|
if (it->incr)
|
2011-12-15 20:56:09 +07:00
|
|
|
it->expires += it->incr;
|
2017-01-31 10:09:35 +07:00
|
|
|
else
|
2011-12-15 20:56:09 +07:00
|
|
|
it->expires = 0;
|
2009-07-29 17:15:26 +07:00
|
|
|
|
2009-08-10 09:52:30 +07:00
|
|
|
trace_itimer_expire(signo == SIGPROF ?
|
|
|
|
ITIMER_PROF : ITIMER_VIRTUAL,
|
2017-06-04 16:32:13 +07:00
|
|
|
task_tgid(tsk), cur_time);
|
2009-07-29 17:15:26 +07:00
|
|
|
__group_send_sig_info(signo, SEND_SIG_PRIV, tsk);
|
|
|
|
}
|
|
|
|
|
2017-01-31 10:09:35 +07:00
|
|
|
if (it->expires && (!*expires || it->expires < *expires))
|
|
|
|
*expires = it->expires;
|
2009-07-29 17:15:26 +07:00
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* Check for any per-thread CPU timers that have fired and move them
|
|
|
|
* off the tsk->*_timers list onto the firing list. Per-thread timers
|
|
|
|
* have already been taken off.
|
|
|
|
*/
|
|
|
|
static void check_process_timers(struct task_struct *tsk,
|
|
|
|
struct list_head *firing)
|
|
|
|
{
|
|
|
|
struct signal_struct *const sig = tsk->signal;
|
2019-08-27 01:22:24 +07:00
|
|
|
struct posix_cputimer_base *base = sig->posix_cputimers.bases;
|
2017-01-31 10:09:34 +07:00
|
|
|
u64 utime, ptime, virt_expires, prof_expires;
|
|
|
|
u64 sum_sched_runtime, sched_expires;
|
|
|
|
struct task_cputime cputime;
|
2010-03-06 04:42:53 +07:00
|
|
|
unsigned long soft;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2015-10-15 02:07:54 +07:00
|
|
|
/*
|
|
|
|
* If cputimer is not running, then there are no active
|
|
|
|
* process wide timers (POSIX 1.b, itimers, RLIMIT_CPU).
|
|
|
|
*/
|
|
|
|
if (!READ_ONCE(tsk->signal->cputimer.running))
|
|
|
|
return;
|
|
|
|
|
2019-08-22 02:08:53 +07:00
|
|
|
/*
|
2015-10-15 02:07:56 +07:00
|
|
|
* Signify that a thread is checking for process timers.
|
|
|
|
* Write access to this field is protected by the sighand lock.
|
|
|
|
*/
|
|
|
|
sig->cputimer.checking_timer = true;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
2019-08-22 02:08:53 +07:00
|
|
|
* Collect the current process totals. Group accounting is active
|
|
|
|
* so the sample can be taken directly.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2019-08-22 02:08:53 +07:00
|
|
|
sample_cputime_atomic(&cputime, &sig->cputimer.cputime_atomic);
|
2017-01-31 10:09:34 +07:00
|
|
|
utime = cputime.utime;
|
|
|
|
ptime = utime + cputime.stime;
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
sum_sched_runtime = cputime.sum_exec_runtime;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
prof_expires = check_timers_list(&base[CPUCLOCK_PROF].cpu_timers,
|
|
|
|
firing, ptime);
|
|
|
|
virt_expires = check_timers_list(&base[CPUCLOCK_VIRT].cpu_timers,
|
|
|
|
firing, utime);
|
|
|
|
sched_expires = check_timers_list(&base[CPUCLOCK_SCHED].cpu_timers,
|
|
|
|
firing, sum_sched_runtime);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Check for the special case process timers.
|
|
|
|
*/
|
2009-07-29 17:15:26 +07:00
|
|
|
check_cpu_itimer(tsk, &sig->it[CPUCLOCK_PROF], &prof_expires, ptime,
|
|
|
|
SIGPROF);
|
|
|
|
check_cpu_itimer(tsk, &sig->it[CPUCLOCK_VIRT], &virt_expires, utime,
|
|
|
|
SIGVTALRM);
|
2017-07-06 00:25:48 +07:00
|
|
|
soft = task_rlimit(tsk, RLIMIT_CPU);
|
2010-03-06 04:42:53 +07:00
|
|
|
if (soft != RLIM_INFINITY) {
|
2017-01-31 10:09:34 +07:00
|
|
|
unsigned long psecs = div_u64(ptime, NSEC_PER_SEC);
|
2017-07-06 00:25:48 +07:00
|
|
|
unsigned long hard = task_rlimit_max(tsk, RLIMIT_CPU);
|
2017-01-31 10:09:34 +07:00
|
|
|
u64 x;
|
2010-03-06 04:42:53 +07:00
|
|
|
if (psecs >= hard) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* At the hard limit, we just die.
|
|
|
|
* No need to calculate anything else now.
|
|
|
|
*/
|
2017-05-24 04:27:38 +07:00
|
|
|
if (print_fatal_signals) {
|
|
|
|
pr_info("RT Watchdog Timeout (hard): %s[%d]\n",
|
|
|
|
tsk->comm, task_pid_nr(tsk));
|
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
__group_send_sig_info(SIGKILL, SEND_SIG_PRIV, tsk);
|
|
|
|
return;
|
|
|
|
}
|
2010-03-06 04:42:53 +07:00
|
|
|
if (psecs >= soft) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* At the soft limit, send a SIGXCPU every second.
|
|
|
|
*/
|
2017-05-24 04:27:38 +07:00
|
|
|
if (print_fatal_signals) {
|
|
|
|
pr_info("CPU Watchdog Timeout (soft): %s[%d]\n",
|
|
|
|
tsk->comm, task_pid_nr(tsk));
|
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
__group_send_sig_info(SIGXCPU, SEND_SIG_PRIV, tsk);
|
2010-03-06 04:42:53 +07:00
|
|
|
if (soft < hard) {
|
|
|
|
soft++;
|
|
|
|
sig->rlim[RLIMIT_CPU].rlim_cur = soft;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
}
|
2017-01-31 10:09:34 +07:00
|
|
|
x = soft * NSEC_PER_SEC;
|
|
|
|
if (!prof_expires || x < prof_expires)
|
2005-04-17 05:20:36 +07:00
|
|
|
prof_expires = x;
|
|
|
|
}
|
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
base[CPUCLOCK_PROF].nextevt = prof_expires;
|
|
|
|
base[CPUCLOCK_VIRT].nextevt = virt_expires;
|
|
|
|
base[CPUCLOCK_SCHED].nextevt = sched_expires;
|
2019-08-22 02:09:10 +07:00
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
if (expiry_cache_is_zero(&sig->posix_cputimers))
|
2010-04-28 04:12:15 +07:00
|
|
|
stop_process_timers(sig);
|
2015-10-15 02:07:56 +07:00
|
|
|
|
|
|
|
sig->cputimer.checking_timer = false;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2017-05-31 04:15:46 +07:00
|
|
|
* This is called from the signal code (via posixtimer_rearm)
|
2005-04-17 05:20:36 +07:00
|
|
|
* when the last timer signal was delivered and we have to reload the timer.
|
|
|
|
*/
|
2017-05-31 04:15:47 +07:00
|
|
|
static void posix_cpu_timer_rearm(struct k_itimer *timer)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-22 02:08:58 +07:00
|
|
|
clockid_t clkid = CPUCLOCK_WHICH(timer->it_clock);
|
2019-08-19 21:31:46 +07:00
|
|
|
struct task_struct *p = timer->it.cpu.task;
|
2013-10-11 23:56:49 +07:00
|
|
|
struct sighand_struct *sighand;
|
|
|
|
unsigned long flags;
|
2017-01-31 10:09:34 +07:00
|
|
|
u64 now;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-19 21:31:46 +07:00
|
|
|
if (WARN_ON_ONCE(!p))
|
|
|
|
return;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Fetch the current sample and update the timer's expiry time.
|
|
|
|
*/
|
|
|
|
if (CPUCLOCK_PERTHREAD(timer->it_clock)) {
|
2019-08-22 02:09:01 +07:00
|
|
|
now = cpu_clock_sample(clkid, p);
|
2005-04-17 05:20:36 +07:00
|
|
|
bump_cpu_timer(timer, now);
|
2013-10-10 21:55:57 +07:00
|
|
|
if (unlikely(p->exit_state))
|
2017-05-31 04:15:42 +07:00
|
|
|
return;
|
2013-10-10 21:55:57 +07:00
|
|
|
|
2013-10-11 23:56:49 +07:00
|
|
|
/* Protect timer list r/w in arm_timer() */
|
|
|
|
sighand = lock_task_sighand(p, &flags);
|
|
|
|
if (!sighand)
|
2017-05-31 04:15:42 +07:00
|
|
|
return;
|
2005-04-17 05:20:36 +07:00
|
|
|
} else {
|
2013-10-11 23:56:49 +07:00
|
|
|
/*
|
|
|
|
* Protect arm_timer() and timer sampling in case of call to
|
2017-01-31 10:09:34 +07:00
|
|
|
* thread_group_cputime().
|
2013-10-11 23:56:49 +07:00
|
|
|
*/
|
|
|
|
sighand = lock_task_sighand(p, &flags);
|
|
|
|
if (unlikely(sighand == NULL)) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* The process has been reaped.
|
|
|
|
* We can't even collect a sample any more.
|
|
|
|
*/
|
2013-06-28 07:06:42 +07:00
|
|
|
timer->it.cpu.expires = 0;
|
2017-05-31 04:15:42 +07:00
|
|
|
return;
|
2005-04-17 05:20:36 +07:00
|
|
|
} else if (unlikely(p->exit_state) && thread_group_empty(p)) {
|
2017-05-31 04:15:42 +07:00
|
|
|
/* If the process is dying, no need to rearm */
|
|
|
|
goto unlock;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2019-08-22 02:09:01 +07:00
|
|
|
now = cpu_clock_sample_group(clkid, p, true);
|
2005-04-17 05:20:36 +07:00
|
|
|
bump_cpu_timer(timer, now);
|
2013-10-11 23:56:49 +07:00
|
|
|
/* Leave the sighand locked for the call below. */
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Now re-arm for the new expiry time.
|
|
|
|
*/
|
2010-03-12 05:04:38 +07:00
|
|
|
arm_timer(timer);
|
2017-05-31 04:15:42 +07:00
|
|
|
unlock:
|
2013-10-11 23:56:49 +07:00
|
|
|
unlock_task_sighand(p, &flags);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
/**
|
2019-08-27 01:22:24 +07:00
|
|
|
* task_cputimers_expired - Check whether posix CPU timers are expired
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
*
|
2019-08-22 02:09:13 +07:00
|
|
|
* @samples: Array of current samples for the CPUCLOCK clocks
|
2019-08-27 01:22:24 +07:00
|
|
|
* @pct: Pointer to a posix_cputimers container
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
*
|
2019-08-27 01:22:24 +07:00
|
|
|
* Returns true if any member of @samples is greater than the corresponding
|
|
|
|
* member of @pct->bases[CLK].nextevt. False otherwise
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
*/
|
2019-08-27 01:22:24 +07:00
|
|
|
static inline bool
|
|
|
|
task_cputimers_expired(const u64 *sample, struct posix_cputimers *pct)
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
{
|
2019-08-22 02:09:13 +07:00
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < CPUCLOCK_MAX; i++) {
|
2019-08-27 01:22:24 +07:00
|
|
|
if (pct->bases[i].nextevt && sample[i] >= pct->bases[i].nextevt)
|
2019-08-22 02:09:13 +07:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
return false;
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* fastpath_timer_check - POSIX CPU timers fast path.
|
|
|
|
*
|
|
|
|
* @tsk: The task (thread) being checked.
|
|
|
|
*
|
2008-09-12 23:54:39 +07:00
|
|
|
* Check the task and thread group timers. If both are zero (there are no
|
|
|
|
* timers set) return false. Otherwise snapshot the task and thread group
|
|
|
|
* timers and compare them with the corresponding expiration times. Return
|
|
|
|
* true if a timer has expired, else return false.
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
*/
|
2019-08-22 02:09:13 +07:00
|
|
|
static inline bool fastpath_timer_check(struct task_struct *tsk)
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
{
|
2008-11-17 21:39:47 +07:00
|
|
|
struct signal_struct *sig;
|
2008-09-12 23:54:39 +07:00
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
if (!expiry_cache_is_zero(&tsk->posix_cputimers)) {
|
2019-08-22 02:09:13 +07:00
|
|
|
u64 samples[CPUCLOCK_MAX];
|
2008-09-12 23:54:39 +07:00
|
|
|
|
2019-08-22 02:09:13 +07:00
|
|
|
task_sample_cputime(tsk, samples);
|
2019-08-27 01:22:24 +07:00
|
|
|
if (task_cputimers_expired(samples, &tsk->posix_cputimers))
|
2019-08-22 02:09:13 +07:00
|
|
|
return true;
|
2008-09-12 23:54:39 +07:00
|
|
|
}
|
2008-11-17 21:39:47 +07:00
|
|
|
|
|
|
|
sig = tsk->signal;
|
2015-10-15 02:07:56 +07:00
|
|
|
/*
|
|
|
|
* Check if thread group timers expired when the cputimer is
|
|
|
|
* running and no other thread in the group is already checking
|
|
|
|
* for thread group cputimers. These fields are read without the
|
|
|
|
* sighand lock. However, this is fine because this is meant to
|
|
|
|
* be a fastpath heuristic to determine whether we should try to
|
|
|
|
* acquire the sighand lock to check/handle timers.
|
|
|
|
*
|
|
|
|
* In the worst case scenario, if 'running' or 'checking_timer' gets
|
|
|
|
* set but the current thread doesn't see the change yet, we'll wait
|
|
|
|
* until the next thread in the group gets a scheduler interrupt to
|
|
|
|
* handle the timer. This isn't an issue in practice because these
|
|
|
|
* types of delays with signals actually getting sent are expected.
|
|
|
|
*/
|
|
|
|
if (READ_ONCE(sig->cputimer.running) &&
|
|
|
|
!READ_ONCE(sig->cputimer.checking_timer)) {
|
2019-08-22 02:09:13 +07:00
|
|
|
u64 samples[CPUCLOCK_MAX];
|
2008-09-12 23:54:39 +07:00
|
|
|
|
2019-08-22 02:09:13 +07:00
|
|
|
proc_sample_cputime_atomic(&sig->cputimer.cputime_atomic,
|
|
|
|
samples);
|
2010-06-12 01:04:46 +07:00
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
if (task_cputimers_expired(samples, &sig->posix_cputimers))
|
2019-08-22 02:09:13 +07:00
|
|
|
return true;
|
2008-09-12 23:54:39 +07:00
|
|
|
}
|
2009-03-24 02:34:11 +07:00
|
|
|
|
2017-12-12 18:10:24 +07:00
|
|
|
if (dl_task(tsk) && tsk->dl.dl_overrun)
|
2019-08-22 02:09:13 +07:00
|
|
|
return true;
|
2017-12-12 18:10:24 +07:00
|
|
|
|
2019-08-22 02:09:13 +07:00
|
|
|
return false;
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
}
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* This is called from the timer interrupt handler. The irq handler has
|
|
|
|
* already updated our counts. We need to check if any timers fire now.
|
|
|
|
* Interrupts are disabled.
|
|
|
|
*/
|
2019-08-19 21:31:47 +07:00
|
|
|
void run_posix_cpu_timers(void)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-19 21:31:47 +07:00
|
|
|
struct task_struct *tsk = current;
|
2005-04-17 05:20:36 +07:00
|
|
|
struct k_itimer *timer, *next;
|
2010-06-11 06:10:18 +07:00
|
|
|
unsigned long flags;
|
2019-08-19 21:31:47 +07:00
|
|
|
LIST_HEAD(firing);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2017-11-06 22:01:28 +07:00
|
|
|
lockdep_assert_irqs_disabled();
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/*
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
* The fast path checks that there are no expired thread or thread
|
2008-09-12 23:54:39 +07:00
|
|
|
* group timers. If that's so, just return.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2008-09-12 23:54:39 +07:00
|
|
|
if (!fastpath_timer_check(tsk))
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
return;
|
2008-09-14 22:11:46 +07:00
|
|
|
|
2010-06-11 06:10:18 +07:00
|
|
|
if (!lock_task_sighand(tsk, &flags))
|
|
|
|
return;
|
2008-09-12 23:54:39 +07:00
|
|
|
/*
|
|
|
|
* Here we take off tsk->signal->cpu_timers[N] and
|
|
|
|
* tsk->cpu_timers[N] all the timers that are firing, and
|
|
|
|
* put them on the firing list.
|
|
|
|
*/
|
|
|
|
check_thread_timers(tsk, &firing);
|
2015-10-15 02:07:54 +07:00
|
|
|
|
|
|
|
check_process_timers(tsk, &firing);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2008-09-12 23:54:39 +07:00
|
|
|
/*
|
|
|
|
* We must release these locks before taking any timer's lock.
|
|
|
|
* There is a potential race with timer deletion here, as the
|
|
|
|
* siglock now protects our private firing list. We have set
|
|
|
|
* the firing flag in each timer, so that a deletion attempt
|
|
|
|
* that gets the timer lock before we do will give it up and
|
|
|
|
* spin until we've taken care of that timer below.
|
|
|
|
*/
|
2010-06-11 06:10:18 +07:00
|
|
|
unlock_task_sighand(tsk, &flags);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Now that all the timers on our list have the firing flag,
|
2011-03-31 08:57:33 +07:00
|
|
|
* no one will touch their list entries but us. We'll take
|
2005-04-17 05:20:36 +07:00
|
|
|
* each timer's lock before clearing its firing flag, so no
|
|
|
|
* timer call will interfere.
|
|
|
|
*/
|
|
|
|
list_for_each_entry_safe(timer, next, &firing, it.cpu.entry) {
|
2009-04-30 06:14:32 +07:00
|
|
|
int cpu_firing;
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
spin_lock(&timer->it_lock);
|
|
|
|
list_del_init(&timer->it.cpu.entry);
|
2009-04-30 06:14:32 +07:00
|
|
|
cpu_firing = timer->it.cpu.firing;
|
2005-04-17 05:20:36 +07:00
|
|
|
timer->it.cpu.firing = 0;
|
|
|
|
/*
|
|
|
|
* The firing flag is -1 if we collided with a reset
|
|
|
|
* of the timer, which already reported this
|
|
|
|
* almost-firing as an overrun. So don't generate an event.
|
|
|
|
*/
|
2009-04-30 06:14:32 +07:00
|
|
|
if (likely(cpu_firing >= 0))
|
2005-04-17 05:20:36 +07:00
|
|
|
cpu_timer_fire(timer);
|
|
|
|
spin_unlock(&timer->it_lock);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2010-03-12 05:04:37 +07:00
|
|
|
* Set one of the process-wide special case CPU timers or RLIMIT_CPU.
|
timers: fix itimer/many thread hang
Overview
This patch reworks the handling of POSIX CPU timers, including the
ITIMER_PROF, ITIMER_VIRT timers and rlimit handling. It was put together
with the help of Roland McGrath, the owner and original writer of this code.
The problem we ran into, and the reason for this rework, has to do with using
a profiling timer in a process with a large number of threads. It appears
that the performance of the old implementation of run_posix_cpu_timers() was
at least O(n*3) (where "n" is the number of threads in a process) or worse.
Everything is fine with an increasing number of threads until the time taken
for that routine to run becomes the same as or greater than the tick time, at
which point things degrade rather quickly.
This patch fixes bug 9906, "Weird hang with NPTL and SIGPROF."
Code Changes
This rework corrects the implementation of run_posix_cpu_timers() to make it
run in constant time for a particular machine. (Performance may vary between
one machine and another depending upon whether the kernel is built as single-
or multiprocessor and, in the latter case, depending upon the number of
running processors.) To do this, at each tick we now update fields in
signal_struct as well as task_struct. The run_posix_cpu_timers() function
uses those fields to make its decisions.
We define a new structure, "task_cputime," to contain user, system and
scheduler times and use these in appropriate places:
struct task_cputime {
cputime_t utime;
cputime_t stime;
unsigned long long sum_exec_runtime;
};
This is included in the structure "thread_group_cputime," which is a new
substructure of signal_struct and which varies for uniprocessor versus
multiprocessor kernels. For uniprocessor kernels, it uses "task_cputime" as
a simple substructure, while for multiprocessor kernels it is a pointer:
struct thread_group_cputime {
struct task_cputime totals;
};
struct thread_group_cputime {
struct task_cputime *totals;
};
We also add a new task_cputime substructure directly to signal_struct, to
cache the earliest expiration of process-wide timers, and task_cputime also
replaces the it_*_expires fields of task_struct (used for earliest expiration
of thread timers). The "thread_group_cputime" structure contains process-wide
timers that are updated via account_user_time() and friends. In the non-SMP
case the structure is a simple aggregator; unfortunately in the SMP case that
simplicity was not achievable due to cache-line contention between CPUs (in
one measured case performance was actually _worse_ on a 16-cpu system than
the same test on a 4-cpu system, due to this contention). For SMP, the
thread_group_cputime counters are maintained as a per-cpu structure allocated
using alloc_percpu(). The timer functions update only the timer field in
the structure corresponding to the running CPU, obtained using per_cpu_ptr().
We define a set of inline functions in sched.h that we use to maintain the
thread_group_cputime structure and hide the differences between UP and SMP
implementations from the rest of the kernel. The thread_group_cputime_init()
function initializes the thread_group_cputime structure for the given task.
The thread_group_cputime_alloc() is a no-op for UP; for SMP it calls the
out-of-line function thread_group_cputime_alloc_smp() to allocate and fill
in the per-cpu structures and fields. The thread_group_cputime_free()
function, also a no-op for UP, in SMP frees the per-cpu structures. The
thread_group_cputime_clone_thread() function (also a UP no-op) for SMP calls
thread_group_cputime_alloc() if the per-cpu structures haven't yet been
allocated. The thread_group_cputime() function fills the task_cputime
structure it is passed with the contents of the thread_group_cputime fields;
in UP it's that simple but in SMP it must also safely check that tsk->signal
is non-NULL (if it is it just uses the appropriate fields of task_struct) and,
if so, sums the per-cpu values for each online CPU. Finally, the three
functions account_group_user_time(), account_group_system_time() and
account_group_exec_runtime() are used by timer functions to update the
respective fields of the thread_group_cputime structure.
Non-SMP operation is trivial and will not be mentioned further.
The per-cpu structure is always allocated when a task creates its first new
thread, via a call to thread_group_cputime_clone_thread() from copy_signal().
It is freed at process exit via a call to thread_group_cputime_free() from
cleanup_signal().
All functions that formerly summed utime/stime/sum_sched_runtime values from
from all threads in the thread group now use thread_group_cputime() to
snapshot the values in the thread_group_cputime structure or the values in
the task structure itself if the per-cpu structure hasn't been allocated.
Finally, the code in kernel/posix-cpu-timers.c has changed quite a bit.
The run_posix_cpu_timers() function has been split into a fast path and a
slow path; the former safely checks whether there are any expired thread
timers and, if not, just returns, while the slow path does the heavy lifting.
With the dedicated thread group fields, timers are no longer "rebalanced" and
the process_timer_rebalance() function and related code has gone away. All
summing loops are gone and all code that used them now uses the
thread_group_cputime() inline. When process-wide timers are set, the new
task_cputime structure in signal_struct is used to cache the earliest
expiration; this is checked in the fast path.
Performance
The fix appears not to add significant overhead to existing operations. It
generally performs the same as the current code except in two cases, one in
which it performs slightly worse (Case 5 below) and one in which it performs
very significantly better (Case 2 below). Overall it's a wash except in those
two cases.
I've since done somewhat more involved testing on a dual-core Opteron system.
Case 1: With no itimer running, for a test with 100,000 threads, the fixed
kernel took 1428.5 seconds, 513 seconds more than the unfixed system,
all of which was spent in the system. There were twice as many
voluntary context switches with the fix as without it.
Case 2: With an itimer running at .01 second ticks and 4000 threads (the most
an unmodified kernel can handle), the fixed kernel ran the test in
eight percent of the time (5.8 seconds as opposed to 70 seconds) and
had better tick accuracy (.012 seconds per tick as opposed to .023
seconds per tick).
Case 3: A 4000-thread test with an initial timer tick of .01 second and an
interval of 10,000 seconds (i.e. a timer that ticks only once) had
very nearly the same performance in both cases: 6.3 seconds elapsed
for the fixed kernel versus 5.5 seconds for the unfixed kernel.
With fewer threads (eight in these tests), the Case 1 test ran in essentially
the same time on both the modified and unmodified kernels (5.2 seconds versus
5.8 seconds). The Case 2 test ran in about the same time as well, 5.9 seconds
versus 5.4 seconds but again with much better tick accuracy, .013 seconds per
tick versus .025 seconds per tick for the unmodified kernel.
Since the fix affected the rlimit code, I also tested soft and hard CPU limits.
Case 4: With a hard CPU limit of 20 seconds and eight threads (and an itimer
running), the modified kernel was very slightly favored in that while
it killed the process in 19.997 seconds of CPU time (5.002 seconds of
wall time), only .003 seconds of that was system time, the rest was
user time. The unmodified kernel killed the process in 20.001 seconds
of CPU (5.014 seconds of wall time) of which .016 seconds was system
time. Really, though, the results were too close to call. The results
were essentially the same with no itimer running.
Case 5: With a soft limit of 20 seconds and a hard limit of 2000 seconds
(where the hard limit would never be reached) and an itimer running,
the modified kernel exhibited worse tick accuracy than the unmodified
kernel: .050 seconds/tick versus .028 seconds/tick. Otherwise,
performance was almost indistinguishable. With no itimer running this
test exhibited virtually identical behavior and times in both cases.
In times past I did some limited performance testing. those results are below.
On a four-cpu Opteron system without this fix, a sixteen-thread test executed
in 3569.991 seconds, of which user was 3568.435s and system was 1.556s. On
the same system with the fix, user and elapsed time were about the same, but
system time dropped to 0.007 seconds. Performance with eight, four and one
thread were comparable. Interestingly, the timer ticks with the fix seemed
more accurate: The sixteen-thread test with the fix received 149543 ticks
for 0.024 seconds per tick, while the same test without the fix received 58720
for 0.061 seconds per tick. Both cases were configured for an interval of
0.01 seconds. Again, the other tests were comparable. Each thread in this
test computed the primes up to 25,000,000.
I also did a test with a large number of threads, 100,000 threads, which is
impossible without the fix. In this case each thread computed the primes only
up to 10,000 (to make the runtime manageable). System time dominated, at
1546.968 seconds out of a total 2176.906 seconds (giving a user time of
629.938s). It received 147651 ticks for 0.015 seconds per tick, still quite
accurate. There is obviously no comparable test without the fix.
Signed-off-by: Frank Mayhar <fmayhar@google.com>
Cc: Roland McGrath <roland@redhat.com>
Cc: Alexey Dobriyan <adobriyan@gmail.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-09-12 23:54:39 +07:00
|
|
|
* The tsk->sighand->siglock must be held by the caller.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2019-08-22 02:09:09 +07:00
|
|
|
void set_process_cpu_timer(struct task_struct *tsk, unsigned int clkid,
|
2017-01-31 10:09:35 +07:00
|
|
|
u64 *newval, u64 *oldval)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2019-08-27 01:22:24 +07:00
|
|
|
u64 now, *nextevt;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-22 02:09:09 +07:00
|
|
|
if (WARN_ON_ONCE(clkid >= CPUCLOCK_SCHED))
|
2019-08-19 21:31:46 +07:00
|
|
|
return;
|
|
|
|
|
2019-08-27 01:22:24 +07:00
|
|
|
nextevt = &tsk->signal->posix_cputimers.bases[clkid].nextevt;
|
2019-08-22 02:09:09 +07:00
|
|
|
now = cpu_clock_sample_group(clkid, tsk, true);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2019-08-22 02:08:59 +07:00
|
|
|
if (oldval) {
|
2010-03-12 05:04:37 +07:00
|
|
|
/*
|
|
|
|
* We are setting itimer. The *oldval is absolute and we update
|
|
|
|
* it to be relative, *newval argument is relative and we update
|
|
|
|
* it to be absolute.
|
|
|
|
*/
|
2011-12-15 20:56:09 +07:00
|
|
|
if (*oldval) {
|
2017-01-31 10:09:35 +07:00
|
|
|
if (*oldval <= now) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/* Just about to fire. */
|
2017-01-31 10:09:35 +07:00
|
|
|
*oldval = TICK_NSEC;
|
2005-04-17 05:20:36 +07:00
|
|
|
} else {
|
2017-01-31 10:09:35 +07:00
|
|
|
*oldval -= now;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2011-12-15 20:56:09 +07:00
|
|
|
if (!*newval)
|
2015-07-18 03:25:49 +07:00
|
|
|
return;
|
2017-01-31 10:09:35 +07:00
|
|
|
*newval += now;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
2019-08-22 02:09:09 +07:00
|
|
|
* Update expiration cache if this is the earliest timer. CPUCLOCK_PROF
|
|
|
|
* expiry cache is also used by RLIMIT_CPU!.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2019-08-27 01:22:24 +07:00
|
|
|
if (expires_gt(*nextevt, *newval))
|
|
|
|
*nextevt = *newval;
|
2015-07-18 03:25:49 +07:00
|
|
|
|
|
|
|
tick_dep_set_signal(tsk->signal, TICK_DEP_BIT_POSIX_TIMER);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2006-09-29 16:00:29 +07:00
|
|
|
static int do_cpu_nanosleep(const clockid_t which_clock, int flags,
|
2017-06-14 04:29:14 +07:00
|
|
|
const struct timespec64 *rqtp)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2017-06-07 15:42:26 +07:00
|
|
|
struct itimerspec64 it;
|
2017-06-14 04:29:14 +07:00
|
|
|
struct k_itimer timer;
|
|
|
|
u64 expires;
|
2005-04-17 05:20:36 +07:00
|
|
|
int error;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Set up a temporary timer and then wait for it to go off.
|
|
|
|
*/
|
|
|
|
memset(&timer, 0, sizeof timer);
|
|
|
|
spin_lock_init(&timer.it_lock);
|
|
|
|
timer.it_clock = which_clock;
|
|
|
|
timer.it_overrun = -1;
|
|
|
|
error = posix_cpu_timer_create(&timer);
|
|
|
|
timer.it_process = current;
|
|
|
|
if (!error) {
|
2017-03-27 02:04:17 +07:00
|
|
|
static struct itimerspec64 zero_it;
|
2017-06-07 15:42:31 +07:00
|
|
|
struct restart_block *restart;
|
2006-09-29 16:00:29 +07:00
|
|
|
|
2017-06-07 15:42:31 +07:00
|
|
|
memset(&it, 0, sizeof(it));
|
2017-06-07 15:42:26 +07:00
|
|
|
it.it_value = *rqtp;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
spin_lock_irq(&timer.it_lock);
|
2017-06-07 15:42:26 +07:00
|
|
|
error = posix_cpu_timer_set(&timer, flags, &it, NULL);
|
2005-04-17 05:20:36 +07:00
|
|
|
if (error) {
|
|
|
|
spin_unlock_irq(&timer.it_lock);
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
|
|
|
while (!signal_pending(current)) {
|
2013-06-28 07:06:42 +07:00
|
|
|
if (timer.it.cpu.expires == 0) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
2013-02-15 17:08:11 +07:00
|
|
|
* Our timer fired and was reset, below
|
|
|
|
* deletion can not fail.
|
2005-04-17 05:20:36 +07:00
|
|
|
*/
|
2013-02-15 17:08:11 +07:00
|
|
|
posix_cpu_timer_del(&timer);
|
2005-04-17 05:20:36 +07:00
|
|
|
spin_unlock_irq(&timer.it_lock);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Block until cpu_timer_fire (or a signal) wakes us.
|
|
|
|
*/
|
|
|
|
__set_current_state(TASK_INTERRUPTIBLE);
|
|
|
|
spin_unlock_irq(&timer.it_lock);
|
|
|
|
schedule();
|
|
|
|
spin_lock_irq(&timer.it_lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We were interrupted by a signal.
|
|
|
|
*/
|
2017-06-14 04:29:14 +07:00
|
|
|
expires = timer.it.cpu.expires;
|
2017-06-07 15:42:26 +07:00
|
|
|
error = posix_cpu_timer_set(&timer, 0, &zero_it, &it);
|
2013-02-15 17:08:11 +07:00
|
|
|
if (!error) {
|
|
|
|
/*
|
|
|
|
* Timer is now unarmed, deletion can not fail.
|
|
|
|
*/
|
|
|
|
posix_cpu_timer_del(&timer);
|
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
spin_unlock_irq(&timer.it_lock);
|
|
|
|
|
2013-02-15 17:08:11 +07:00
|
|
|
while (error == TIMER_RETRY) {
|
|
|
|
/*
|
|
|
|
* We need to handle case when timer was or is in the
|
|
|
|
* middle of firing. In other cases we already freed
|
|
|
|
* resources.
|
|
|
|
*/
|
|
|
|
spin_lock_irq(&timer.it_lock);
|
|
|
|
error = posix_cpu_timer_del(&timer);
|
|
|
|
spin_unlock_irq(&timer.it_lock);
|
|
|
|
}
|
|
|
|
|
2017-06-07 15:42:26 +07:00
|
|
|
if ((it.it_value.tv_sec | it.it_value.tv_nsec) == 0) {
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
* It actually did fire already.
|
|
|
|
*/
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-09-29 16:00:29 +07:00
|
|
|
error = -ERESTART_RESTARTBLOCK;
|
2017-06-07 15:42:26 +07:00
|
|
|
/*
|
|
|
|
* Report back to the user the time still remaining.
|
|
|
|
*/
|
2017-06-07 15:42:31 +07:00
|
|
|
restart = ¤t->restart_block;
|
2017-06-14 04:29:14 +07:00
|
|
|
restart->nanosleep.expires = expires;
|
2017-06-25 01:45:06 +07:00
|
|
|
if (restart->nanosleep.type != TT_NONE)
|
|
|
|
error = nanosleep_copyout(restart, &it.it_value);
|
2006-09-29 16:00:29 +07:00
|
|
|
}
|
|
|
|
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2011-02-01 20:52:12 +07:00
|
|
|
static long posix_cpu_nsleep_restart(struct restart_block *restart_block);
|
|
|
|
|
|
|
|
static int posix_cpu_nsleep(const clockid_t which_clock, int flags,
|
2017-06-14 04:34:33 +07:00
|
|
|
const struct timespec64 *rqtp)
|
2006-09-29 16:00:29 +07:00
|
|
|
{
|
2015-02-13 06:01:14 +07:00
|
|
|
struct restart_block *restart_block = ¤t->restart_block;
|
2006-09-29 16:00:29 +07:00
|
|
|
int error;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Diagnose required errors first.
|
|
|
|
*/
|
|
|
|
if (CPUCLOCK_PERTHREAD(which_clock) &&
|
|
|
|
(CPUCLOCK_PID(which_clock) == 0 ||
|
2017-04-13 22:32:16 +07:00
|
|
|
CPUCLOCK_PID(which_clock) == task_pid_vnr(current)))
|
2006-09-29 16:00:29 +07:00
|
|
|
return -EINVAL;
|
|
|
|
|
2017-06-07 15:42:26 +07:00
|
|
|
error = do_cpu_nanosleep(which_clock, flags, rqtp);
|
2006-09-29 16:00:29 +07:00
|
|
|
|
|
|
|
if (error == -ERESTART_RESTARTBLOCK) {
|
|
|
|
|
2011-02-01 20:51:20 +07:00
|
|
|
if (flags & TIMER_ABSTIME)
|
2006-09-29 16:00:29 +07:00
|
|
|
return -ERESTARTNOHAND;
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2006-09-29 16:00:28 +07:00
|
|
|
restart_block->fn = posix_cpu_nsleep_restart;
|
2011-05-20 18:05:15 +07:00
|
|
|
restart_block->nanosleep.clockid = which_clock;
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
return error;
|
|
|
|
}
|
|
|
|
|
2011-02-01 20:52:12 +07:00
|
|
|
static long posix_cpu_nsleep_restart(struct restart_block *restart_block)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2011-05-20 18:05:15 +07:00
|
|
|
clockid_t which_clock = restart_block->nanosleep.clockid;
|
2017-03-27 02:04:18 +07:00
|
|
|
struct timespec64 t;
|
2006-01-10 11:52:37 +07:00
|
|
|
|
2017-03-27 02:04:18 +07:00
|
|
|
t = ns_to_timespec64(restart_block->nanosleep.expires);
|
2006-01-10 11:52:37 +07:00
|
|
|
|
2017-06-07 15:42:26 +07:00
|
|
|
return do_cpu_nanosleep(which_clock, TIMER_ABSTIME, &t);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
2017-12-29 10:11:36 +07:00
|
|
|
#define PROCESS_CLOCK make_process_cpuclock(0, CPUCLOCK_SCHED)
|
|
|
|
#define THREAD_CLOCK make_thread_cpuclock(0, CPUCLOCK_SCHED)
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2006-01-10 11:52:27 +07:00
|
|
|
static int process_cpu_clock_getres(const clockid_t which_clock,
|
2017-03-27 02:04:15 +07:00
|
|
|
struct timespec64 *tp)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
return posix_cpu_clock_getres(PROCESS_CLOCK, tp);
|
|
|
|
}
|
2006-01-10 11:52:27 +07:00
|
|
|
static int process_cpu_clock_get(const clockid_t which_clock,
|
2017-03-27 02:04:14 +07:00
|
|
|
struct timespec64 *tp)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
return posix_cpu_clock_get(PROCESS_CLOCK, tp);
|
|
|
|
}
|
|
|
|
static int process_cpu_timer_create(struct k_itimer *timer)
|
|
|
|
{
|
|
|
|
timer->it_clock = PROCESS_CLOCK;
|
|
|
|
return posix_cpu_timer_create(timer);
|
|
|
|
}
|
2006-01-10 11:52:27 +07:00
|
|
|
static int process_cpu_nsleep(const clockid_t which_clock, int flags,
|
2017-06-14 04:34:33 +07:00
|
|
|
const struct timespec64 *rqtp)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
2017-06-07 15:42:30 +07:00
|
|
|
return posix_cpu_nsleep(PROCESS_CLOCK, flags, rqtp);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2006-01-10 11:52:27 +07:00
|
|
|
static int thread_cpu_clock_getres(const clockid_t which_clock,
|
2017-03-27 02:04:15 +07:00
|
|
|
struct timespec64 *tp)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
return posix_cpu_clock_getres(THREAD_CLOCK, tp);
|
|
|
|
}
|
2006-01-10 11:52:27 +07:00
|
|
|
static int thread_cpu_clock_get(const clockid_t which_clock,
|
2017-03-27 02:04:14 +07:00
|
|
|
struct timespec64 *tp)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
return posix_cpu_clock_get(THREAD_CLOCK, tp);
|
|
|
|
}
|
|
|
|
static int thread_cpu_timer_create(struct k_itimer *timer)
|
|
|
|
{
|
|
|
|
timer->it_clock = THREAD_CLOCK;
|
|
|
|
return posix_cpu_timer_create(timer);
|
|
|
|
}
|
|
|
|
|
2017-05-26 16:03:11 +07:00
|
|
|
const struct k_clock clock_posix_cpu = {
|
2011-02-01 20:51:06 +07:00
|
|
|
.clock_getres = posix_cpu_clock_getres,
|
|
|
|
.clock_set = posix_cpu_clock_set,
|
|
|
|
.clock_get = posix_cpu_clock_get,
|
|
|
|
.timer_create = posix_cpu_timer_create,
|
|
|
|
.nsleep = posix_cpu_nsleep,
|
|
|
|
.timer_set = posix_cpu_timer_set,
|
|
|
|
.timer_del = posix_cpu_timer_del,
|
|
|
|
.timer_get = posix_cpu_timer_get,
|
2017-05-31 04:15:47 +07:00
|
|
|
.timer_rearm = posix_cpu_timer_rearm,
|
2011-02-01 20:51:06 +07:00
|
|
|
};
|
|
|
|
|
2017-05-26 16:03:11 +07:00
|
|
|
const struct k_clock clock_process = {
|
|
|
|
.clock_getres = process_cpu_clock_getres,
|
|
|
|
.clock_get = process_cpu_clock_get,
|
|
|
|
.timer_create = process_cpu_timer_create,
|
|
|
|
.nsleep = process_cpu_nsleep,
|
|
|
|
};
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2017-05-26 16:03:11 +07:00
|
|
|
const struct k_clock clock_thread = {
|
|
|
|
.clock_getres = thread_cpu_clock_getres,
|
|
|
|
.clock_get = thread_cpu_clock_get,
|
|
|
|
.timer_create = thread_cpu_timer_create,
|
|
|
|
};
|