mirror of
https://github.com/AuxXxilium/linux_dsm_epyc7002.git
synced 2024-12-25 18:45:06 +07:00
85572c2c4a
The scheduler code calling cpufreq_update_util() may run during CPU
offline on the target CPU after the IRQ work lists have been flushed
for it, so the target CPU should be prevented from running code that
may queue up an IRQ work item on it at that point.
Unfortunately, that may not be the case if dvfs_possible_from_any_cpu
is set for at least one cpufreq policy in the system, because that
allows the CPU going offline to run the utilization update callback
of the cpufreq governor on behalf of another (online) CPU in some
cases.
If that happens, the cpufreq governor callback may queue up an IRQ
work on the CPU running it, which is going offline, and the IRQ work
may not be flushed after that point. Moreover, that IRQ work cannot
be flushed until the "offlining" CPU goes back online, so if any
other CPU calls irq_work_sync() to wait for the completion of that
IRQ work, it will have to wait until the "offlining" CPU is back
online and that may not happen forever. In particular, a system-wide
deadlock may occur during CPU online as a result of that.
The failing scenario is as follows. CPU0 is the boot CPU, so it
creates a cpufreq policy and becomes the "leader" of it
(policy->cpu). It cannot go offline, because it is the boot CPU.
Next, other CPUs join the cpufreq policy as they go online and they
leave it when they go offline. The last CPU to go offline, say CPU3,
may queue up an IRQ work while running the governor callback on
behalf of CPU0 after leaving the cpufreq policy because of the
dvfs_possible_from_any_cpu effect described above. Then, CPU0 is
the only online CPU in the system and the stale IRQ work is still
queued on CPU3. When, say, CPU1 goes back online, it will run
irq_work_sync() to wait for that IRQ work to complete and so it
will wait for CPU3 to go back online (which may never happen even
in principle), but (worse yet) CPU0 is waiting for CPU1 at that
point too and a system-wide deadlock occurs.
To address this problem notice that CPUs which cannot run cpufreq
utilization update code for themselves (for example, because they
have left the cpufreq policies that they belonged to), should also
be prevented from running that code on behalf of the other CPUs that
belong to a cpufreq policy with dvfs_possible_from_any_cpu set and so
in that case the cpufreq_update_util_data pointer of the CPU running
the code must not be NULL as well as for the CPU which is the target
of the cpufreq utilization update in progress.
Accordingly, change cpufreq_this_cpu_can_update() into a regular
function in kernel/sched/cpufreq.c (instead of a static inline in a
header file) and make it check the cpufreq_update_util_data pointer
of the local CPU if dvfs_possible_from_any_cpu is set for the target
cpufreq policy.
Also update the schedutil governor to do the
cpufreq_this_cpu_can_update() check in the non-fast-switch
case too to avoid the stale IRQ work issues.
Fixes: 99d14d0e16 ("cpufreq: Process remote callbacks from any CPU if the platform permits")
Link: https://lore.kernel.org/linux-pm/20191121093557.bycvdo4xyinbc5cb@vireshk-i7/
Reported-by: Anson Huang <anson.huang@nxp.com>
Tested-by: Anson Huang <anson.huang@nxp.com>
Cc: 4.14+ <stable@vger.kernel.org> # 4.14+
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Acked-by: Viresh Kumar <viresh.kumar@linaro.org>
Tested-by: Peng Fan <peng.fan@nxp.com> (i.MX8QXP-MEK)
Signed-off-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
78 lines
2.7 KiB
C
78 lines
2.7 KiB
C
// SPDX-License-Identifier: GPL-2.0
|
|
/*
|
|
* Scheduler code and data structures related to cpufreq.
|
|
*
|
|
* Copyright (C) 2016, Intel Corporation
|
|
* Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
|
|
*/
|
|
#include <linux/cpufreq.h>
|
|
|
|
#include "sched.h"
|
|
|
|
DEFINE_PER_CPU(struct update_util_data __rcu *, cpufreq_update_util_data);
|
|
|
|
/**
|
|
* cpufreq_add_update_util_hook - Populate the CPU's update_util_data pointer.
|
|
* @cpu: The CPU to set the pointer for.
|
|
* @data: New pointer value.
|
|
* @func: Callback function to set for the CPU.
|
|
*
|
|
* Set and publish the update_util_data pointer for the given CPU.
|
|
*
|
|
* The update_util_data pointer of @cpu is set to @data and the callback
|
|
* function pointer in the target struct update_util_data is set to @func.
|
|
* That function will be called by cpufreq_update_util() from RCU-sched
|
|
* read-side critical sections, so it must not sleep. @data will always be
|
|
* passed to it as the first argument which allows the function to get to the
|
|
* target update_util_data structure and its container.
|
|
*
|
|
* The update_util_data pointer of @cpu must be NULL when this function is
|
|
* called or it will WARN() and return with no effect.
|
|
*/
|
|
void cpufreq_add_update_util_hook(int cpu, struct update_util_data *data,
|
|
void (*func)(struct update_util_data *data, u64 time,
|
|
unsigned int flags))
|
|
{
|
|
if (WARN_ON(!data || !func))
|
|
return;
|
|
|
|
if (WARN_ON(per_cpu(cpufreq_update_util_data, cpu)))
|
|
return;
|
|
|
|
data->func = func;
|
|
rcu_assign_pointer(per_cpu(cpufreq_update_util_data, cpu), data);
|
|
}
|
|
EXPORT_SYMBOL_GPL(cpufreq_add_update_util_hook);
|
|
|
|
/**
|
|
* cpufreq_remove_update_util_hook - Clear the CPU's update_util_data pointer.
|
|
* @cpu: The CPU to clear the pointer for.
|
|
*
|
|
* Clear the update_util_data pointer for the given CPU.
|
|
*
|
|
* Callers must use RCU callbacks to free any memory that might be
|
|
* accessed via the old update_util_data pointer or invoke synchronize_rcu()
|
|
* right after this function to avoid use-after-free.
|
|
*/
|
|
void cpufreq_remove_update_util_hook(int cpu)
|
|
{
|
|
rcu_assign_pointer(per_cpu(cpufreq_update_util_data, cpu), NULL);
|
|
}
|
|
EXPORT_SYMBOL_GPL(cpufreq_remove_update_util_hook);
|
|
|
|
/**
|
|
* cpufreq_this_cpu_can_update - Check if cpufreq policy can be updated.
|
|
* @policy: cpufreq policy to check.
|
|
*
|
|
* Return 'true' if:
|
|
* - the local and remote CPUs share @policy,
|
|
* - dvfs_possible_from_any_cpu is set in @policy and the local CPU is not going
|
|
* offline (in which case it is not expected to run cpufreq updates any more).
|
|
*/
|
|
bool cpufreq_this_cpu_can_update(struct cpufreq_policy *policy)
|
|
{
|
|
return cpumask_test_cpu(smp_processor_id(), policy->cpus) ||
|
|
(policy->dvfs_possible_from_any_cpu &&
|
|
rcu_dereference_sched(*this_cpu_ptr(&cpufreq_update_util_data)));
|
|
}
|