2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
|
* Switch a MMU context.
|
|
|
|
|
*
|
|
|
|
|
* This file is subject to the terms and conditions of the GNU General Public
|
|
|
|
|
* License. See the file "COPYING" in the main directory of this archive
|
|
|
|
|
* for more details.
|
|
|
|
|
*
|
|
|
|
|
* Copyright (C) 1996, 1997, 1998, 1999 by Ralf Baechle
|
|
|
|
|
* Copyright (C) 1999 Silicon Graphics, Inc.
|
|
|
|
|
*/
|
|
|
|
|
#ifndef _ASM_MMU_CONTEXT_H
|
|
|
|
|
#define _ASM_MMU_CONTEXT_H
|
|
|
|
|
|
|
|
|
|
#include <linux/errno.h>
|
|
|
|
|
#include <linux/sched.h>
|
2017-02-04 06:16:44 +07:00
|
|
|
#include <linux/mm_types.h>
|
2009-06-19 20:05:26 +07:00
|
|
|
#include <linux/smp.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <linux/slab.h>
|
2017-02-04 06:16:44 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <asm/cacheflush.h>
|
MIPS: Use per-mm page to execute branch delay slot instructions
In some cases the kernel needs to execute an instruction from the delay
slot of an emulated branch instruction. These cases include:
- Emulated floating point branch instructions (bc1[ft]l?) for systems
which don't include an FPU, or upon which the kernel is run with the
"nofpu" parameter.
- MIPSr6 systems running binaries targeting older revisions of the
architecture, which may include branch instructions whose encodings
are no longer valid in MIPSr6.
Executing instructions from such delay slots is done by writing the
instruction to memory followed by a trap, as part of an "emuframe", and
executing it. This avoids the requirement of an emulator for the entire
MIPS instruction set. Prior to this patch such emuframes are written to
the user stack and executed from there.
This patch moves FP branch delay emuframes off of the user stack and
into a per-mm page. Allocating a page per-mm leaves userland with access
to only what it had access to previously, and compared to other
solutions is relatively simple.
When a thread requires a delay slot emulation, it is allocated a frame.
A thread may only have one frame allocated at any one time, since it may
only ever be executing one instruction at any one time. In order to
ensure that we can free up allocated frame later, its index is recorded
in struct thread_struct. In the typical case, after executing the delay
slot instruction we'll execute a break instruction with the BRK_MEMU
code. This traps back to the kernel & leads to a call to do_dsemulret
which frees the allocated frame & moves the user PC back to the
instruction that would have executed following the emulated branch.
In some cases the delay slot instruction may be invalid, such as a
branch, or may trigger an exception. In these cases the BRK_MEMU break
instruction will not be hit. In order to ensure that frames are freed
this patch introduces dsemul_thread_cleanup() and calls it to free any
allocated frame upon thread exit. If the instruction generated an
exception & leads to a signal being delivered to the thread, or indeed
if a signal simply happens to be delivered to the thread whilst it is
executing from the struct emuframe, then we need to take care to exit
the frame appropriately. This is done by either rolling back the user PC
to the branch or advancing it to the continuation PC prior to signal
delivery, using dsemul_thread_rollback(). If this were not done then a
sigreturn would return to the struct emuframe, and if that frame had
meanwhile been used in response to an emulated branch instruction within
the signal handler then we would execute the wrong user code.
Whilst a user could theoretically place something like a compact branch
to self in a delay slot and cause their thread to become stuck in an
infinite loop with the frame never being deallocated, this would:
- Only affect the users single process.
- Be architecturally invalid since there would be a branch in the
delay slot, which is forbidden.
- Be extremely unlikely to happen by mistake, and provide a program
with no more ability to harm the system than a simple infinite loop
would.
If a thread requires a delay slot emulation & no frame is available to
it (ie. the process has enough other threads that all frames are
currently in use) then the thread joins a waitqueue. It will sleep until
a frame is freed by another thread in the process.
Since we now know whether a thread has an allocated frame due to our
tracking of its index, the cookie field of struct emuframe is removed as
we can be more certain whether we have a valid frame. Since a thread may
only ever have a single frame at any given time, the epc field of struct
emuframe is also removed & the PC to continue from is instead stored in
struct thread_struct. Together these changes simplify & shrink struct
emuframe somewhat, allowing twice as many frames to fit into the page
allocated for them.
The primary benefit of this patch is that we are now free to mark the
user stack non-executable where that is possible.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Maciej Rozycki <maciej.rozycki@imgtec.com>
Cc: Faraz Shahbazker <faraz.shahbazker@imgtec.com>
Cc: Raghu Gandham <raghu.gandham@imgtec.com>
Cc: Matthew Fortune <matthew.fortune@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13764/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-07-08 17:06:19 +07:00
|
|
|
#include <asm/dsemul.h>
|
2009-10-14 04:23:28 +07:00
|
|
|
#include <asm/hazards.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
#include <asm/tlbflush.h>
|
2007-05-03 00:27:14 +07:00
|
|
|
#include <asm-generic/mm_hooks.h>
|
2005-04-17 05:20:36 +07:00
|
|
|
|
MIPS: mm: Use the Hardware Page Table Walker if the core supports it
The Hardware Page Table Walker aims to speed up TLB refill exceptions
by handling them in the hardware level instead of having a software
TLB refill handler. However, a TLB refill exception can still be
thrown in certain cases such as, synchronus exceptions, or address
translation or memory errors during the HTW operation. As a result of
which, HTW must not be considered a complete replacement for the TLB
refill software handler, but rather a fast-path for it.
For HTW to work, the PWBase register must contain the task's page
global directory address so the HTW will kick in on TLB refill
exceptions.
Due to HTW being a separate engine embedded deep in the CPU pipeline,
we need to restart the HTW everytime a PTE changes to avoid HTW
fetching a old entry from the page tables. It's also necessary to
restart the HTW on context switches to prevent it from fetching a
page from the previous process. Finally, since HTW is using the
entryhi register to write the translations to the TLB, it's necessary
to stop the HTW whenever the entryhi changes (eg for tlb probe
perations) and enable it back afterwards.
== Performance ==
The following trivial test was used to measure the performance of the
HTW. Using the same root filesystem, the following command was used
to measure the number of tlb refill handler executions with and
without (using 'nohtw' kernel parameter) HTW support. The kernel was
modified to use a scratch register as a counter for the TLB refill
exceptions.
find /usr -type f -exec ls -lh {} \;
HTW Enabled:
TLB refill exceptions: 12306
HTW Disabled:
TLB refill exceptions: 17805
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Markos Chandras <markos.chandras@imgtec.com>
Patchwork: https://patchwork.linux-mips.org/patch/7336/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2014-07-14 18:47:09 +07:00
|
|
|
#define htw_set_pwbase(pgd) \
|
|
|
|
|
do { \
|
|
|
|
|
if (cpu_has_htw) { \
|
|
|
|
|
write_c0_pwbase(pgd); \
|
|
|
|
|
back_to_back_c0_hazard(); \
|
|
|
|
|
} \
|
|
|
|
|
} while (0)
|
|
|
|
|
|
2016-10-08 05:58:53 +07:00
|
|
|
extern void tlbmiss_handler_setup_pgd(unsigned long);
|
MIPS: Consistently declare TLB functions
Since at least the beginning of the git era we've declared our TLB
exception handling functions inconsistently. They're actually functions,
but we declare them as arrays of u32 where each u32 is an encoded
instruction. This has always been the case for arch/mips/mm/tlbex.c, and
has also been true for arch/mips/kernel/traps.c since commit
86a1708a9d54 ("MIPS: Make tlb exception handler definitions and
declarations match.") which aimed for consistency but did so by
consistently making the our C code inconsistent with our assembly.
This is all usually harmless, but when using GCC 7 or newer to build a
kernel targeting microMIPS (ie. CONFIG_CPU_MICROMIPS=y) it becomes
problematic. With microMIPS bit 0 of the program counter indicates the
ISA mode. When bit 0 is zero instructions are decoded using the standard
MIPS32 or MIPS64 ISA. When bit 0 is one instructions are decoded using
microMIPS. This means that function pointers become odd - their least
significant bit is one for microMIPS code. We work around this in cases
where we need to access code using loads & stores with our
msk_isa16_mode() macro which simply clears bit 0 of the value it is
given:
#define msk_isa16_mode(x) ((x) & ~0x1)
For example we do this for our TLB load handler in
build_r4000_tlb_load_handler():
u32 *p = (u32 *)msk_isa16_mode((ulong)handle_tlbl);
We then write code to p, expecting it to be suitably aligned (our LEAF
macro aligns functions on 4 byte boundaries, so (ulong)handle_tlbl will
give a value one greater than a multiple of 4 - ie. the start of a
function on a 4 byte boundary, with the ISA mode bit 0 set).
This worked fine up to GCC 6, but GCC 7 & onwards is smart enough to
presume that handle_tlbl which we declared as an array of u32s must be
aligned sufficiently that bit 0 of its address will never be set, and as
a result optimize out msk_isa16_mode(). This leads to p having an
address with bit 0 set, and when we go on to attempt to store code at
that address we take an address error exception due to the unaligned
memory access.
This leads to an exception prior to the kernel having configured its own
exception handlers, so we jump to whatever handlers the bootloader
configured. In the case of QEMU this results in a silent hang, since it
has no useful general exception vector.
Fix this by consistently declaring our TLB-related functions as
functions. For handle_tlbl(), handle_tlbs() & handle_tlbm() we do this
in asm/tlbex.h & we make use of the existing declaration of
tlbmiss_handler_setup_pgd() in asm/mmu_context.h. Our TLB handler
generation code in arch/mips/mm/tlbex.c is adjusted to deal with these
definitions, in most cases simply by casting the function pointers to
u32 pointers.
This allows us to include asm/mmu_context.h in arch/mips/mm/tlbex.c to
get the definitions of tlbmiss_handler_setup_pgd & pgd_current, removing
some needless duplication. Consistently using msk_isa16_mode() on
function pointers means we no longer need the
tlbmiss_handler_setup_pgd_start symbol so that is removed entirely.
Now that we're declaring our functions as functions GCC stops optimizing
out msk_isa16_mode() & a microMIPS kernel built with either GCC 7.3.0 or
8.1.0 boots successfully.
Signed-off-by: Paul Burton <paul.burton@mips.com>
2018-08-11 06:03:31 +07:00
|
|
|
extern char tlbmiss_handler_setup_pgd_end[];
|
2016-10-08 05:58:53 +07:00
|
|
|
|
|
|
|
|
/* Note: This is also implemented with uasm in arch/mips/kvm/entry.c */
|
2013-03-21 17:28:10 +07:00
|
|
|
#define TLBMISS_HANDLER_SETUP_PGD(pgd) \
|
|
|
|
|
do { \
|
|
|
|
|
tlbmiss_handler_setup_pgd((unsigned long)(pgd)); \
|
MIPS: mm: Use the Hardware Page Table Walker if the core supports it
The Hardware Page Table Walker aims to speed up TLB refill exceptions
by handling them in the hardware level instead of having a software
TLB refill handler. However, a TLB refill exception can still be
thrown in certain cases such as, synchronus exceptions, or address
translation or memory errors during the HTW operation. As a result of
which, HTW must not be considered a complete replacement for the TLB
refill software handler, but rather a fast-path for it.
For HTW to work, the PWBase register must contain the task's page
global directory address so the HTW will kick in on TLB refill
exceptions.
Due to HTW being a separate engine embedded deep in the CPU pipeline,
we need to restart the HTW everytime a PTE changes to avoid HTW
fetching a old entry from the page tables. It's also necessary to
restart the HTW on context switches to prevent it from fetching a
page from the previous process. Finally, since HTW is using the
entryhi register to write the translations to the TLB, it's necessary
to stop the HTW whenever the entryhi changes (eg for tlb probe
perations) and enable it back afterwards.
== Performance ==
The following trivial test was used to measure the performance of the
HTW. Using the same root filesystem, the following command was used
to measure the number of tlb refill handler executions with and
without (using 'nohtw' kernel parameter) HTW support. The kernel was
modified to use a scratch register as a counter for the TLB refill
exceptions.
find /usr -type f -exec ls -lh {} \;
HTW Enabled:
TLB refill exceptions: 12306
HTW Disabled:
TLB refill exceptions: 17805
Signed-off-by: Markos Chandras <markos.chandras@imgtec.com>
Cc: linux-mips@linux-mips.org
Cc: Markos Chandras <markos.chandras@imgtec.com>
Patchwork: https://patchwork.linux-mips.org/patch/7336/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2014-07-14 18:47:09 +07:00
|
|
|
htw_set_pwbase((unsigned long)pgd); \
|
2013-03-21 17:28:10 +07:00
|
|
|
} while (0)
|
2009-10-15 02:16:56 +07:00
|
|
|
|
2013-09-25 17:58:04 +07:00
|
|
|
#ifdef CONFIG_MIPS_PGD_C0_CONTEXT
|
2014-03-04 17:20:43 +07:00
|
|
|
|
|
|
|
|
#define TLBMISS_HANDLER_RESTORE() \
|
|
|
|
|
write_c0_xcontext((unsigned long) smp_processor_id() << \
|
|
|
|
|
SMP_CPUID_REGSHIFT)
|
|
|
|
|
|
2009-10-15 02:16:56 +07:00
|
|
|
#define TLBMISS_HANDLER_SETUP() \
|
|
|
|
|
do { \
|
|
|
|
|
TLBMISS_HANDLER_SETUP_PGD(swapper_pg_dir); \
|
2014-03-04 17:20:43 +07:00
|
|
|
TLBMISS_HANDLER_RESTORE(); \
|
2009-10-15 02:16:56 +07:00
|
|
|
} while (0)
|
|
|
|
|
|
2013-08-11 18:40:16 +07:00
|
|
|
#else /* !CONFIG_MIPS_PGD_C0_CONTEXT: using pgd_current*/
|
2009-10-15 02:16:56 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
/*
|
|
|
|
|
* For the fast tlb miss handlers, we keep a per cpu array of pointers
|
|
|
|
|
* to the current pgd for each processor. Also, the proc. id is stuffed
|
|
|
|
|
* into the context register.
|
|
|
|
|
*/
|
|
|
|
|
extern unsigned long pgd_current[];
|
|
|
|
|
|
2014-03-04 17:20:43 +07:00
|
|
|
#define TLBMISS_HANDLER_RESTORE() \
|
2013-08-11 18:40:16 +07:00
|
|
|
write_c0_context((unsigned long) smp_processor_id() << \
|
2014-03-04 17:20:43 +07:00
|
|
|
SMP_CPUID_REGSHIFT)
|
|
|
|
|
|
|
|
|
|
#define TLBMISS_HANDLER_SETUP() \
|
|
|
|
|
TLBMISS_HANDLER_RESTORE(); \
|
2009-10-14 04:23:28 +07:00
|
|
|
back_to_back_c0_hazard(); \
|
2005-04-17 05:20:36 +07:00
|
|
|
TLBMISS_HANDLER_SETUP_PGD(swapper_pg_dir)
|
2009-10-15 02:16:56 +07:00
|
|
|
#endif /* CONFIG_MIPS_PGD_C0_CONTEXT*/
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2016-05-06 20:36:23 +07:00
|
|
|
/*
|
|
|
|
|
* All unused by hardware upper bits will be considered
|
|
|
|
|
* as a software asid extension.
|
|
|
|
|
*/
|
|
|
|
|
static unsigned long asid_version_mask(unsigned int cpu)
|
|
|
|
|
{
|
|
|
|
|
unsigned long asid_mask = cpu_asid_mask(&cpu_data[cpu]);
|
2013-05-14 03:56:44 +07:00
|
|
|
|
2016-05-06 20:36:23 +07:00
|
|
|
return ~(asid_mask | (asid_mask - 1));
|
|
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2016-05-06 20:36:23 +07:00
|
|
|
static unsigned long asid_first_version(unsigned int cpu)
|
|
|
|
|
{
|
|
|
|
|
return ~asid_version_mask(cpu) + 1;
|
|
|
|
|
}
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2010-02-19 07:13:04 +07:00
|
|
|
#define cpu_context(cpu, mm) ((mm)->context.asid[cpu])
|
2005-04-17 05:20:36 +07:00
|
|
|
#define asid_cache(cpu) (cpu_data[cpu].asid_cache)
|
2016-05-06 20:36:23 +07:00
|
|
|
#define cpu_asid(cpu, mm) \
|
|
|
|
|
(cpu_context((cpu), (mm)) & cpu_asid_mask(&cpu_data[cpu]))
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
|
static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
|
|
|
|
|
{
|
|
|
|
|
}
|
|
|
|
|
|
2013-05-14 03:56:44 +07:00
|
|
|
|
2006-04-05 15:45:45 +07:00
|
|
|
/* Normal, classic MIPS get_new_mmu_context */
|
2005-04-17 05:20:36 +07:00
|
|
|
static inline void
|
|
|
|
|
get_new_mmu_context(struct mm_struct *mm, unsigned long cpu)
|
|
|
|
|
{
|
|
|
|
|
unsigned long asid = asid_cache(cpu);
|
|
|
|
|
|
2016-05-06 20:36:23 +07:00
|
|
|
if (!((asid += cpu_asid_inc()) & cpu_asid_mask(&cpu_data[cpu]))) {
|
2005-04-17 05:20:36 +07:00
|
|
|
if (cpu_has_vtag_icache)
|
|
|
|
|
flush_icache_all();
|
|
|
|
|
local_flush_tlb_all(); /* start new asid cycle */
|
|
|
|
|
if (!asid) /* fix version if needed */
|
2016-05-06 20:36:23 +07:00
|
|
|
asid = asid_first_version(cpu);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
2012-11-22 09:34:11 +07:00
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
cpu_context(cpu, mm) = asid_cache(cpu) = asid;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Initialize the context related info for a new mm_struct
|
|
|
|
|
* instance.
|
|
|
|
|
*/
|
|
|
|
|
static inline int
|
|
|
|
|
init_new_context(struct task_struct *tsk, struct mm_struct *mm)
|
|
|
|
|
{
|
|
|
|
|
int i;
|
|
|
|
|
|
MIPS: Init new mmu_context for each possible CPU to avoid memory corruption
Currently, init_new_context() only for each online CPU, this may cause
memory corruption when CPU hotplug and fork() happens at the same time.
To avoid this, we make init_new_context() cover each possible CPU.
Scenario:
1, CPU#1 is being offline;
2, On CPU#0, do_fork() call dup_mm() and copy a mm_struct to the child;
3, On CPU#0, dup_mm() call init_new_context(), since CPU#1 is offline
and init_new_context() only covers the online CPUs, child has the
same asid as its parent on CPU#1 (however, child's asid should be 0);
4, CPU#1 is being online;
5, Now, if both parent and child run on CPU#1, memory corruption (e.g.
segfault, bus error, etc.) will occur.
Signed-off-by: Huacai Chen <chenhc@lemote.com>
Acked-by: David Daney <david.daney@cavium.com>
Patchwork: http://patchwork.linux-mips.org/patch/4995/
Acked-by: John Crispin <blogic@openwrt.org>
2013-03-17 18:50:14 +07:00
|
|
|
for_each_possible_cpu(i)
|
2005-04-17 05:20:36 +07:00
|
|
|
cpu_context(i, mm) = 0;
|
|
|
|
|
|
MIPS: Use per-mm page to execute branch delay slot instructions
In some cases the kernel needs to execute an instruction from the delay
slot of an emulated branch instruction. These cases include:
- Emulated floating point branch instructions (bc1[ft]l?) for systems
which don't include an FPU, or upon which the kernel is run with the
"nofpu" parameter.
- MIPSr6 systems running binaries targeting older revisions of the
architecture, which may include branch instructions whose encodings
are no longer valid in MIPSr6.
Executing instructions from such delay slots is done by writing the
instruction to memory followed by a trap, as part of an "emuframe", and
executing it. This avoids the requirement of an emulator for the entire
MIPS instruction set. Prior to this patch such emuframes are written to
the user stack and executed from there.
This patch moves FP branch delay emuframes off of the user stack and
into a per-mm page. Allocating a page per-mm leaves userland with access
to only what it had access to previously, and compared to other
solutions is relatively simple.
When a thread requires a delay slot emulation, it is allocated a frame.
A thread may only have one frame allocated at any one time, since it may
only ever be executing one instruction at any one time. In order to
ensure that we can free up allocated frame later, its index is recorded
in struct thread_struct. In the typical case, after executing the delay
slot instruction we'll execute a break instruction with the BRK_MEMU
code. This traps back to the kernel & leads to a call to do_dsemulret
which frees the allocated frame & moves the user PC back to the
instruction that would have executed following the emulated branch.
In some cases the delay slot instruction may be invalid, such as a
branch, or may trigger an exception. In these cases the BRK_MEMU break
instruction will not be hit. In order to ensure that frames are freed
this patch introduces dsemul_thread_cleanup() and calls it to free any
allocated frame upon thread exit. If the instruction generated an
exception & leads to a signal being delivered to the thread, or indeed
if a signal simply happens to be delivered to the thread whilst it is
executing from the struct emuframe, then we need to take care to exit
the frame appropriately. This is done by either rolling back the user PC
to the branch or advancing it to the continuation PC prior to signal
delivery, using dsemul_thread_rollback(). If this were not done then a
sigreturn would return to the struct emuframe, and if that frame had
meanwhile been used in response to an emulated branch instruction within
the signal handler then we would execute the wrong user code.
Whilst a user could theoretically place something like a compact branch
to self in a delay slot and cause their thread to become stuck in an
infinite loop with the frame never being deallocated, this would:
- Only affect the users single process.
- Be architecturally invalid since there would be a branch in the
delay slot, which is forbidden.
- Be extremely unlikely to happen by mistake, and provide a program
with no more ability to harm the system than a simple infinite loop
would.
If a thread requires a delay slot emulation & no frame is available to
it (ie. the process has enough other threads that all frames are
currently in use) then the thread joins a waitqueue. It will sleep until
a frame is freed by another thread in the process.
Since we now know whether a thread has an allocated frame due to our
tracking of its index, the cookie field of struct emuframe is removed as
we can be more certain whether we have a valid frame. Since a thread may
only ever have a single frame at any given time, the epc field of struct
emuframe is also removed & the PC to continue from is instead stored in
struct thread_struct. Together these changes simplify & shrink struct
emuframe somewhat, allowing twice as many frames to fit into the page
allocated for them.
The primary benefit of this patch is that we are now free to mark the
user stack non-executable where that is possible.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Maciej Rozycki <maciej.rozycki@imgtec.com>
Cc: Faraz Shahbazker <faraz.shahbazker@imgtec.com>
Cc: Raghu Gandham <raghu.gandham@imgtec.com>
Cc: Matthew Fortune <matthew.fortune@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13764/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-07-08 17:06:19 +07:00
|
|
|
mm->context.bd_emupage_allocmap = NULL;
|
|
|
|
|
spin_lock_init(&mm->context.bd_emupage_lock);
|
|
|
|
|
init_waitqueue_head(&mm->context.bd_emupage_queue);
|
|
|
|
|
|
2005-04-17 05:20:36 +07:00
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,
|
2013-01-22 18:59:30 +07:00
|
|
|
struct task_struct *tsk)
|
2005-04-17 05:20:36 +07:00
|
|
|
{
|
|
|
|
|
unsigned int cpu = smp_processor_id();
|
|
|
|
|
unsigned long flags;
|
2006-04-05 15:45:45 +07:00
|
|
|
local_irq_save(flags);
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2015-01-26 20:04:33 +07:00
|
|
|
htw_stop();
|
2005-04-17 05:20:36 +07:00
|
|
|
/* Check if our ASID is of an older version and thus invalid */
|
2016-05-06 20:36:23 +07:00
|
|
|
if ((cpu_context(cpu, next) ^ asid_cache(cpu)) & asid_version_mask(cpu))
|
2005-04-17 05:20:36 +07:00
|
|
|
get_new_mmu_context(next, cpu);
|
2009-05-27 23:29:37 +07:00
|
|
|
write_c0_entryhi(cpu_asid(cpu, next));
|
2005-04-17 05:20:36 +07:00
|
|
|
TLBMISS_HANDLER_SETUP_PGD(next->pgd);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Mark current->active_mm as not "active" anymore.
|
|
|
|
|
* We don't want to mislead possible IPI tlb flush routines.
|
|
|
|
|
*/
|
2009-09-24 22:34:50 +07:00
|
|
|
cpumask_clear_cpu(cpu, mm_cpumask(prev));
|
|
|
|
|
cpumask_set_cpu(cpu, mm_cpumask(next));
|
2015-01-26 20:04:33 +07:00
|
|
|
htw_start();
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
|
local_irq_restore(flags);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Destroy context related info for an mm_struct that is about
|
|
|
|
|
* to be put to rest.
|
|
|
|
|
*/
|
|
|
|
|
static inline void destroy_context(struct mm_struct *mm)
|
|
|
|
|
{
|
MIPS: Use per-mm page to execute branch delay slot instructions
In some cases the kernel needs to execute an instruction from the delay
slot of an emulated branch instruction. These cases include:
- Emulated floating point branch instructions (bc1[ft]l?) for systems
which don't include an FPU, or upon which the kernel is run with the
"nofpu" parameter.
- MIPSr6 systems running binaries targeting older revisions of the
architecture, which may include branch instructions whose encodings
are no longer valid in MIPSr6.
Executing instructions from such delay slots is done by writing the
instruction to memory followed by a trap, as part of an "emuframe", and
executing it. This avoids the requirement of an emulator for the entire
MIPS instruction set. Prior to this patch such emuframes are written to
the user stack and executed from there.
This patch moves FP branch delay emuframes off of the user stack and
into a per-mm page. Allocating a page per-mm leaves userland with access
to only what it had access to previously, and compared to other
solutions is relatively simple.
When a thread requires a delay slot emulation, it is allocated a frame.
A thread may only have one frame allocated at any one time, since it may
only ever be executing one instruction at any one time. In order to
ensure that we can free up allocated frame later, its index is recorded
in struct thread_struct. In the typical case, after executing the delay
slot instruction we'll execute a break instruction with the BRK_MEMU
code. This traps back to the kernel & leads to a call to do_dsemulret
which frees the allocated frame & moves the user PC back to the
instruction that would have executed following the emulated branch.
In some cases the delay slot instruction may be invalid, such as a
branch, or may trigger an exception. In these cases the BRK_MEMU break
instruction will not be hit. In order to ensure that frames are freed
this patch introduces dsemul_thread_cleanup() and calls it to free any
allocated frame upon thread exit. If the instruction generated an
exception & leads to a signal being delivered to the thread, or indeed
if a signal simply happens to be delivered to the thread whilst it is
executing from the struct emuframe, then we need to take care to exit
the frame appropriately. This is done by either rolling back the user PC
to the branch or advancing it to the continuation PC prior to signal
delivery, using dsemul_thread_rollback(). If this were not done then a
sigreturn would return to the struct emuframe, and if that frame had
meanwhile been used in response to an emulated branch instruction within
the signal handler then we would execute the wrong user code.
Whilst a user could theoretically place something like a compact branch
to self in a delay slot and cause their thread to become stuck in an
infinite loop with the frame never being deallocated, this would:
- Only affect the users single process.
- Be architecturally invalid since there would be a branch in the
delay slot, which is forbidden.
- Be extremely unlikely to happen by mistake, and provide a program
with no more ability to harm the system than a simple infinite loop
would.
If a thread requires a delay slot emulation & no frame is available to
it (ie. the process has enough other threads that all frames are
currently in use) then the thread joins a waitqueue. It will sleep until
a frame is freed by another thread in the process.
Since we now know whether a thread has an allocated frame due to our
tracking of its index, the cookie field of struct emuframe is removed as
we can be more certain whether we have a valid frame. Since a thread may
only ever have a single frame at any given time, the epc field of struct
emuframe is also removed & the PC to continue from is instead stored in
struct thread_struct. Together these changes simplify & shrink struct
emuframe somewhat, allowing twice as many frames to fit into the page
allocated for them.
The primary benefit of this patch is that we are now free to mark the
user stack non-executable where that is possible.
Signed-off-by: Paul Burton <paul.burton@imgtec.com>
Cc: Leonid Yegoshin <leonid.yegoshin@imgtec.com>
Cc: Maciej Rozycki <maciej.rozycki@imgtec.com>
Cc: Faraz Shahbazker <faraz.shahbazker@imgtec.com>
Cc: Raghu Gandham <raghu.gandham@imgtec.com>
Cc: Matthew Fortune <matthew.fortune@imgtec.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/13764/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-07-08 17:06:19 +07:00
|
|
|
dsemul_mm_cleanup(mm);
|
2005-04-17 05:20:36 +07:00
|
|
|
}
|
|
|
|
|
|
2007-10-12 05:46:15 +07:00
|
|
|
#define deactivate_mm(tsk, mm) do { } while (0)
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* After we have set current->mm to a new value, this activates
|
|
|
|
|
* the context for the new mm so we see the new mappings.
|
|
|
|
|
*/
|
|
|
|
|
static inline void
|
|
|
|
|
activate_mm(struct mm_struct *prev, struct mm_struct *next)
|
|
|
|
|
{
|
|
|
|
|
unsigned long flags;
|
|
|
|
|
unsigned int cpu = smp_processor_id();
|
|
|
|
|
|
|
|
|
|
local_irq_save(flags);
|
|
|
|
|
|
2015-01-26 20:04:33 +07:00
|
|
|
htw_stop();
|
2005-04-17 05:20:36 +07:00
|
|
|
/* Unconditionally get a new ASID. */
|
|
|
|
|
get_new_mmu_context(next, cpu);
|
|
|
|
|
|
2009-05-27 23:29:37 +07:00
|
|
|
write_c0_entryhi(cpu_asid(cpu, next));
|
2005-04-17 05:20:36 +07:00
|
|
|
TLBMISS_HANDLER_SETUP_PGD(next->pgd);
|
|
|
|
|
|
|
|
|
|
/* mark mmu ownership change */
|
2009-09-24 22:34:50 +07:00
|
|
|
cpumask_clear_cpu(cpu, mm_cpumask(prev));
|
|
|
|
|
cpumask_set_cpu(cpu, mm_cpumask(next));
|
2015-01-26 20:04:33 +07:00
|
|
|
htw_start();
|
2005-04-17 05:20:36 +07:00
|
|
|
|
|
|
|
|
local_irq_restore(flags);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If mm is currently active_mm, we can't really drop it. Instead,
|
|
|
|
|
* we will get a new one for it.
|
|
|
|
|
*/
|
|
|
|
|
static inline void
|
|
|
|
|
drop_mmu_context(struct mm_struct *mm, unsigned cpu)
|
|
|
|
|
{
|
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
|
|
local_irq_save(flags);
|
2015-01-26 20:04:33 +07:00
|
|
|
htw_stop();
|
2005-04-17 05:20:36 +07:00
|
|
|
|
2009-09-24 22:34:50 +07:00
|
|
|
if (cpumask_test_cpu(cpu, mm_cpumask(mm))) {
|
2005-04-17 05:20:36 +07:00
|
|
|
get_new_mmu_context(mm, cpu);
|
|
|
|
|
write_c0_entryhi(cpu_asid(cpu, mm));
|
|
|
|
|
} else {
|
|
|
|
|
/* will get a new context next time */
|
|
|
|
|
cpu_context(cpu, mm) = 0;
|
|
|
|
|
}
|
2015-01-26 20:04:33 +07:00
|
|
|
htw_start();
|
2005-04-17 05:20:36 +07:00
|
|
|
local_irq_restore(flags);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif /* _ASM_MMU_CONTEXT_H */
|