AuxXxilium/linux_dsm_epyc7002
1
0
Fork 0
mirror of https://github.com/AuxXxilium/linux_dsm_epyc7002.git synced 2024-12-26 10:55:09 +07:00
Code Issues Actions Packages Projects Releases Wiki Activity
32d9258662
linux_dsm_epyc7002/arch/riscv/include/asm/tlbflush.h

80 lines
2.2 KiB
C
Raw Normal View History

RISC-V: Atomic and Locking Code This contains all the code that directly interfaces with the RISC-V memory model. While this code corforms to the current RISC-V ISA specifications (user 2.2 and priv 1.10), the memory model is somewhat underspecified in those documents. There is a working group that hopes to produce a formal memory model by the end of the year, but my understanding is that the basic definitions we're relying on here won't change significantly. Reviewed-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 08:02:19 +07:00
/*
* Copyright (C) 2009 Chen Liqin <liqin.chen@sunplusct.com>
* Copyright (C) 2012 Regents of the University of California
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation, version 2.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*/
#ifndef _ASM_RISCV_TLBFLUSH_H
#define _ASM_RISCV_TLBFLUSH_H
RISC-V: Flush I$ when making a dirty page executable The RISC-V ISA allows for instruction caches that are not coherent WRT stores, even on a single hart. As a result, we need to explicitly flush the instruction cache whenever marking a dirty page as executable in order to preserve the correct system behavior. Local instruction caches aren't that scary (our implementations actually flush the cache, but RISC-V is defined to allow higher-performance implementations to exist), but RISC-V defines no way to perform an instruction cache shootdown. When explicitly asked to do so we can shoot down remote instruction caches via an IPI, but this is a bit on the slow side. Instead of requiring an IPI to all harts whenever marking a page as executable, we simply flush the currently running harts. In order to maintain correct behavior, we additionally mark every other hart as needing a deferred instruction cache which will be taken before anything runs on it. Signed-off-by: Andrew Waterman <andrew@sifive.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2017-10-26 04:30:32 +07:00
#include <linux/mm_types.h>
RISC-V: Use Linux logical CPU number instead of hartid Setup the cpu_logical_map during boot. Moreover, every SBI call and PLIC context are based on the physical hartid. Use the logical CPU to hartid mapping to pass correct hartid to respective functions. Signed-off-by: Atish Patra <atish.patra@wdc.com> Reviewed-by: Anup Patel <anup@brainfault.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-10-03 02:15:05 +07:00
#include <asm/smp.h>
RISC-V: Flush I$ when making a dirty page executable The RISC-V ISA allows for instruction caches that are not coherent WRT stores, even on a single hart. As a result, we need to explicitly flush the instruction cache whenever marking a dirty page as executable in order to preserve the correct system behavior. Local instruction caches aren't that scary (our implementations actually flush the cache, but RISC-V is defined to allow higher-performance implementations to exist), but RISC-V defines no way to perform an instruction cache shootdown. When explicitly asked to do so we can shoot down remote instruction caches via an IPI, but this is a bit on the slow side. Instead of requiring an IPI to all harts whenever marking a page as executable, we simply flush the currently running harts. In order to maintain correct behavior, we additionally mark every other hart as needing a deferred instruction cache which will be taken before anything runs on it. Signed-off-by: Andrew Waterman <andrew@sifive.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2017-10-26 04:30:32 +07:00
RISC-V: `sfence.vma` orderes the instruction cache This is just a comment change, but it's one that bit me on the mailing list. It turns out that issuing a `sfence.vma` enforces instruction cache ordering in addition to TLB ordering. This isn't explicitly called out in the ISA manual, but Andrew will be making that more clear in a future revision. CC: Andrew Waterman <andrew@sifive.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2017-11-29 05:06:17 +07:00
/*
* Flush entire local TLB. 'sfence.vma' implicitly fences with the instruction
* cache as well, so a 'fence.i' is not necessary.
*/
RISC-V: Atomic and Locking Code This contains all the code that directly interfaces with the RISC-V memory model. While this code corforms to the current RISC-V ISA specifications (user 2.2 and priv 1.10), the memory model is somewhat underspecified in those documents. There is a working group that hopes to produce a formal memory model by the end of the year, but my understanding is that the basic definitions we're relying on here won't change significantly. Reviewed-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 08:02:19 +07:00
static inline void local_flush_tlb_all(void)
{
__asm__ __volatile__ ("sfence.vma" : : : "memory");
}
/* Flush one page from local TLB */
static inline void local_flush_tlb_page(unsigned long addr)
{
__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory");
}
#ifndef CONFIG_SMP
#define flush_tlb_all() local_flush_tlb_all()
#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr)
RISC-V: Limit the scope of TLB shootdowns RISC-V systems perform TLB shootdows via the SBI, which currently performs an IPI to each of the remote harts which then performs a local TLB flush. This process is a bit on the slow side, but we can at least speed it up for some common cases by restricting the set of harts to shoot down to the actual set of harts that are currently participating in the given mm context, as opposed to the entire system. This should provide a measurable performance increase, but we haven't measured it. Regardless, it seems like obviously the right thing to do here. Signed-off-by: Andrew Waterman <andrew@sifive.com> Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-10-26 15:53:40 +07:00
static inline void flush_tlb_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end)
{
local_flush_tlb_all();
}
#define flush_tlb_mm(mm) flush_tlb_all()
RISC-V: Atomic and Locking Code This contains all the code that directly interfaces with the RISC-V memory model. While this code corforms to the current RISC-V ISA specifications (user 2.2 and priv 1.10), the memory model is somewhat underspecified in those documents. There is a working group that hopes to produce a formal memory model by the end of the year, but my understanding is that the basic definitions we're relying on here won't change significantly. Reviewed-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 08:02:19 +07:00
#else /* CONFIG_SMP */
#include <asm/sbi.h>
RISC-V: Use Linux logical CPU number instead of hartid Setup the cpu_logical_map during boot. Moreover, every SBI call and PLIC context are based on the physical hartid. Use the logical CPU to hartid mapping to pass correct hartid to respective functions. Signed-off-by: Atish Patra <atish.patra@wdc.com> Reviewed-by: Anup Patel <anup@brainfault.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-10-03 02:15:05 +07:00
static inline void remote_sfence_vma(struct cpumask *cmask, unsigned long start,
unsigned long size)
{
struct cpumask hmask;
cpumask_clear(&hmask);
riscv_cpuid_to_hartid_mask(cmask, &hmask);
sbi_remote_sfence_vma(hmask.bits, start, size);
}
riscv: use NULL instead of a plain 0 sbi_remote_sfence_vma() & sbi_remote_fence_i() takes a pointer as first argument but some macros call them with a plain 0 which, while legal C, is frowned upon in the kernel. Change this by replacing the 0 by NULL. Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-06-01 22:21:21 +07:00
#define flush_tlb_all() sbi_remote_sfence_vma(NULL, 0, -1)
RISC-V: Atomic and Locking Code This contains all the code that directly interfaces with the RISC-V memory model. While this code corforms to the current RISC-V ISA specifications (user 2.2 and priv 1.10), the memory model is somewhat underspecified in those documents. There is a working group that hopes to produce a formal memory model by the end of the year, but my understanding is that the basic definitions we're relying on here won't change significantly. Reviewed-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 08:02:19 +07:00
#define flush_tlb_page(vma, addr) flush_tlb_range(vma, addr, 0)
#define flush_tlb_range(vma, start, end) \
RISC-V: Use Linux logical CPU number instead of hartid Setup the cpu_logical_map during boot. Moreover, every SBI call and PLIC context are based on the physical hartid. Use the logical CPU to hartid mapping to pass correct hartid to respective functions. Signed-off-by: Atish Patra <atish.patra@wdc.com> Reviewed-by: Anup Patel <anup@brainfault.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-10-03 02:15:05 +07:00
remote_sfence_vma(mm_cpumask((vma)->vm_mm), start, (end) - (start))
RISC-V: Limit the scope of TLB shootdowns RISC-V systems perform TLB shootdows via the SBI, which currently performs an IPI to each of the remote harts which then performs a local TLB flush. This process is a bit on the slow side, but we can at least speed it up for some common cases by restricting the set of harts to shoot down to the actual set of harts that are currently participating in the given mm context, as opposed to the entire system. This should provide a measurable performance increase, but we haven't measured it. Regardless, it seems like obviously the right thing to do here. Signed-off-by: Andrew Waterman <andrew@sifive.com> Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-10-26 15:53:40 +07:00
#define flush_tlb_mm(mm) \
RISC-V: Use Linux logical CPU number instead of hartid Setup the cpu_logical_map during boot. Moreover, every SBI call and PLIC context are based on the physical hartid. Use the logical CPU to hartid mapping to pass correct hartid to respective functions. Signed-off-by: Atish Patra <atish.patra@wdc.com> Reviewed-by: Anup Patel <anup@brainfault.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-10-03 02:15:05 +07:00
remote_sfence_vma(mm_cpumask(mm), 0, -1)
RISC-V: Atomic and Locking Code This contains all the code that directly interfaces with the RISC-V memory model. While this code corforms to the current RISC-V ISA specifications (user 2.2 and priv 1.10), the memory model is somewhat underspecified in those documents. There is a working group that hopes to produce a formal memory model by the end of the year, but my understanding is that the basic definitions we're relying on here won't change significantly. Reviewed-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 08:02:19 +07:00
#endif /* CONFIG_SMP */
/* Flush a range of kernel pages */
static inline void flush_tlb_kernel_range(unsigned long start,
unsigned long end)
{
flush_tlb_all();
}
#endif /* _ASM_RISCV_TLBFLUSH_H */
Reference in New Issue Copy Permalink