AuxXxilium/linux_dsm_epyc7002
1
0
Fork 0
mirror of https://github.com/AuxXxilium/linux_dsm_epyc7002.git synced 2024-12-25 16:55:17 +07:00
Code Issues Actions Packages Projects Releases Wiki Activity
35e6de3885
linux_dsm_epyc7002/arch/mips/include/asm/barrier.h

228 lines
8.0 KiB
C
Raw Normal View History

[MIPS] Cleanup memory barriers for weakly ordered systems. Also the R4000 / R4600 LL/SC instructions imply a sync so no explicit sync needed. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-10-31 10:45:07 +07:00
/*
* This file is subject to the terms and conditions of the GNU General Public
* License. See the file "COPYING" in the main directory of this archive
* for more details.
*
* Copyright (C) 2006 by Ralf Baechle (ralf@linux-mips.org)
*/
#ifndef __ASM_BARRIER_H
#define __ASM_BARRIER_H
Disintegrate asm/system.h for MIPS Disintegrate asm/system.h for MIPS. Signed-off-by: David Howells <dhowells@redhat.com> Acked-by: Ralf Baechle <ralf@linux-mips.org> cc: linux-mips@linux-mips.org
2012-03-29 00:30:02 +07:00
#include <asm/addrspace.h>
MIPS: Barrier: Add definitions of SYNC stype values Add the definitions of sync stype 0 (global completion barrier) and sync stype 0x10 (local ordering barrier) to barrier.h for use with the sync instruction. These types are defined by the MIPS Instruction Set since R2 of the architecture and are documented in document MD00087 table 6.5. Signed-off-by: Matt Redfearn <matt.redfearn@imgtec.com> Reviewed-by: Paul Burton <paul.burton@imgtec.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Michael S. Tsirkin <mst@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/14222/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-09-07 16:45:13 +07:00
/*
* Sync types defined by the MIPS architecture (document MD00087 table 6.5)
* These values are used with the sync instruction to perform memory barriers.
* Types of ordering guarantees available through the SYNC instruction:
* - Completion Barriers
* - Ordering Barriers
* As compared to the completion barrier, the ordering barrier is a
* lighter-weight operation as it does not require the specified instructions
* before the SYNC to be already completed. Instead it only requires that those
* specified instructions which are subsequent to the SYNC in the instruction
* stream are never re-ordered for processing ahead of the specified
* instructions which are before the SYNC in the instruction stream.
* This potentially reduces how many cycles the barrier instruction must stall
* before it completes.
* Implementations that do not use any of the non-zero values of stype to define
* different barriers, such as ordering barriers, must make those stype values
* act the same as stype zero.
*/
/*
* Completion barriers:
* - Every synchronizable specified memory instruction (loads or stores or both)
* that occurs in the instruction stream before the SYNC instruction must be
* already globally performed before any synchronizable specified memory
* instructions that occur after the SYNC are allowed to be performed, with
* respect to any other processor or coherent I/O module.
*
* - The barrier does not guarantee the order in which instruction fetches are
* performed.
*
* - A stype value of zero will always be defined such that it performs the most
* complete set of synchronization operations that are defined.This means
* stype zero always does a completion barrier that affects both loads and
* stores preceding the SYNC instruction and both loads and stores that are
* subsequent to the SYNC instruction. Non-zero values of stype may be defined
* by the architecture or specific implementations to perform synchronization
* behaviors that are less complete than that of stype zero. If an
* implementation does not use one of these non-zero values to define a
* different synchronization behavior, then that non-zero value of stype must
* act the same as stype zero completion barrier. This allows software written
* for an implementation with a lighter-weight barrier to work on another
* implementation which only implements the stype zero completion barrier.
*
* - A completion barrier is required, potentially in conjunction with SSNOP (in
* Release 1 of the Architecture) or EHB (in Release 2 of the Architecture),
* to guarantee that memory reference results are visible across operating
* mode changes. For example, a completion barrier is required on some
* implementations on entry to and exit from Debug Mode to guarantee that
* memory effects are handled correctly.
*/
/*
* stype 0 - A completion barrier that affects preceding loads and stores and
* subsequent loads and stores.
* Older instructions which must reach the load/store ordering point before the
* SYNC instruction completes: Loads, Stores
* Younger instructions which must reach the load/store ordering point only
* after the SYNC instruction completes: Loads, Stores
* Older instructions which must be globally performed when the SYNC instruction
* completes: Loads, Stores
*/
#define STYPE_SYNC 0x0
/*
* Ordering barriers:
* - Every synchronizable specified memory instruction (loads or stores or both)
* that occurs in the instruction stream before the SYNC instruction must
* reach a stage in the load/store datapath after which no instruction
* re-ordering is possible before any synchronizable specified memory
* instruction which occurs after the SYNC instruction in the instruction
* stream reaches the same stage in the load/store datapath.
*
* - If any memory instruction before the SYNC instruction in program order,
* generates a memory request to the external memory and any memory
* instruction after the SYNC instruction in program order also generates a
* memory request to external memory, the memory request belonging to the
* older instruction must be globally performed before the time the memory
* request belonging to the younger instruction is globally performed.
*
* - The barrier does not guarantee the order in which instruction fetches are
* performed.
*/
/*
* stype 0x10 - An ordering barrier that affects preceding loads and stores and
* subsequent loads and stores.
* Older instructions which must reach the load/store ordering point before the
* SYNC instruction completes: Loads, Stores
* Younger instructions which must reach the load/store ordering point only
* after the SYNC instruction completes: Loads, Stores
* Older instructions which must be globally performed when the SYNC instruction
* completes: N/A
*/
#define STYPE_SYNC_MB 0x10
[MIPS] Cleanup memory barriers for weakly ordered systems. Also the R4000 / R4600 LL/SC instructions imply a sync so no explicit sync needed. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-10-31 10:45:07 +07:00
#ifdef CONFIG_CPU_HAS_SYNC
#define __sync() \
__asm__ __volatile__( \
".set push\n\t" \
".set noreorder\n\t" \
".set mips2\n\t" \
"sync\n\t" \
".set pop" \
: /* no output */ \
: /* no input */ \
: "memory")
#else
#define __sync() do { } while(0)
#endif
#define __fast_iob() \
__asm__ __volatile__( \
".set push\n\t" \
".set noreorder\n\t" \
"lw $0,%0\n\t" \
"nop\n\t" \
".set pop" \
: /* no output */ \
: "m" (*(int *)CKSEG1) \
: "memory")
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
#ifdef CONFIG_CPU_CAVIUM_OCTEON
# define OCTEON_SYNCW_STR ".set push\n.set arch=octeon\nsyncw\nsyncw\n.set pop\n"
MIPS: Whitespace cleanup. Having received another series of whitespace patches I decided to do this once and for all rather than dealing with this kind of patches trickling in forever. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2013-01-22 18:59:30 +07:00
# define __syncw() __asm__ __volatile__(OCTEON_SYNCW_STR : : : "memory")
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
# define fast_wmb() __syncw()
# define fast_rmb() barrier()
# define fast_mb() __sync()
# define fast_iob() do { } while (0)
#else /* ! CONFIG_CPU_CAVIUM_OCTEON */
# define fast_wmb() __sync()
# define fast_rmb() __sync()
# define fast_mb() __sync()
# ifdef CONFIG_SGI_IP28
# define fast_iob() \
[MIPS] IP28: switch to "normal" mode after PROM no longer needed SGI-IP28 is running in so called slow mode, when kernel is started from the PROM. PROM calls must be done in slow mode otherwise the PROM will issue an error. To get better memory performance we now switch to normal mode, when the PROM is no longer needed. Signed-off-by: Thomas Bogendoerfer <tsbogend@alpha.franken.de> Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2008-03-19 04:47:56 +07:00
__asm__ __volatile__( \
".set push\n\t" \
".set noreorder\n\t" \
"lw $0,%0\n\t" \
"sync\n\t" \
"lw $0,%0\n\t" \
".set pop" \
: /* no output */ \
: "m" (*(int *)CKSEG1ADDR(0x1fa00004)) \
: "memory")
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
# else
# define fast_iob() \
[MIPS] Cleanup memory barriers for weakly ordered systems. Also the R4000 / R4600 LL/SC instructions imply a sync so no explicit sync needed. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-10-31 10:45:07 +07:00
do { \
__sync(); \
__fast_iob(); \
} while (0)
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
# endif
#endif /* CONFIG_CPU_CAVIUM_OCTEON */
[MIPS] Cleanup memory barriers for weakly ordered systems. Also the R4000 / R4600 LL/SC instructions imply a sync so no explicit sync needed. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-10-31 10:45:07 +07:00
#ifdef CONFIG_CPU_HAS_WB
#include <asm/wbflush.h>
#define mb() wbflush()
#define iob() wbflush()
#else /* !CONFIG_CPU_HAS_WB */
#define mb() fast_mb()
#define iob() fast_iob()
#endif /* !CONFIG_CPU_HAS_WB */
arch: Add lightweight memory barriers dma_rmb() and dma_wmb() There are a number of situations where the mandatory barriers rmb() and wmb() are used to order memory/memory operations in the device drivers and those barriers are much heavier than they actually need to be. For example in the case of PowerPC wmb() calls the heavy-weight sync instruction when for coherent memory operations all that is really needed is an lsync or eieio instruction. This commit adds a coherent only version of the mandatory memory barriers rmb() and wmb(). In most cases this should result in the barrier being the same as the SMP barriers for the SMP case, however in some cases we use a barrier that is somewhere in between rmb() and smp_rmb(). For example on ARM the rmb barriers break down as follows: Barrier Call Explanation --------- -------- ---------------------------------- rmb() dsb() Data synchronization barrier - system dma_rmb() dmb(osh) data memory barrier - outer sharable smp_rmb() dmb(ish) data memory barrier - inner sharable These new barriers are not as safe as the standard rmb() and wmb(). Specifically they do not guarantee ordering between coherent and incoherent memories. The primary use case for these would be to enforce ordering of reads and writes when accessing coherent memory that is shared between the CPU and a device. It may also be noted that there is no dma_mb(). Most architectures don't provide a good mechanism for performing a coherent only full barrier without resorting to the same mechanism used in mb(). As such there isn't much to be gained in trying to define such a function. Cc: Frederic Weisbecker <fweisbec@gmail.com> Cc: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> Cc: Michael Ellerman <michael@ellerman.id.au> Cc: Michael Neuling <mikey@neuling.org> Cc: Russell King <linux@arm.linux.org.uk> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Tony Luck <tony.luck@intel.com> Cc: Oleg Nesterov <oleg@redhat.com> Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Ingo Molnar <mingo@kernel.org> Cc: David Miller <davem@davemloft.net> Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Alexander Duyck <alexander.h.duyck@redhat.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2014-12-12 06:02:06 +07:00
#define wmb() fast_wmb()
#define rmb() fast_rmb()
mips: define __smp_xxx This defines __smp_xxx barriers for mips, for use by virtualization. smp_xxx barriers are removed as they are defined correctly by asm-generic/barriers.h Note: the only exception is smp_mb__before_llsc which is mips-specific. We define both the __smp_mb__before_llsc variant (for use in asm/barriers.h) and smp_mb__before_llsc (for use elsewhere on this architecture). Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2015-12-27 20:04:42 +07:00
#if defined(CONFIG_WEAK_ORDERING)
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
# ifdef CONFIG_CPU_CAVIUM_OCTEON
mips: define __smp_xxx This defines __smp_xxx barriers for mips, for use by virtualization. smp_xxx barriers are removed as they are defined correctly by asm-generic/barriers.h Note: the only exception is smp_mb__before_llsc which is mips-specific. We define both the __smp_mb__before_llsc variant (for use in asm/barriers.h) and smp_mb__before_llsc (for use elsewhere on this architecture). Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2015-12-27 20:04:42 +07:00
# define __smp_mb() __sync()
# define __smp_rmb() barrier()
# define __smp_wmb() __syncw()
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
# else
mips: define __smp_xxx This defines __smp_xxx barriers for mips, for use by virtualization. smp_xxx barriers are removed as they are defined correctly by asm-generic/barriers.h Note: the only exception is smp_mb__before_llsc which is mips-specific. We define both the __smp_mb__before_llsc variant (for use in asm/barriers.h) and smp_mb__before_llsc (for use elsewhere on this architecture). Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2015-12-27 20:04:42 +07:00
# define __smp_mb() __asm__ __volatile__("sync" : : :"memory")
# define __smp_rmb() __asm__ __volatile__("sync" : : :"memory")
# define __smp_wmb() __asm__ __volatile__("sync" : : :"memory")
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
# endif
[MIPS] Cleanup memory barriers for weakly ordered systems. Also the R4000 / R4600 LL/SC instructions imply a sync so no explicit sync needed. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-10-31 10:45:07 +07:00
#else
mips: define __smp_xxx This defines __smp_xxx barriers for mips, for use by virtualization. smp_xxx barriers are removed as they are defined correctly by asm-generic/barriers.h Note: the only exception is smp_mb__before_llsc which is mips-specific. We define both the __smp_mb__before_llsc variant (for use in asm/barriers.h) and smp_mb__before_llsc (for use elsewhere on this architecture). Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2015-12-27 20:04:42 +07:00
#define __smp_mb() barrier()
#define __smp_rmb() barrier()
#define __smp_wmb() barrier()
[MIPS] Cleanup memory barriers for weakly ordered systems. Also the R4000 / R4600 LL/SC instructions imply a sync so no explicit sync needed. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-10-31 10:45:07 +07:00
#endif
MIPS: New macro smp_mb__before_llsc. Replace some instances of smp_llsc_mb() with a new macro smp_mb__before_llsc(). It is used before ll/sc sequences that are documented as needing write barrier semantics. The default implementation of smp_mb__before_llsc() is just smp_llsc_mb(), so there are no changes in semantics. Also simplify definition of smp_mb(), smp_rmb(), and smp_wmb() to be just barrier() in the non-SMP case. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/851/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:43 +07:00
[MIPS] Make support for weakly ordered LL/SC a config option. None of weakly ordered processor supported in tree need this but it seems like this could change ... Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-07-14 19:24:05 +07:00
#if defined(CONFIG_WEAK_REORDERING_BEYOND_LLSC) && defined(CONFIG_SMP)
MIPS: Whitespace cleanup. Having received another series of whitespace patches I decided to do this once and for all rather than dealing with this kind of patches trickling in forever. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2013-01-22 18:59:30 +07:00
#define __WEAK_LLSC_MB " sync \n"
[MIPS] Make support for weakly ordered LL/SC a config option. None of weakly ordered processor supported in tree need this but it seems like this could change ... Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-07-14 19:24:05 +07:00
#else
#define __WEAK_LLSC_MB " \n"
#endif
[MIPS] Cleanup memory barriers for weakly ordered systems. Also the R4000 / R4600 LL/SC instructions imply a sync so no explicit sync needed. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-10-31 10:45:07 +07:00
[MIPS] Make support for weakly ordered LL/SC a config option. None of weakly ordered processor supported in tree need this but it seems like this could change ... Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2007-07-14 19:24:05 +07:00
#define smp_llsc_mb() __asm__ __volatile__(__WEAK_LLSC_MB : : :"memory")
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
#ifdef CONFIG_CPU_CAVIUM_OCTEON
#define smp_mb__before_llsc() smp_wmb()
mips: define __smp_xxx This defines __smp_xxx barriers for mips, for use by virtualization. smp_xxx barriers are removed as they are defined correctly by asm-generic/barriers.h Note: the only exception is smp_mb__before_llsc which is mips-specific. We define both the __smp_mb__before_llsc variant (for use in asm/barriers.h) and smp_mb__before_llsc (for use elsewhere on this architecture). Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2015-12-27 20:04:42 +07:00
#define __smp_mb__before_llsc() __smp_wmb()
MIPS: Optimize spinlocks. The current locking mechanism uses a ll/sc sequence to release a spinlock. This is slower than a wmb() followed by a store to unlock. The branching forward to .subsection 2 on sc failure slows down the contended case. So we get rid of that part too. Since we are now working on naturally aligned u16 values, we can get rid of a masking operation as the LHU already does the right thing. The ANDI are reversed for better scheduling on multi-issue CPUs On a 12 CPU 750MHz Octeon cn5750 this patch improves ipv4 UDP packet forwarding rates from 3.58*10^6 PPS to 3.99*10^6 PPS, or about 11%. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/937/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-02-05 02:31:49 +07:00
/* Cause previous writes to become visible on all CPUs as soon as possible */
#define nudge_writes() __asm__ __volatile__(".set push\n\t" \
".set arch=octeon\n\t" \
"syncw\n\t" \
".set pop" : : : "memory")
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
#else
MIPS: New macro smp_mb__before_llsc. Replace some instances of smp_llsc_mb() with a new macro smp_mb__before_llsc(). It is used before ll/sc sequences that are documented as needing write barrier semantics. The default implementation of smp_mb__before_llsc() is just smp_llsc_mb(), so there are no changes in semantics. Also simplify definition of smp_mb(), smp_rmb(), and smp_wmb() to be just barrier() in the non-SMP case. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/851/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:43 +07:00
#define smp_mb__before_llsc() smp_llsc_mb()
mips: define __smp_xxx This defines __smp_xxx barriers for mips, for use by virtualization. smp_xxx barriers are removed as they are defined correctly by asm-generic/barriers.h Note: the only exception is smp_mb__before_llsc which is mips-specific. We define both the __smp_mb__before_llsc variant (for use in asm/barriers.h) and smp_mb__before_llsc (for use elsewhere on this architecture). Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2015-12-27 20:04:42 +07:00
#define __smp_mb__before_llsc() smp_llsc_mb()
MIPS: Optimize spinlocks. The current locking mechanism uses a ll/sc sequence to release a spinlock. This is slower than a wmb() followed by a store to unlock. The branching forward to .subsection 2 on sc failure slows down the contended case. So we get rid of that part too. Since we are now working on naturally aligned u16 values, we can get rid of a masking operation as the LHU already does the right thing. The ANDI are reversed for better scheduling on multi-issue CPUs On a 12 CPU 750MHz Octeon cn5750 this patch improves ipv4 UDP packet forwarding rates from 3.58*10^6 PPS to 3.99*10^6 PPS, or about 11%. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/937/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-02-05 02:31:49 +07:00
#define nudge_writes() mb()
MIPS: Octeon: Use optimized memory barrier primitives. In order to achieve correct synchronization semantics, the Octeon port had defined CONFIG_WEAK_REORDERING_BEYOND_LLSC. This resulted in code that looks like: sync ll ... . . . sc ... . . sync The second SYNC was redundant, but harmless. Octeon has a SYNCW instruction that acts as a write-memory-barrier (due to an erratum in some parts two SYNCW are used). It is much faster than SYNC because it imposes ordering on the writes, but doesn't otherwise stall the execution pipeline. On Octeon, SYNC stalls execution until all preceeding writes are committed to the coherent memory system. Using: syncw;syncw ll . . . sc . . Has identical semantics to the first sequence, but is much faster. The SYNCW orders the writes, and the SC will not complete successfully until the write is committed to the coherent memory system. So at the end all preceeding writes have been committed. Since Octeon does not do speculative reads, this functions as a full barrier. The patch removes CONFIG_WEAK_REORDERING_BEYOND_LLSC, and substitutes SYNCW for SYNC in write-memory-barriers. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/850/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:44 +07:00
#endif
MIPS: New macro smp_mb__before_llsc. Replace some instances of smp_llsc_mb() with a new macro smp_mb__before_llsc(). It is used before ll/sc sequences that are documented as needing write barrier semantics. The default implementation of smp_mb__before_llsc() is just smp_llsc_mb(), so there are no changes in semantics. Also simplify definition of smp_mb(), smp_rmb(), and smp_wmb() to be just barrier() in the non-SMP case. Signed-off-by: David Daney <ddaney@caviumnetworks.com> To: linux-mips@linux-mips.org Patchwork: http://patchwork.linux-mips.org/patch/851/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2010-01-09 08:17:43 +07:00
mips: define __smp_xxx This defines __smp_xxx barriers for mips, for use by virtualization. smp_xxx barriers are removed as they are defined correctly by asm-generic/barriers.h Note: the only exception is smp_mb__before_llsc which is mips-specific. We define both the __smp_mb__before_llsc variant (for use in asm/barriers.h) and smp_mb__before_llsc (for use elsewhere on this architecture). Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2015-12-27 20:04:42 +07:00
#define __smp_mb__before_atomic() __smp_mb__before_llsc()
#define __smp_mb__after_atomic() smp_llsc_mb()
arch,mips: Convert smp_mb__*() MIPS is interesting and has hardware variants that reorder over ll/sc as well as those that do not. Implement the 2 new barrier functions as per the old barriers. Signed-off-by: Peter Zijlstra <peterz@infradead.org> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Link: http://lkml.kernel.org/n/tip-9ph49jbae3hol9v721sbc2g6@git.kernel.org Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Maciej W. Rozycki" <macro@codesourcery.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Will Deacon <will.deacon@arm.com> Cc: linux-kernel@vger.kernel.org Cc: linux-mips@linux-mips.org Signed-off-by: Ingo Molnar <mingo@kernel.org>
2014-03-14 01:00:36 +07:00
mips: reuse asm-generic/barrier.h On mips dma_rmb, dma_wmb, smp_store_mb, read_barrier_depends, smp_read_barrier_depends, smp_store_release and smp_load_acquire match the asm-generic variants exactly. Drop the local definitions and pull in asm-generic/barrier.h instead. This is in preparation to refactoring this code area. Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Arnd Bergmann <arnd@arndb.de> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
2015-12-21 14:22:18 +07:00
#include <asm-generic/barrier.h>
[MIPS] Cleanup memory barriers for weakly ordered systems. Also the R4000 / R4600 LL/SC instructions imply a sync so no explicit sync needed. Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2006-10-31 10:45:07 +07:00
#endif /* __ASM_BARRIER_H */
Reference in New Issue Copy Permalink