AuxXxilium/linux_dsm_epyc7002
1
0
Fork 0
mirror of https://github.com/AuxXxilium/linux_dsm_epyc7002.git synced 2024-12-26 04:55:16 +07:00
Code Issues Actions Packages Projects Releases Wiki Activity

READ_ONCE: Drop pointer qualifiers when reading from scalar types

Browse Source 
Passing a volatile-qualified pointer to READ_ONCE() is an absolute
trainwreck for code generation: the use of 'typeof()' to define a
temporary variable inside the macro means that the final evaluation in
macro scope ends up forcing a read back from the stack. When stack
protector is enabled (the default for arm64, at least), this causes
the compiler to vomit up all sorts of junk.

Unfortunately, dropping pointer qualifiers inside the macro poses quite
a challenge, especially since the pointed-to type is permitted to be an
aggregate, and this is relied upon by mm/ code accessing things like
'pmd_t'. Based on numerous hacks and discussions on the mailing list,
this is the best I've managed to come up with.

Introduce '__unqual_scalar_typeof()' which takes an expression and, if
the expression is an optionally qualified 8, 16, 32 or 64-bit scalar
type, evaluates to the unqualified type. Other input types, including
aggregates, remain unchanged. Hopefully READ_ONCE() on volatile aggregate
pointers isn't something we do on a fast-path.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Reported-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Will Deacon <will@kernel.org>
This commit is contained in:
Will Deacon 2019-12-19 14:22:31 +00:00
parent 9e343b467c
commit dee081bf8f
2 changed files with 24 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
include/linux/compiler.h
View File

@ -203,13 +203,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
* atomicity or dependency ordering guarantees. Note that this may result
* in tears!
*/
#define __READ_ONCE(x) (*(const volatile typeof(x) *)&(x))
#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
#define __READ_ONCE_SCALAR(x) \
({ \
typeof(x) __x = __READ_ONCE(x); \
__unqual_scalar_typeof(x) __x = __READ_ONCE(x); \
smp_read_barrier_depends(); \
__x; \
(typeof(x))__x; \
})
#define READ_ONCE(x) \

21
include/linux/compiler_types.h
View File

@ -210,6 +210,27 @@ struct ftrace_likely_data {
/* Are two types/vars the same type (ignoring qualifiers)? */
#define __same_type(a, b) __builtin_types_compatible_p(typeof(a), typeof(b))
/*
* __unqual_scalar_typeof(x) - Declare an unqualified scalar type, leaving
* non-scalar types unchanged.
*
* We build this out of a couple of helper macros in a vain attempt to
* help you keep your lunch down while reading it.
*/
#define __pick_scalar_type(x, type, otherwise) \
__builtin_choose_expr(__same_type(x, type), (type)0, otherwise)
#define __pick_integer_type(x, type, otherwise) \
__pick_scalar_type(x, unsigned type, \
__pick_scalar_type(x, signed type, otherwise))
#define __unqual_scalar_typeof(x) typeof( \
__pick_integer_type(x, char, \
__pick_integer_type(x, short, \
__pick_integer_type(x, int, \
__pick_integer_type(x, long, \
__pick_integer_type(x, long long, x))))))
/* Is this type a native word size -- useful for atomic operations */
#define __native_word(t) \
(sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || \