mirror of
https://github.com/AuxXxilium/linux_dsm_epyc7002.git
synced 2024-11-24 06:50:58 +07:00
3c5c3cfb9e
Patch series "kasan: support backing vmalloc space with real shadow
memory", v11.
Currently, vmalloc space is backed by the early shadow page. This means
that kasan is incompatible with VMAP_STACK.
This series provides a mechanism to back vmalloc space with real,
dynamically allocated memory. I have only wired up x86, because that's
the only currently supported arch I can work with easily, but it's very
easy to wire up other architectures, and it appears that there is some
work-in-progress code to do this on arm64 and s390.
This has been discussed before in the context of VMAP_STACK:
- https://bugzilla.kernel.org/show_bug.cgi?id=202009
- https://lkml.org/lkml/2018/7/22/198
- https://lkml.org/lkml/2019/7/19/822
In terms of implementation details:
Most mappings in vmalloc space are small, requiring less than a full
page of shadow space. Allocating a full shadow page per mapping would
therefore be wasteful. Furthermore, to ensure that different mappings
use different shadow pages, mappings would have to be aligned to
KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE.
Instead, share backing space across multiple mappings. Allocate a
backing page when a mapping in vmalloc space uses a particular page of
the shadow region. This page can be shared by other vmalloc mappings
later on.
We hook in to the vmap infrastructure to lazily clean up unused shadow
memory.
Testing with test_vmalloc.sh on an x86 VM with 2 vCPUs shows that:
- Turning on KASAN, inline instrumentation, without vmalloc, introuduces
a 4.1x-4.2x slowdown in vmalloc operations.
- Turning this on introduces the following slowdowns over KASAN:
* ~1.76x slower single-threaded (test_vmalloc.sh performance)
* ~2.18x slower when both cpus are performing operations
simultaneously (test_vmalloc.sh sequential_test_order=1)
This is unfortunate but given that this is a debug feature only, not the
end of the world. The benchmarks are also a stress-test for the vmalloc
subsystem: they're not indicative of an overall 2x slowdown!
This patch (of 4):
Hook into vmalloc and vmap, and dynamically allocate real shadow memory
to back the mappings.
Most mappings in vmalloc space are small, requiring less than a full
page of shadow space. Allocating a full shadow page per mapping would
therefore be wasteful. Furthermore, to ensure that different mappings
use different shadow pages, mappings would have to be aligned to
KASAN_SHADOW_SCALE_SIZE * PAGE_SIZE.
Instead, share backing space across multiple mappings. Allocate a
backing page when a mapping in vmalloc space uses a particular page of
the shadow region. This page can be shared by other vmalloc mappings
later on.
We hook in to the vmap infrastructure to lazily clean up unused shadow
memory.
To avoid the difficulties around swapping mappings around, this code
expects that the part of the shadow region that covers the vmalloc space
will not be covered by the early shadow page, but will be left unmapped.
This will require changes in arch-specific code.
This allows KASAN with VMAP_STACK, and may be helpful for architectures
that do not have a separate module space (e.g. powerpc64, which I am
currently working on). It also allows relaxing the module alignment
back to PAGE_SIZE.
Testing with test_vmalloc.sh on an x86 VM with 2 vCPUs shows that:
- Turning on KASAN, inline instrumentation, without vmalloc, introuduces
a 4.1x-4.2x slowdown in vmalloc operations.
- Turning this on introduces the following slowdowns over KASAN:
* ~1.76x slower single-threaded (test_vmalloc.sh performance)
* ~2.18x slower when both cpus are performing operations
simultaneously (test_vmalloc.sh sequential_test_order=3D1)
This is unfortunate but given that this is a debug feature only, not the
end of the world.
The full benchmark results are:
Performance
No KASAN KASAN original x baseline KASAN vmalloc x baseline x KASAN
fix_size_alloc_test 662004 11404956 17.23 19144610 28.92 1.68
full_fit_alloc_test 710950 12029752 16.92 13184651 18.55 1.10
long_busy_list_alloc_test 9431875 43990172 4.66 82970178 8.80 1.89
random_size_alloc_test 5033626 23061762 4.58 47158834 9.37 2.04
fix_align_alloc_test 1252514 15276910 12.20 31266116 24.96 2.05
random_size_align_alloc_te 1648501 14578321 8.84 25560052 15.51 1.75
align_shift_alloc_test 147 830 5.65 5692 38.72 6.86
pcpu_alloc_test 80732 125520 1.55 140864 1.74 1.12
Total Cycles 119240774314 763211341128 6.40 1390338696894 11.66 1.82
Sequential, 2 cpus
No KASAN KASAN original x baseline KASAN vmalloc x baseline x KASAN
fix_size_alloc_test 1423150 14276550 10.03 27733022 19.49 1.94
full_fit_alloc_test 1754219 14722640 8.39 15030786 8.57 1.02
long_busy_list_alloc_test 11451858 52154973 4.55 107016027 9.34 2.05
random_size_alloc_test 5989020 26735276 4.46 68885923 11.50 2.58
fix_align_alloc_test 2050976 20166900 9.83 50491675 24.62 2.50
random_size_align_alloc_te 2858229 17971700 6.29 38730225 13.55 2.16
align_shift_alloc_test 405 6428 15.87 26253 64.82 4.08
pcpu_alloc_test 127183 151464 1.19 216263 1.70 1.43
Total Cycles 54181269392 308723699764 5.70 650772566394 12.01 2.11
fix_size_alloc_test 1420404 14289308 10.06 27790035 19.56 1.94
full_fit_alloc_test 1736145 14806234 8.53 15274301 8.80 1.03
long_busy_list_alloc_test 11404638 52270785 4.58 107550254 9.43 2.06
random_size_alloc_test 6017006 26650625 4.43 68696127 11.42 2.58
fix_align_alloc_test 2045504 20280985 9.91 50414862 24.65 2.49
random_size_align_alloc_te 2845338 17931018 6.30 38510276 13.53 2.15
align_shift_alloc_test 472 3760 7.97 9656 20.46 2.57
pcpu_alloc_test 118643 132732 1.12 146504 1.23 1.10
Total Cycles 54040011688 309102805492 5.72 651325675652 12.05 2.11
[dja@axtens.net: fixups]
Link: http://lkml.kernel.org/r/20191120052719.7201-1-dja@axtens.net
Link: https://bugzilla.kernel.org/show_bug.cgi?id=3D202009
Link: http://lkml.kernel.org/r/20191031093909.9228-2-dja@axtens.net
Signed-off-by: Mark Rutland <mark.rutland@arm.com> [shadow rework]
Signed-off-by: Daniel Axtens <dja@axtens.net>
Co-developed-by: Mark Rutland <mark.rutland@arm.com>
Acked-by: Vasily Gorbik <gor@linux.ibm.com>
Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Christophe Leroy <christophe.leroy@c-s.fr>
Cc: Qian Cai <cai@lca.pw>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
168 lines
5.9 KiB
Plaintext
168 lines
5.9 KiB
Plaintext
# SPDX-License-Identifier: GPL-2.0-only
|
|
# This config refers to the generic KASAN mode.
|
|
config HAVE_ARCH_KASAN
|
|
bool
|
|
|
|
config HAVE_ARCH_KASAN_SW_TAGS
|
|
bool
|
|
|
|
config HAVE_ARCH_KASAN_VMALLOC
|
|
bool
|
|
|
|
config CC_HAS_KASAN_GENERIC
|
|
def_bool $(cc-option, -fsanitize=kernel-address)
|
|
|
|
config CC_HAS_KASAN_SW_TAGS
|
|
def_bool $(cc-option, -fsanitize=kernel-hwaddress)
|
|
|
|
config KASAN
|
|
bool "KASAN: runtime memory debugger"
|
|
depends on (HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC) || \
|
|
(HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS)
|
|
depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
|
|
help
|
|
Enables KASAN (KernelAddressSANitizer) - runtime memory debugger,
|
|
designed to find out-of-bounds accesses and use-after-free bugs.
|
|
See Documentation/dev-tools/kasan.rst for details.
|
|
|
|
choice
|
|
prompt "KASAN mode"
|
|
depends on KASAN
|
|
default KASAN_GENERIC
|
|
help
|
|
KASAN has two modes: generic KASAN (similar to userspace ASan,
|
|
x86_64/arm64/xtensa, enabled with CONFIG_KASAN_GENERIC) and
|
|
software tag-based KASAN (a version based on software memory
|
|
tagging, arm64 only, similar to userspace HWASan, enabled with
|
|
CONFIG_KASAN_SW_TAGS).
|
|
Both generic and tag-based KASAN are strictly debugging features.
|
|
|
|
config KASAN_GENERIC
|
|
bool "Generic mode"
|
|
depends on HAVE_ARCH_KASAN && CC_HAS_KASAN_GENERIC
|
|
depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
|
|
select SLUB_DEBUG if SLUB
|
|
select CONSTRUCTORS
|
|
select STACKDEPOT
|
|
help
|
|
Enables generic KASAN mode.
|
|
Supported in both GCC and Clang. With GCC it requires version 4.9.2
|
|
or later for basic support and version 5.0 or later for detection of
|
|
out-of-bounds accesses for stack and global variables and for inline
|
|
instrumentation mode (CONFIG_KASAN_INLINE). With Clang it requires
|
|
version 3.7.0 or later and it doesn't support detection of
|
|
out-of-bounds accesses for global variables yet.
|
|
This mode consumes about 1/8th of available memory at kernel start
|
|
and introduces an overhead of ~x1.5 for the rest of the allocations.
|
|
The performance slowdown is ~x3.
|
|
For better error detection enable CONFIG_STACKTRACE.
|
|
Currently CONFIG_KASAN_GENERIC doesn't work with CONFIG_DEBUG_SLAB
|
|
(the resulting kernel does not boot).
|
|
|
|
config KASAN_SW_TAGS
|
|
bool "Software tag-based mode"
|
|
depends on HAVE_ARCH_KASAN_SW_TAGS && CC_HAS_KASAN_SW_TAGS
|
|
depends on (SLUB && SYSFS) || (SLAB && !DEBUG_SLAB)
|
|
select SLUB_DEBUG if SLUB
|
|
select CONSTRUCTORS
|
|
select STACKDEPOT
|
|
help
|
|
Enables software tag-based KASAN mode.
|
|
This mode requires Top Byte Ignore support by the CPU and therefore
|
|
is only supported for arm64.
|
|
This mode requires Clang version 7.0.0 or later.
|
|
This mode consumes about 1/16th of available memory at kernel start
|
|
and introduces an overhead of ~20% for the rest of the allocations.
|
|
This mode may potentially introduce problems relating to pointer
|
|
casting and comparison, as it embeds tags into the top byte of each
|
|
pointer.
|
|
For better error detection enable CONFIG_STACKTRACE.
|
|
Currently CONFIG_KASAN_SW_TAGS doesn't work with CONFIG_DEBUG_SLAB
|
|
(the resulting kernel does not boot).
|
|
|
|
endchoice
|
|
|
|
choice
|
|
prompt "Instrumentation type"
|
|
depends on KASAN
|
|
default KASAN_OUTLINE
|
|
|
|
config KASAN_OUTLINE
|
|
bool "Outline instrumentation"
|
|
help
|
|
Before every memory access compiler insert function call
|
|
__asan_load*/__asan_store*. These functions performs check
|
|
of shadow memory. This is slower than inline instrumentation,
|
|
however it doesn't bloat size of kernel's .text section so
|
|
much as inline does.
|
|
|
|
config KASAN_INLINE
|
|
bool "Inline instrumentation"
|
|
help
|
|
Compiler directly inserts code checking shadow memory before
|
|
memory accesses. This is faster than outline (in some workloads
|
|
it gives about x2 boost over outline instrumentation), but
|
|
make kernel's .text size much bigger.
|
|
For CONFIG_KASAN_GENERIC this requires GCC 5.0 or later.
|
|
|
|
endchoice
|
|
|
|
config KASAN_STACK_ENABLE
|
|
bool "Enable stack instrumentation (unsafe)" if CC_IS_CLANG && !COMPILE_TEST
|
|
depends on KASAN
|
|
help
|
|
The LLVM stack address sanitizer has a know problem that
|
|
causes excessive stack usage in a lot of functions, see
|
|
https://bugs.llvm.org/show_bug.cgi?id=38809
|
|
Disabling asan-stack makes it safe to run kernels build
|
|
with clang-8 with KASAN enabled, though it loses some of
|
|
the functionality.
|
|
This feature is always disabled when compile-testing with clang
|
|
to avoid cluttering the output in stack overflow warnings,
|
|
but clang users can still enable it for builds without
|
|
CONFIG_COMPILE_TEST. On gcc it is assumed to always be safe
|
|
to use and enabled by default.
|
|
|
|
config KASAN_STACK
|
|
int
|
|
default 1 if KASAN_STACK_ENABLE || CC_IS_GCC
|
|
default 0
|
|
|
|
config KASAN_S390_4_LEVEL_PAGING
|
|
bool "KASan: use 4-level paging"
|
|
depends on KASAN && S390
|
|
help
|
|
Compiling the kernel with KASan disables automatic 3-level vs
|
|
4-level paging selection. 3-level paging is used by default (up
|
|
to 3TB of RAM with KASan enabled). This options allows to force
|
|
4-level paging instead.
|
|
|
|
config KASAN_SW_TAGS_IDENTIFY
|
|
bool "Enable memory corruption identification"
|
|
depends on KASAN_SW_TAGS
|
|
help
|
|
This option enables best-effort identification of bug type
|
|
(use-after-free or out-of-bounds) at the cost of increased
|
|
memory consumption.
|
|
|
|
config KASAN_VMALLOC
|
|
bool "Back mappings in vmalloc space with real shadow memory"
|
|
depends on KASAN && HAVE_ARCH_KASAN_VMALLOC
|
|
help
|
|
By default, the shadow region for vmalloc space is the read-only
|
|
zero page. This means that KASAN cannot detect errors involving
|
|
vmalloc space.
|
|
|
|
Enabling this option will hook in to vmap/vmalloc and back those
|
|
mappings with real shadow memory allocated on demand. This allows
|
|
for KASAN to detect more sorts of errors (and to support vmapped
|
|
stacks), but at the cost of higher memory usage.
|
|
|
|
config TEST_KASAN
|
|
tristate "Module for testing KASAN for bug detection"
|
|
depends on m && KASAN
|
|
help
|
|
This is a test module doing various nasty things like
|
|
out of bounds accesses, use after free. It is useful for testing
|
|
kernel debugging features like KASAN.
|