License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 21:07:57 +07:00
# SPDX-License-Identifier: GPL-2.0
2007-10-16 03:25:06 +07:00
# Backward compatibility
asflags-y += $( EXTRA_AFLAGS)
ccflags-y += $( EXTRA_CFLAGS)
cppflags-y += $( EXTRA_CPPFLAGS)
ldflags-y += $( EXTRA_LDFLAGS)
2017-10-09 10:49:11 +07:00
# flags that take effect in current and sub directories
2017-10-10 18:43:21 +07:00
KBUILD_AFLAGS += $( subdir-asflags-y)
KBUILD_CFLAGS += $( subdir-ccflags-y)
2009-04-19 16:04:26 +07:00
2005-04-17 05:20:36 +07:00
# Figure out what we need to build from the various variables
# ===========================================================================
# When an object is listed to be built compiled-in and modular,
# only build the compiled-in version
obj-m := $( filter-out $( obj-y) ,$( obj-m) )
# Libraries are always collected in one lib file.
# Filter out objects already built-in
lib-y := $( filter-out $( obj-y) , $( sort $( lib-y) $( lib-m) ) )
2017-10-09 10:49:11 +07:00
# Determine modorder.
# Unfortunately, we don't have information about ordering between -y
# and -m subdirs. Just put -y's first.
modorder := $( patsubst %/,%/modules.order, $( filter %/, $( obj-y) ) $( obj-m:.o= .ko) )
2005-04-17 05:20:36 +07:00
# Handle objects in subdirs
# ---------------------------------------------------------------------------
2018-02-10 21:25:04 +07:00
# o if we encounter foo/ in $(obj-y), replace it by foo/built-in.a
2005-04-17 05:20:36 +07:00
# and add the directory to the list of dirs to descend into: $(subdir-y)
2014-04-28 14:26:18 +07:00
# o if we encounter foo/ in $(obj-m), remove it from $(obj-m)
2005-04-17 05:20:36 +07:00
# and add the directory to the list of dirs to descend into: $(subdir-m)
__subdir-y := $( patsubst %/,%,$( filter %/, $( obj-y) ) )
subdir-y += $( __subdir-y)
__subdir-m := $( patsubst %/,%,$( filter %/, $( obj-m) ) )
subdir-m += $( __subdir-m)
2018-02-10 21:25:04 +07:00
obj-y := $( patsubst %/, %/built-in.a, $( obj-y) )
2005-04-17 05:20:36 +07:00
obj-m := $( filter-out %/, $( obj-m) )
# Subdirectories we need to descend into
subdir-ym := $( sort $( subdir-y) $( subdir-m) )
2019-08-06 13:39:19 +07:00
# If $(foo-objs), $(foo-y), $(foo-m), or $(foo-) exists, foo.o is a composite object
multi-used-y := $( sort $( foreach m,$( obj-y) , $( if $( strip $( $( m:.o= -objs) ) $( $( m:.o= -y) ) $( $( m:.o= -) ) ) , $( m) ) ) )
multi-used-m := $( sort $( foreach m,$( obj-m) , $( if $( strip $( $( m:.o= -objs) ) $( $( m:.o= -y) ) $( $( m:.o= -m) ) $( $( m:.o= -) ) ) , $( m) ) ) )
2005-04-17 05:20:36 +07:00
multi-used := $( multi-used-y) $( multi-used-m)
2007-09-15 13:55:39 +07:00
# $(subdir-obj-y) is the list of objects in $(obj-y) which uses dir/ to
# tell kbuild to descend
2018-02-10 21:25:04 +07:00
subdir-obj-y := $( filter %/built-in.a, $( obj-y) )
2005-04-17 05:20:36 +07:00
kbuild: link $(real-obj-y) instead of $(obj-y) into built-in.a
In Kbuild, Makefiles can add the same object to obj-y multiple
times. So,
obj-y += foo.o
obj-y += foo.o
is fine.
However, this is not true when the same object is added multiple
times via composite objects. For example,
obj-y += foo.o bar.o
foo-objs := foo-bar-common.o foo-only.o
bar-objs := foo-bar-common.o bar-only.o
causes build error because two instances of foo-bar-common.o are
linked into the vmlinux.
Makefiles tend to invent ugly work-around, for example
- lib/zstd/Makefile
- drivers/net/ethernet/cavium/liquidio/Makefile
The technique used in Kbuild to avoid the multiple definition error
is to use $(filter $(obj-y), $^). Here, $^ lists the names of all
the prerequisites with duplicated names removed.
By replacing it with $(filter $(real-obj-y), $^) we can do likewise
for composite objects. For built-in objects, we do not need to keep
the composite object structure. We can simply expand them, and link
$(real-obj-y) to built-in.a.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-03-19 18:26:08 +07:00
# Replace multi-part objects by their individual parts,
# including built-in.a from subdirectories
2019-08-06 13:39:19 +07:00
real-obj-y := $( foreach m, $( obj-y) , $( if $( strip $( $( m:.o= -objs) ) $( $( m:.o= -y) ) $( $( m:.o= -) ) ) ,$( $( m:.o= -objs) ) $( $( m:.o= -y) ) ,$( m) ) )
real-obj-m := $( foreach m, $( obj-m) , $( if $( strip $( $( m:.o= -objs) ) $( $( m:.o= -y) ) $( $( m:.o= -m) ) $( $( m:.o= -) ) ) ,$( $( m:.o= -objs) ) $( $( m:.o= -y) ) $( $( m:.o= -m) ) ,$( m) ) )
2005-04-17 05:20:36 +07:00
2017-11-05 12:30:53 +07:00
# DTB
# If CONFIG_OF_ALL_DTBS is enabled, all DT blobs are built
extra-y += $( dtb-y)
extra-$(CONFIG_OF_ALL_DTBS) += $( dtb-)
2018-09-07 01:26:07 +07:00
i f n e q ( $( CHECK_DTBS ) , )
extra-y += $( patsubst %.dtb,%.dt.yaml, $( dtb-y) )
extra-$(CONFIG_OF_ALL_DTBS) += $( patsubst %.dtb,%.dt.yaml, $( dtb-) )
e n d i f
2005-04-17 05:20:36 +07:00
# Add subdir path
extra-y := $( addprefix $( obj) /,$( extra-y) )
always := $( addprefix $( obj) /,$( always) )
targets := $( addprefix $( obj) /,$( targets) )
2007-12-07 19:04:30 +07:00
modorder := $( addprefix $( obj) /,$( modorder) )
2005-04-17 05:20:36 +07:00
obj-m := $( addprefix $( obj) /,$( obj-m) )
lib-y := $( addprefix $( obj) /,$( lib-y) )
subdir-obj-y := $( addprefix $( obj) /,$( subdir-obj-y) )
2018-03-19 18:26:07 +07:00
real-obj-y := $( addprefix $( obj) /,$( real-obj-y) )
real-obj-m := $( addprefix $( obj) /,$( real-obj-m) )
2005-04-17 05:20:36 +07:00
multi-used-m := $( addprefix $( obj) /,$( multi-used-m) )
subdir-ym := $( addprefix $( obj) /,$( subdir-ym) )
2018-03-19 16:01:27 +07:00
# Finds the multi-part object the current object will be linked into.
# If the object belongs to two or more multi-part objects, all of them are
# concatenated with a colon separator.
modname-multi = $( subst $( space) ,:,$( sort $( foreach m,$( multi-used) ,\
$( if $( filter $* .o, $( $( m:.o= -objs) ) $( $( m:.o= -y) ) $( $( m:.o= -m) ) ) ,$( m:.o= ) ) ) ) )
modname = $( if $( modname-multi) ,$( modname-multi) ,$( basetarget) )
kbuild: change *FLAGS_<basetarget>.o to take the path relative to $(obj)
Kbuild provides per-file compiler flag addition/removal:
CFLAGS_<basetarget>.o
CFLAGS_REMOVE_<basetarget>.o
AFLAGS_<basetarget>.o
AFLAGS_REMOVE_<basetarget>.o
CPPFLAGS_<basetarget>.lds
HOSTCFLAGS_<basetarget>.o
HOSTCXXFLAGS_<basetarget>.o
The <basetarget> is the filename of the target with its directory and
suffix stripped.
This syntax comes into a trouble when two files with the same basename
appear in one Makefile, for example:
obj-y += foo.o
obj-y += dir/foo.o
CFLAGS_foo.o := <some-flags>
Here, the <some-flags> applies to both foo.o and dir/foo.o
The real world problem is:
scripts/kconfig/util.c
scripts/kconfig/lxdialog/util.c
Both files are compiled into scripts/kconfig/mconf, but only the
latter should be given with the ncurses flags.
It is more sensible to use the relative path to the Makefile, like this:
obj-y += foo.o
CFLAGS_foo.o := <some-flags>
obj-y += dir/foo.o
CFLAGS_dir/foo.o := <other-flags>
At first, I attempted to replace $(basetarget) with $*. The $* variable
is replaced with the stem ('%') part in a pattern rule. This works with
most of cases, but does not for explicit rules.
For example, arch/ia64/lib/Makefile reuses rule_as_o_S in its own
explicit rules, so $* will be empty, resulting in ignoring the per-file
AFLAGS.
I introduced a new variable, target-stem, which can be used also from
explicit rules.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Marc Zyngier <maz@kernel.org>
2019-08-30 11:34:01 +07:00
# target with $(obj)/ and its suffix stripped
target-stem = $( basename $( patsubst $( obj) /%,%,$@ ) )
2005-04-17 05:20:36 +07:00
# These flags are needed for modversions and compiling, so we define them here
2017-10-09 10:49:11 +07:00
# $(modname_flags) defines KBUILD_MODNAME as the name of the module it will
2005-04-17 05:20:36 +07:00
# end up in (or would, if it gets compiled in)
2016-03-17 22:32:14 +07:00
name-fix = $( squote) $( quote) $( subst $( comma) ,_,$( subst -,_,$1 ) ) $( quote) $( squote)
basename_flags = -DKBUILD_BASENAME= $( call name-fix,$( basetarget) )
kbuild: define KBUILD_MODNAME even if multiple modules share objects
Currently, KBUILD_MODNAME is defined only when $(modname) contains
just one word. If an object is shared among multiple modules,
undefined KBUILD_MODNAME could cause a build error. For example,
if CONFIG_DYNAMIC_DEBUG is enabled, any call of printk() populates
.modname, then fails to build due to undefined KBUILD_MODNAME.
Take the following code as an example:
obj-m += foo.o
obj-m += bar.o
foo-objs := foo-bar-common.o foo-only.o
bar-objs := foo-bar-common.o bar-only.o
In this case, there is room for argument what to define for
KBUILD_MODNAME when foo-bar-common.o is being compiled.
"foo", "bar", or what else?
One idea is to define colon-separated modules that share the object,
in this case, "bar:foo" (modules are sorted alphabetically by
$(sort ...)).
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Cao jin <caoj.fnst@cn.fujitsu.com>
2018-03-19 16:01:24 +07:00
modname_flags = -DKBUILD_MODNAME= $( call name-fix,$( modname) )
2005-04-17 05:20:36 +07:00
2017-10-10 18:43:21 +07:00
orig_c_flags = $( KBUILD_CPPFLAGS) $( KBUILD_CFLAGS) \
kbuild: change *FLAGS_<basetarget>.o to take the path relative to $(obj)
Kbuild provides per-file compiler flag addition/removal:
CFLAGS_<basetarget>.o
CFLAGS_REMOVE_<basetarget>.o
AFLAGS_<basetarget>.o
AFLAGS_REMOVE_<basetarget>.o
CPPFLAGS_<basetarget>.lds
HOSTCFLAGS_<basetarget>.o
HOSTCXXFLAGS_<basetarget>.o
The <basetarget> is the filename of the target with its directory and
suffix stripped.
This syntax comes into a trouble when two files with the same basename
appear in one Makefile, for example:
obj-y += foo.o
obj-y += dir/foo.o
CFLAGS_foo.o := <some-flags>
Here, the <some-flags> applies to both foo.o and dir/foo.o
The real world problem is:
scripts/kconfig/util.c
scripts/kconfig/lxdialog/util.c
Both files are compiled into scripts/kconfig/mconf, but only the
latter should be given with the ncurses flags.
It is more sensible to use the relative path to the Makefile, like this:
obj-y += foo.o
CFLAGS_foo.o := <some-flags>
obj-y += dir/foo.o
CFLAGS_dir/foo.o := <other-flags>
At first, I attempted to replace $(basetarget) with $*. The $* variable
is replaced with the stem ('%') part in a pattern rule. This works with
most of cases, but does not for explicit rules.
For example, arch/ia64/lib/Makefile reuses rule_as_o_S in its own
explicit rules, so $* will be empty, resulting in ignoring the per-file
AFLAGS.
I introduced a new variable, target-stem, which can be used also from
explicit rules.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Marc Zyngier <maz@kernel.org>
2019-08-30 11:34:01 +07:00
$( ccflags-y) $( CFLAGS_$( target-stem) .o)
_c_flags = $( filter-out $( CFLAGS_REMOVE_$( target-stem) .o) , $( orig_c_flags) )
2017-10-10 18:43:21 +07:00
orig_a_flags = $( KBUILD_CPPFLAGS) $( KBUILD_AFLAGS) \
kbuild: change *FLAGS_<basetarget>.o to take the path relative to $(obj)
Kbuild provides per-file compiler flag addition/removal:
CFLAGS_<basetarget>.o
CFLAGS_REMOVE_<basetarget>.o
AFLAGS_<basetarget>.o
AFLAGS_REMOVE_<basetarget>.o
CPPFLAGS_<basetarget>.lds
HOSTCFLAGS_<basetarget>.o
HOSTCXXFLAGS_<basetarget>.o
The <basetarget> is the filename of the target with its directory and
suffix stripped.
This syntax comes into a trouble when two files with the same basename
appear in one Makefile, for example:
obj-y += foo.o
obj-y += dir/foo.o
CFLAGS_foo.o := <some-flags>
Here, the <some-flags> applies to both foo.o and dir/foo.o
The real world problem is:
scripts/kconfig/util.c
scripts/kconfig/lxdialog/util.c
Both files are compiled into scripts/kconfig/mconf, but only the
latter should be given with the ncurses flags.
It is more sensible to use the relative path to the Makefile, like this:
obj-y += foo.o
CFLAGS_foo.o := <some-flags>
obj-y += dir/foo.o
CFLAGS_dir/foo.o := <other-flags>
At first, I attempted to replace $(basetarget) with $*. The $* variable
is replaced with the stem ('%') part in a pattern rule. This works with
most of cases, but does not for explicit rules.
For example, arch/ia64/lib/Makefile reuses rule_as_o_S in its own
explicit rules, so $* will be empty, resulting in ignoring the per-file
AFLAGS.
I introduced a new variable, target-stem, which can be used also from
explicit rules.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Marc Zyngier <maz@kernel.org>
2019-08-30 11:34:01 +07:00
$( asflags-y) $( AFLAGS_$( target-stem) .o)
_a_flags = $( filter-out $( AFLAGS_REMOVE_$( target-stem) .o) , $( orig_a_flags) )
_cpp_flags = $( KBUILD_CPPFLAGS) $( cppflags-y) $( CPPFLAGS_$( target-stem) .lds)
2005-04-17 05:20:36 +07:00
2009-06-18 06:28:08 +07:00
#
# Enable gcov profiling flags for a file, directory or for all files depending
# on variables GCOV_PROFILE_obj.o, GCOV_PROFILE and CONFIG_GCOV_PROFILE_ALL
# (in this order)
#
i f e q ( $( CONFIG_GCOV_KERNEL ) , y )
_c_flags += $( if $( patsubst n%,, \
$( GCOV_PROFILE_$( basetarget) .o) $( GCOV_PROFILE) $( CONFIG_GCOV_PROFILE_ALL) ) , \
$( CFLAGS_GCOV) )
e n d i f
kasan: add kernel address sanitizer infrastructure
Kernel Address sanitizer (KASan) is a dynamic memory error detector. It
provides fast and comprehensive solution for finding use-after-free and
out-of-bounds bugs.
KASAN uses compile-time instrumentation for checking every memory access,
therefore GCC > v4.9.2 required. v4.9.2 almost works, but has issues with
putting symbol aliases into the wrong section, which breaks kasan
instrumentation of globals.
This patch only adds infrastructure for kernel address sanitizer. It's
not available for use yet. The idea and some code was borrowed from [1].
Basic idea:
The main idea of KASAN is to use shadow memory to record whether each byte
of memory is safe to access or not, and use compiler's instrumentation to
check the shadow memory on each memory access.
Address sanitizer uses 1/8 of the memory addressable in kernel for shadow
memory and uses direct mapping with a scale and offset to translate a
memory address to its corresponding shadow address.
Here is function to translate address to corresponding shadow address:
unsigned long kasan_mem_to_shadow(unsigned long addr)
{
return (addr >> KASAN_SHADOW_SCALE_SHIFT) + KASAN_SHADOW_OFFSET;
}
where KASAN_SHADOW_SCALE_SHIFT = 3.
So for every 8 bytes there is one corresponding byte of shadow memory.
The following encoding used for each shadow byte: 0 means that all 8 bytes
of the corresponding memory region are valid for access; k (1 <= k <= 7)
means that the first k bytes are valid for access, and other (8 - k) bytes
are not; Any negative value indicates that the entire 8-bytes are
inaccessible. Different negative values used to distinguish between
different kinds of inaccessible memory (redzones, freed memory) (see
mm/kasan/kasan.h).
To be able to detect accesses to bad memory we need a special compiler.
Such compiler inserts a specific function calls (__asan_load*(addr),
__asan_store*(addr)) before each memory access of size 1, 2, 4, 8 or 16.
These functions check whether memory region is valid to access or not by
checking corresponding shadow memory. If access is not valid an error
printed.
Historical background of the address sanitizer from Dmitry Vyukov:
"We've developed the set of tools, AddressSanitizer (Asan),
ThreadSanitizer and MemorySanitizer, for user space. We actively use
them for testing inside of Google (continuous testing, fuzzing,
running prod services). To date the tools have found more than 10'000
scary bugs in Chromium, Google internal codebase and various
open-source projects (Firefox, OpenSSL, gcc, clang, ffmpeg, MySQL and
lots of others): [2] [3] [4].
The tools are part of both gcc and clang compilers.
We have not yet done massive testing under the Kernel AddressSanitizer
(it's kind of chicken and egg problem, you need it to be upstream to
start applying it extensively). To date it has found about 50 bugs.
Bugs that we've found in upstream kernel are listed in [5].
We've also found ~20 bugs in out internal version of the kernel. Also
people from Samsung and Oracle have found some.
[...]
As others noted, the main feature of AddressSanitizer is its
performance due to inline compiler instrumentation and simple linear
shadow memory. User-space Asan has ~2x slowdown on computational
programs and ~2x memory consumption increase. Taking into account that
kernel usually consumes only small fraction of CPU and memory when
running real user-space programs, I would expect that kernel Asan will
have ~10-30% slowdown and similar memory consumption increase (when we
finish all tuning).
I agree that Asan can well replace kmemcheck. We have plans to start
working on Kernel MemorySanitizer that finds uses of unitialized
memory. Asan+Msan will provide feature-parity with kmemcheck. As
others noted, Asan will unlikely replace debug slab and pagealloc that
can be enabled at runtime. Asan uses compiler instrumentation, so even
if it is disabled, it still incurs visible overheads.
Asan technology is easily portable to other architectures. Compiler
instrumentation is fully portable. Runtime has some arch-dependent
parts like shadow mapping and atomic operation interception. They are
relatively easy to port."
Comparison with other debugging features:
========================================
KMEMCHECK:
- KASan can do almost everything that kmemcheck can. KASan uses
compile-time instrumentation, which makes it significantly faster than
kmemcheck. The only advantage of kmemcheck over KASan is detection of
uninitialized memory reads.
Some brief performance testing showed that kasan could be
x500-x600 times faster than kmemcheck:
$ netperf -l 30
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost (127.0.0.1) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
no debug: 87380 16384 16384 30.00 41624.72
kasan inline: 87380 16384 16384 30.00 12870.54
kasan outline: 87380 16384 16384 30.00 10586.39
kmemcheck: 87380 16384 16384 30.03 20.23
- Also kmemcheck couldn't work on several CPUs. It always sets
number of CPUs to 1. KASan doesn't have such limitation.
DEBUG_PAGEALLOC:
- KASan is slower than DEBUG_PAGEALLOC, but KASan works on sub-page
granularity level, so it able to find more bugs.
SLUB_DEBUG (poisoning, redzones):
- SLUB_DEBUG has lower overhead than KASan.
- SLUB_DEBUG in most cases are not able to detect bad reads,
KASan able to detect both reads and writes.
- In some cases (e.g. redzone overwritten) SLUB_DEBUG detect
bugs only on allocation/freeing of object. KASan catch
bugs right before it will happen, so we always know exact
place of first bad read/write.
[1] https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel
[2] https://code.google.com/p/address-sanitizer/wiki/FoundBugs
[3] https://code.google.com/p/thread-sanitizer/wiki/FoundBugs
[4] https://code.google.com/p/memory-sanitizer/wiki/FoundBugs
[5] https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel#Trophies
Based on work by Andrey Konovalov.
Signed-off-by: Andrey Ryabinin <a.ryabinin@samsung.com>
Acked-by: Michal Marek <mmarek@suse.cz>
Signed-off-by: Andrey Konovalov <adech.fo@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Konstantin Serebryany <kcc@google.com>
Cc: Dmitry Chernenkov <dmitryc@google.com>
Cc: Yuri Gribov <tetra2005@gmail.com>
Cc: Konstantin Khlebnikov <koct9i@gmail.com>
Cc: Sasha Levin <sasha.levin@oracle.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-14 05:39:17 +07:00
#
# Enable address sanitizer flags for kernel except some files or directories
# we don't want to check (depends on variables KASAN_SANITIZE_obj.o, KASAN_SANITIZE)
#
i f e q ( $( CONFIG_KASAN ) , y )
_c_flags += $( if $( patsubst n%,, \
$( KASAN_SANITIZE_$( basetarget) .o) $( KASAN_SANITIZE) y) , \
2018-02-07 06:36:00 +07:00
$( CFLAGS_KASAN) , $( CFLAGS_KASAN_NOSANITIZE) )
kasan: add kernel address sanitizer infrastructure
Kernel Address sanitizer (KASan) is a dynamic memory error detector. It
provides fast and comprehensive solution for finding use-after-free and
out-of-bounds bugs.
KASAN uses compile-time instrumentation for checking every memory access,
therefore GCC > v4.9.2 required. v4.9.2 almost works, but has issues with
putting symbol aliases into the wrong section, which breaks kasan
instrumentation of globals.
This patch only adds infrastructure for kernel address sanitizer. It's
not available for use yet. The idea and some code was borrowed from [1].
Basic idea:
The main idea of KASAN is to use shadow memory to record whether each byte
of memory is safe to access or not, and use compiler's instrumentation to
check the shadow memory on each memory access.
Address sanitizer uses 1/8 of the memory addressable in kernel for shadow
memory and uses direct mapping with a scale and offset to translate a
memory address to its corresponding shadow address.
Here is function to translate address to corresponding shadow address:
unsigned long kasan_mem_to_shadow(unsigned long addr)
{
return (addr >> KASAN_SHADOW_SCALE_SHIFT) + KASAN_SHADOW_OFFSET;
}
where KASAN_SHADOW_SCALE_SHIFT = 3.
So for every 8 bytes there is one corresponding byte of shadow memory.
The following encoding used for each shadow byte: 0 means that all 8 bytes
of the corresponding memory region are valid for access; k (1 <= k <= 7)
means that the first k bytes are valid for access, and other (8 - k) bytes
are not; Any negative value indicates that the entire 8-bytes are
inaccessible. Different negative values used to distinguish between
different kinds of inaccessible memory (redzones, freed memory) (see
mm/kasan/kasan.h).
To be able to detect accesses to bad memory we need a special compiler.
Such compiler inserts a specific function calls (__asan_load*(addr),
__asan_store*(addr)) before each memory access of size 1, 2, 4, 8 or 16.
These functions check whether memory region is valid to access or not by
checking corresponding shadow memory. If access is not valid an error
printed.
Historical background of the address sanitizer from Dmitry Vyukov:
"We've developed the set of tools, AddressSanitizer (Asan),
ThreadSanitizer and MemorySanitizer, for user space. We actively use
them for testing inside of Google (continuous testing, fuzzing,
running prod services). To date the tools have found more than 10'000
scary bugs in Chromium, Google internal codebase and various
open-source projects (Firefox, OpenSSL, gcc, clang, ffmpeg, MySQL and
lots of others): [2] [3] [4].
The tools are part of both gcc and clang compilers.
We have not yet done massive testing under the Kernel AddressSanitizer
(it's kind of chicken and egg problem, you need it to be upstream to
start applying it extensively). To date it has found about 50 bugs.
Bugs that we've found in upstream kernel are listed in [5].
We've also found ~20 bugs in out internal version of the kernel. Also
people from Samsung and Oracle have found some.
[...]
As others noted, the main feature of AddressSanitizer is its
performance due to inline compiler instrumentation and simple linear
shadow memory. User-space Asan has ~2x slowdown on computational
programs and ~2x memory consumption increase. Taking into account that
kernel usually consumes only small fraction of CPU and memory when
running real user-space programs, I would expect that kernel Asan will
have ~10-30% slowdown and similar memory consumption increase (when we
finish all tuning).
I agree that Asan can well replace kmemcheck. We have plans to start
working on Kernel MemorySanitizer that finds uses of unitialized
memory. Asan+Msan will provide feature-parity with kmemcheck. As
others noted, Asan will unlikely replace debug slab and pagealloc that
can be enabled at runtime. Asan uses compiler instrumentation, so even
if it is disabled, it still incurs visible overheads.
Asan technology is easily portable to other architectures. Compiler
instrumentation is fully portable. Runtime has some arch-dependent
parts like shadow mapping and atomic operation interception. They are
relatively easy to port."
Comparison with other debugging features:
========================================
KMEMCHECK:
- KASan can do almost everything that kmemcheck can. KASan uses
compile-time instrumentation, which makes it significantly faster than
kmemcheck. The only advantage of kmemcheck over KASan is detection of
uninitialized memory reads.
Some brief performance testing showed that kasan could be
x500-x600 times faster than kmemcheck:
$ netperf -l 30
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost (127.0.0.1) port 0 AF_INET
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
no debug: 87380 16384 16384 30.00 41624.72
kasan inline: 87380 16384 16384 30.00 12870.54
kasan outline: 87380 16384 16384 30.00 10586.39
kmemcheck: 87380 16384 16384 30.03 20.23
- Also kmemcheck couldn't work on several CPUs. It always sets
number of CPUs to 1. KASan doesn't have such limitation.
DEBUG_PAGEALLOC:
- KASan is slower than DEBUG_PAGEALLOC, but KASan works on sub-page
granularity level, so it able to find more bugs.
SLUB_DEBUG (poisoning, redzones):
- SLUB_DEBUG has lower overhead than KASan.
- SLUB_DEBUG in most cases are not able to detect bad reads,
KASan able to detect both reads and writes.
- In some cases (e.g. redzone overwritten) SLUB_DEBUG detect
bugs only on allocation/freeing of object. KASan catch
bugs right before it will happen, so we always know exact
place of first bad read/write.
[1] https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel
[2] https://code.google.com/p/address-sanitizer/wiki/FoundBugs
[3] https://code.google.com/p/thread-sanitizer/wiki/FoundBugs
[4] https://code.google.com/p/memory-sanitizer/wiki/FoundBugs
[5] https://code.google.com/p/address-sanitizer/wiki/AddressSanitizerForKernel#Trophies
Based on work by Andrey Konovalov.
Signed-off-by: Andrey Ryabinin <a.ryabinin@samsung.com>
Acked-by: Michal Marek <mmarek@suse.cz>
Signed-off-by: Andrey Konovalov <adech.fo@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Konstantin Serebryany <kcc@google.com>
Cc: Dmitry Chernenkov <dmitryc@google.com>
Cc: Yuri Gribov <tetra2005@gmail.com>
Cc: Konstantin Khlebnikov <koct9i@gmail.com>
Cc: Sasha Levin <sasha.levin@oracle.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Andi Kleen <andi@firstfloor.org>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2015-02-14 05:39:17 +07:00
e n d i f
2016-01-21 06:00:55 +07:00
i f e q ( $( CONFIG_UBSAN ) , y )
_c_flags += $( if $( patsubst n%,, \
$( UBSAN_SANITIZE_$( basetarget) .o) $( UBSAN_SANITIZE) $( CONFIG_UBSAN_SANITIZE_ALL) ) , \
$( CFLAGS_UBSAN) )
e n d i f
kernel: add kcov code coverage
kcov provides code coverage collection for coverage-guided fuzzing
(randomized testing). Coverage-guided fuzzing is a testing technique
that uses coverage feedback to determine new interesting inputs to a
system. A notable user-space example is AFL
(http://lcamtuf.coredump.cx/afl/). However, this technique is not
widely used for kernel testing due to missing compiler and kernel
support.
kcov does not aim to collect as much coverage as possible. It aims to
collect more or less stable coverage that is function of syscall inputs.
To achieve this goal it does not collect coverage in soft/hard
interrupts and instrumentation of some inherently non-deterministic or
non-interesting parts of kernel is disbled (e.g. scheduler, locking).
Currently there is a single coverage collection mode (tracing), but the
API anticipates additional collection modes. Initially I also
implemented a second mode which exposes coverage in a fixed-size hash
table of counters (what Quentin used in his original patch). I've
dropped the second mode for simplicity.
This patch adds the necessary support on kernel side. The complimentary
compiler support was added in gcc revision 231296.
We've used this support to build syzkaller system call fuzzer, which has
found 90 kernel bugs in just 2 months:
https://github.com/google/syzkaller/wiki/Found-Bugs
We've also found 30+ bugs in our internal systems with syzkaller.
Another (yet unexplored) direction where kcov coverage would greatly
help is more traditional "blob mutation". For example, mounting a
random blob as a filesystem, or receiving a random blob over wire.
Why not gcov. Typical fuzzing loop looks as follows: (1) reset
coverage, (2) execute a bit of code, (3) collect coverage, repeat. A
typical coverage can be just a dozen of basic blocks (e.g. an invalid
input). In such context gcov becomes prohibitively expensive as
reset/collect coverage steps depend on total number of basic
blocks/edges in program (in case of kernel it is about 2M). Cost of
kcov depends only on number of executed basic blocks/edges. On top of
that, kernel requires per-thread coverage because there are always
background threads and unrelated processes that also produce coverage.
With inlined gcov instrumentation per-thread coverage is not possible.
kcov exposes kernel PCs and control flow to user-space which is
insecure. But debugfs should not be mapped as user accessible.
Based on a patch by Quentin Casasnovas.
[akpm@linux-foundation.org: make task_struct.kcov_mode have type `enum kcov_mode']
[akpm@linux-foundation.org: unbreak allmodconfig]
[akpm@linux-foundation.org: follow x86 Makefile layout standards]
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Cc: syzkaller <syzkaller@googlegroups.com>
Cc: Vegard Nossum <vegard.nossum@oracle.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Tavis Ormandy <taviso@google.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Quentin Casasnovas <quentin.casasnovas@oracle.com>
Cc: Kostya Serebryany <kcc@google.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Kees Cook <keescook@google.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Sasha Levin <sasha.levin@oracle.com>
Cc: David Drysdale <drysdale@google.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Kirill A. Shutemov <kirill@shutemov.name>
Cc: Jiri Slaby <jslaby@suse.cz>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-03-23 04:27:30 +07:00
i f e q ( $( CONFIG_KCOV ) , y )
_c_flags += $( if $( patsubst n%,, \
2016-08-03 04:07:30 +07:00
$( KCOV_INSTRUMENT_$( basetarget) .o) $( KCOV_INSTRUMENT) $( CONFIG_KCOV_INSTRUMENT_ALL) ) , \
kernel: add kcov code coverage
kcov provides code coverage collection for coverage-guided fuzzing
(randomized testing). Coverage-guided fuzzing is a testing technique
that uses coverage feedback to determine new interesting inputs to a
system. A notable user-space example is AFL
(http://lcamtuf.coredump.cx/afl/). However, this technique is not
widely used for kernel testing due to missing compiler and kernel
support.
kcov does not aim to collect as much coverage as possible. It aims to
collect more or less stable coverage that is function of syscall inputs.
To achieve this goal it does not collect coverage in soft/hard
interrupts and instrumentation of some inherently non-deterministic or
non-interesting parts of kernel is disbled (e.g. scheduler, locking).
Currently there is a single coverage collection mode (tracing), but the
API anticipates additional collection modes. Initially I also
implemented a second mode which exposes coverage in a fixed-size hash
table of counters (what Quentin used in his original patch). I've
dropped the second mode for simplicity.
This patch adds the necessary support on kernel side. The complimentary
compiler support was added in gcc revision 231296.
We've used this support to build syzkaller system call fuzzer, which has
found 90 kernel bugs in just 2 months:
https://github.com/google/syzkaller/wiki/Found-Bugs
We've also found 30+ bugs in our internal systems with syzkaller.
Another (yet unexplored) direction where kcov coverage would greatly
help is more traditional "blob mutation". For example, mounting a
random blob as a filesystem, or receiving a random blob over wire.
Why not gcov. Typical fuzzing loop looks as follows: (1) reset
coverage, (2) execute a bit of code, (3) collect coverage, repeat. A
typical coverage can be just a dozen of basic blocks (e.g. an invalid
input). In such context gcov becomes prohibitively expensive as
reset/collect coverage steps depend on total number of basic
blocks/edges in program (in case of kernel it is about 2M). Cost of
kcov depends only on number of executed basic blocks/edges. On top of
that, kernel requires per-thread coverage because there are always
background threads and unrelated processes that also produce coverage.
With inlined gcov instrumentation per-thread coverage is not possible.
kcov exposes kernel PCs and control flow to user-space which is
insecure. But debugfs should not be mapped as user accessible.
Based on a patch by Quentin Casasnovas.
[akpm@linux-foundation.org: make task_struct.kcov_mode have type `enum kcov_mode']
[akpm@linux-foundation.org: unbreak allmodconfig]
[akpm@linux-foundation.org: follow x86 Makefile layout standards]
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Cc: syzkaller <syzkaller@googlegroups.com>
Cc: Vegard Nossum <vegard.nossum@oracle.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Tavis Ormandy <taviso@google.com>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Quentin Casasnovas <quentin.casasnovas@oracle.com>
Cc: Kostya Serebryany <kcc@google.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Kees Cook <keescook@google.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Sasha Levin <sasha.levin@oracle.com>
Cc: David Drysdale <drysdale@google.com>
Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Kirill A. Shutemov <kirill@shutemov.name>
Cc: Jiri Slaby <jslaby@suse.cz>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-03-23 04:27:30 +07:00
$( CFLAGS_KCOV) )
e n d i f
2019-05-13 13:22:17 +07:00
# $(srctree)/$(src) for including checkin headers from generated source files
# $(objtree)/$(obj) for including generated headers from checkin source files
2019-01-16 09:56:40 +07:00
i f e q ( $( KBUILD_EXTMOD ) , )
2019-07-06 10:07:12 +07:00
i f d e f b u i l d i n g _ o u t _ o f _ s r c t r e e
2019-05-13 13:22:17 +07:00
_c_flags += -I $( srctree) /$( src) -I $( objtree) /$( obj)
_a_flags += -I $( srctree) /$( src) -I $( objtree) /$( obj)
_cpp_flags += -I $( srctree) /$( src) -I $( objtree) /$( obj)
2005-04-17 05:20:36 +07:00
e n d i f
2019-01-16 09:56:40 +07:00
e n d i f
2005-04-17 05:20:36 +07:00
2019-08-19 15:58:43 +07:00
part-of-module = $( if $( filter $( basename $@ ) .o, $( real-obj-m) ) ,y)
quiet_modtag = $( if $( part-of-module) ,[ M] , )
modkern_cflags = \
$( if $( part-of-module) , \
$( KBUILD_CFLAGS_MODULE) $( CFLAGS_MODULE) , \
$( KBUILD_CFLAGS_KERNEL) $( CFLAGS_KERNEL) )
modkern_aflags = $( if $( part-of-module) , \
$( KBUILD_AFLAGS_MODULE) $( AFLAGS_MODULE) , \
$( KBUILD_AFLAGS_KERNEL) $( AFLAGS_KERNEL) )
2008-11-22 03:50:02 +07:00
c_flags = -Wp,-MD,$( depfile) $( NOSTDINC_FLAGS) $( LINUXINCLUDE) \
2018-03-27 19:52:50 +07:00
-include $( srctree) /include/linux/compiler_types.h \
2019-05-13 13:22:17 +07:00
$( _c_flags) $( modkern_cflags) \
2016-03-17 22:32:14 +07:00
$( basename_flags) $( modname_flags)
2005-04-17 05:20:36 +07:00
2008-11-22 03:50:02 +07:00
a_flags = -Wp,-MD,$( depfile) $( NOSTDINC_FLAGS) $( LINUXINCLUDE) \
2019-05-13 13:22:17 +07:00
$( _a_flags) $( modkern_aflags)
2005-04-17 05:20:36 +07:00
2008-11-22 03:50:02 +07:00
cpp_flags = -Wp,-MD,$( depfile) $( NOSTDINC_FLAGS) $( LINUXINCLUDE) \
2019-05-13 13:22:17 +07:00
$( _cpp_flags)
2005-04-17 05:20:36 +07:00
2018-08-24 06:20:39 +07:00
ld_flags = $( KBUILD_LDFLAGS) $( ldflags-y) $( LDFLAGS_$( @F) )
2005-04-17 05:20:36 +07:00
2017-07-25 09:23:36 +07:00
DTC_INCLUDE := $( srctree) /scripts/dtc/include-prefixes
2013-05-31 17:14:20 +07:00
dtc_cpp_flags = -Wp,-MD,$( depfile) .pre.tmp -nostdinc \
2017-07-25 09:23:36 +07:00
$( addprefix -I,$( DTC_INCLUDE) ) \
2013-02-13 05:03:37 +07:00
-undef -D__DTS__
2014-08-19 14:34:20 +07:00
# Useful for describing the dependency of composite objects
# Usage:
# $(call multi_depend, multi_used_targets, suffix_to_remove, suffix_to_add)
d e f i n e m u l t i _ d e p e n d
$( foreach m , $ ( notdir $ 1) , \
$( eval $( obj) /$m : \
$( addprefix $( obj) /, $( foreach s, $3 , $( $( m:%$( strip $2 ) = %$( s) ) ) ) ) ) )
e n d e f
2005-04-17 05:20:36 +07:00
# Shipped files
# ===========================================================================
quiet_cmd_shipped = SHIPPED $@
cmd_shipped = cat $< > $@
2011-06-08 00:09:28 +07:00
$(obj)/% : $( src ) /%_shipped
2005-04-17 05:20:36 +07:00
$( call cmd,shipped)
# Commands useful for building a boot image
# ===========================================================================
2014-04-28 14:26:18 +07:00
#
2005-04-17 05:20:36 +07:00
# Use as following:
#
# target: source(s) FORCE
# $(if_changed,ld/objcopy/gzip)
#
2007-10-01 01:34:36 +07:00
# and add target to extra-y so that we know we have to
2005-04-17 05:20:36 +07:00
# read in the saved command line
# Linking
# ---------------------------------------------------------------------------
quiet_cmd_ld = LD $@
2019-01-17 17:02:43 +07:00
cmd_ld = $( LD) $( ld_flags) $( real-prereqs) -o $@
2005-04-17 05:20:36 +07:00
2019-02-05 14:33:37 +07:00
# Archive
# ---------------------------------------------------------------------------
quiet_cmd_ar = AR $@
2019-09-21 13:49:54 +07:00
cmd_ar = rm -f $@ ; $( AR) cDPrsT $@ $( real-prereqs)
2019-02-05 14:33:37 +07:00
2005-04-17 05:20:36 +07:00
# Objcopy
# ---------------------------------------------------------------------------
quiet_cmd_objcopy = OBJCOPY $@
cmd_objcopy = $( OBJCOPY) $( OBJCOPYFLAGS) $( OBJCOPYFLAGS_$( @F) ) $< $@
# Gzip
# ---------------------------------------------------------------------------
quiet_cmd_gzip = GZIP $@
2019-01-17 17:02:43 +07:00
cmd_gzip = cat $( real-prereqs) | gzip -n -f -9 > $@
2005-04-17 05:20:36 +07:00
2010-12-23 02:57:26 +07:00
# DTC
# ---------------------------------------------------------------------------
2016-02-12 05:28:13 +07:00
DTC ?= $( objtree) /scripts/dtc/dtc
2010-12-23 02:57:26 +07:00
2016-03-24 22:52:42 +07:00
# Disable noisy checks by default
2019-08-31 23:25:55 +07:00
i f e q ( $( findstring 1,$ ( KBUILD_EXTRA_WARN ) ) , )
2017-03-21 21:14:20 +07:00
DTC_FLAGS += -Wno-unit_address_vs_reg \
-Wno-unit_address_format \
2018-02-28 21:09:45 +07:00
-Wno-avoid_unnecessary_addr_size \
-Wno-alias_paths \
2018-05-08 04:46:47 +07:00
-Wno-graph_child_address \
2018-11-29 08:11:50 +07:00
-Wno-simple_bus_reg \
2018-05-08 04:46:47 +07:00
-Wno-unique_unit_address \
2017-03-21 21:14:20 +07:00
-Wno-pci_device_reg
2016-03-24 22:52:42 +07:00
e n d i f
2019-08-31 23:25:55 +07:00
i f n e q ( $( findstring 2,$ ( KBUILD_EXTRA_WARN ) ) , )
2017-03-21 21:14:20 +07:00
DTC_FLAGS += -Wnode_name_chars_strict \
-Wproperty_name_chars_strict
2016-03-24 22:52:42 +07:00
e n d i f
2017-04-26 07:09:53 +07:00
DTC_FLAGS += $( DTC_FLAGS_$( basetarget) )
2010-12-23 02:57:26 +07:00
# Generate an assembly file to wrap the output of the device tree compiler
2013-06-13 17:53:09 +07:00
quiet_cmd_dt_S_dtb = DTB $@
2010-12-23 02:57:26 +07:00
cmd_dt_S_dtb = \
2019-01-17 17:02:44 +07:00
{ \
2010-12-23 02:57:26 +07:00
echo '\#include <asm-generic/vmlinux.lds.h>' ; \
echo '.section .dtb.init.rodata,"a"' ; \
echo '.balign STRUCT_ALIGNMENT' ; \
kbuild: Handle builtin dtb file names containing hyphens
cmd_dt_S_dtb constructs the assembly source to incorporate a devicetree
FDT (that is, the .dtb file) as binary data in the kernel image. This
assembly source contains labels before and after the binary data. The
label names incorporate the file name of the corresponding .dtb file.
Hyphens are not legal characters in labels, so .dtb files built into the
kernel with hyphens in the file name result in errors like the
following:
bcm3368-netgear-cvg834g.dtb.S: Assembler messages:
bcm3368-netgear-cvg834g.dtb.S:5: Error: : no such section
bcm3368-netgear-cvg834g.dtb.S:5: Error: junk at end of line, first unrecognized character is `-'
bcm3368-netgear-cvg834g.dtb.S:6: Error: unrecognized opcode `__dtb_bcm3368-netgear-cvg834g_begin:'
bcm3368-netgear-cvg834g.dtb.S:8: Error: unrecognized opcode `__dtb_bcm3368-netgear-cvg834g_end:'
bcm3368-netgear-cvg834g.dtb.S:9: Error: : no such section
bcm3368-netgear-cvg834g.dtb.S:9: Error: junk at end of line, first unrecognized character is `-'
Fix this by updating cmd_dt_S_dtb to transform all hyphens from the file
name to underscores when constructing the labels.
As of v4.16-rc2, 1139 .dts files across ARM64, ARM, MIPS and PowerPC
contain hyphens in their names, but the issue only currently manifests
on Broadcom MIPS platforms, as that is the only place where such files
are built into the kernel. For example when CONFIG_DT_NETGEAR_CVG834G=y,
or on BMIPS kernels when the dtbs target is used (in the latter case it
admittedly shouldn't really build all the dtb.o files, but thats a
separate issue).
Fixes: 695835511f96 ("MIPS: BMIPS: rename bcm96358nb4ser to bcm6358-neufbox4-sercom")
Signed-off-by: James Hogan <jhogan@kernel.org>
Reviewed-by: Frank Rowand <frowand.list@gmail.com>
Cc: Rob Herring <robh+dt@kernel.org>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: <stable@vger.kernel.org> # 4.9+
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-03-08 18:02:46 +07:00
echo '.global __dtb_$(subst -,_,$(*F))_begin' ; \
echo '__dtb_$(subst -,_,$(*F))_begin:' ; \
2010-12-23 02:57:26 +07:00
echo '.incbin "$<" ' ; \
kbuild: Handle builtin dtb file names containing hyphens
cmd_dt_S_dtb constructs the assembly source to incorporate a devicetree
FDT (that is, the .dtb file) as binary data in the kernel image. This
assembly source contains labels before and after the binary data. The
label names incorporate the file name of the corresponding .dtb file.
Hyphens are not legal characters in labels, so .dtb files built into the
kernel with hyphens in the file name result in errors like the
following:
bcm3368-netgear-cvg834g.dtb.S: Assembler messages:
bcm3368-netgear-cvg834g.dtb.S:5: Error: : no such section
bcm3368-netgear-cvg834g.dtb.S:5: Error: junk at end of line, first unrecognized character is `-'
bcm3368-netgear-cvg834g.dtb.S:6: Error: unrecognized opcode `__dtb_bcm3368-netgear-cvg834g_begin:'
bcm3368-netgear-cvg834g.dtb.S:8: Error: unrecognized opcode `__dtb_bcm3368-netgear-cvg834g_end:'
bcm3368-netgear-cvg834g.dtb.S:9: Error: : no such section
bcm3368-netgear-cvg834g.dtb.S:9: Error: junk at end of line, first unrecognized character is `-'
Fix this by updating cmd_dt_S_dtb to transform all hyphens from the file
name to underscores when constructing the labels.
As of v4.16-rc2, 1139 .dts files across ARM64, ARM, MIPS and PowerPC
contain hyphens in their names, but the issue only currently manifests
on Broadcom MIPS platforms, as that is the only place where such files
are built into the kernel. For example when CONFIG_DT_NETGEAR_CVG834G=y,
or on BMIPS kernels when the dtbs target is used (in the latter case it
admittedly shouldn't really build all the dtb.o files, but thats a
separate issue).
Fixes: 695835511f96 ("MIPS: BMIPS: rename bcm96358nb4ser to bcm6358-neufbox4-sercom")
Signed-off-by: James Hogan <jhogan@kernel.org>
Reviewed-by: Frank Rowand <frowand.list@gmail.com>
Cc: Rob Herring <robh+dt@kernel.org>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: Kevin Cernekee <cernekee@gmail.com>
Cc: <stable@vger.kernel.org> # 4.9+
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-03-08 18:02:46 +07:00
echo '__dtb_$(subst -,_,$(*F))_end:' ; \
echo '.global __dtb_$(subst -,_,$(*F))_end' ; \
2010-12-23 02:57:26 +07:00
echo '.balign STRUCT_ALIGNMENT' ; \
2019-01-17 17:02:44 +07:00
} > $@
2010-12-23 02:57:26 +07:00
2018-03-23 20:04:34 +07:00
$(obj)/%.dtb.S : $( obj ) /%.dtb FORCE
$( call if_changed,dt_S_dtb)
2010-12-23 02:57:26 +07:00
quiet_cmd_dtc = DTC $@
2015-03-30 19:39:08 +07:00
cmd_dtc = mkdir -p $( dir ${ dtc -tmp } ) ; \
2018-01-11 04:19:37 +07:00
$( HOSTCC) -E $( dtc_cpp_flags) -x assembler-with-cpp -o $( dtc-tmp) $< ; \
2018-09-07 01:26:07 +07:00
$( DTC) -O $( 2) -o $@ -b 0 \
2017-07-25 09:23:36 +07:00
$( addprefix -i,$( dir $<) $( DTC_INCLUDE) ) $( DTC_FLAGS) \
2013-05-31 17:14:20 +07:00
-d $( depfile) .dtc.tmp $( dtc-tmp) ; \
cat $( depfile) .pre.tmp $( depfile) .dtc.tmp > $( depfile)
2005-04-17 05:20:36 +07:00
2018-02-28 06:49:57 +07:00
$(obj)/%.dtb : $( src ) /%.dts $( DTC ) FORCE
2018-09-07 01:26:07 +07:00
$( call if_changed_dep,dtc,dtb)
DT_CHECKER ?= dt-validate
DT_BINDING_DIR := Documentation/devicetree/bindings
DT_TMP_SCHEMA := $( objtree) /$( DT_BINDING_DIR) /processed-schema.yaml
quiet_cmd_dtb_check = CHECK $@
2019-05-09 20:53:44 +07:00
cmd_dtb_check = $( DT_CHECKER) -u $( srctree) /$( DT_BINDING_DIR) -p $( DT_TMP_SCHEMA) $@ ;
2018-09-07 01:26:07 +07:00
d e f i n e r u l e _ d t c _ d t _ y a m l
2018-12-31 11:09:00 +07:00
$( call cmd_and_fixdep,dtc,yaml)
$( call cmd,dtb_check)
2018-09-07 01:26:07 +07:00
e n d e f
$(obj)/%.dt.yaml : $( src ) /%.dts $( DTC ) $( DT_TMP_SCHEMA ) FORCE
$( call if_changed_rule,dtc_dt_yaml)
2012-11-28 06:29:10 +07:00
2013-05-31 17:14:20 +07:00
dtc-tmp = $( subst $( comma) ,_,$( dot-target) .dts.tmp)
2013-02-06 02:06:28 +07:00
2009-01-05 04:46:16 +07:00
# Bzip2
# ---------------------------------------------------------------------------
2009-05-06 11:17:15 +07:00
# Bzip2 and LZMA do not include size in file... so we have to fake that;
# append the size as a 32-bit littleendian number as gzip does.
2009-10-14 03:22:46 +07:00
size_append = printf $( shell \
2009-05-06 11:17:15 +07:00
dec_size = 0; \
2019-01-17 17:02:43 +07:00
f o r F i n $( real -prereqs ) ; d o \
kbuild: Use ls(1) instead of stat(1) to obtain file size
stat(1) is not standardized and different implementations have their own
(conflicting) flags for querying the size of a file.
ls(1) provides the same information (value of st.st_size) in the 5th
column, except when the file is a character or block device. This output
is standardized[0]. The -n option turns on -l, which writes lines
formatted like
"%s %u %s %s %u %s %s\n", <file mode>, <number of links>,
<owner name>, <group name>, <size>, <date and time>,
<pathname>
but instead of writing the <owner name> and <group name>, it writes the
numeric owner and group IDs (this avoids /etc/passwd and /etc/group
lookups as well as potential field splitting issues).
The <size> field is specified as "the value that would be returned for
the file in the st_size field of struct stat".
To avoid duplicating logic in several locations in the tree, create
scripts/file-size.sh and update callers to use that instead of stat(1).
[0] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ls.html#tag_20_73_10
Signed-off-by: Michael Forney <forney@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2018-03-19 07:54:02 +07:00
fsize = $$ ( $( CONFIG_SHELL) $( srctree) /scripts/file-size.sh $$ F) ; \
2009-05-06 11:17:15 +07:00
dec_size = $$ ( expr $$ dec_size + $$ fsize) ; \
d o n e ; \
2009-12-29 02:38:27 +07:00
p r i n t f "%08x\n" $$ d e c _ s i z e | \
sed 's/\(..\)/\1 /g' | { \
read ch0 ch1 ch2 ch3; \
for ch in $$ ch3 $$ ch2 $$ ch1 $$ ch0; do \
printf '%s%03o' '\\' $$ ( ( 0x$$ ch) ) ; \
done ; \
} \
2009-05-06 11:17:15 +07:00
)
quiet_cmd_bzip2 = BZIP2 $@
2019-07-09 17:50:50 +07:00
cmd_bzip2 = { cat $( real-prereqs) | bzip2 -9; $( size_append) ; } > $@
2009-01-05 04:46:16 +07:00
# Lzma
# ---------------------------------------------------------------------------
quiet_cmd_lzma = LZMA $@
2019-07-09 17:50:50 +07:00
cmd_lzma = { cat $( real-prereqs) | lzma -9; $( size_append) ; } > $@
lib: add support for LZO-compressed kernels
This patch series adds generic support for creating and extracting
LZO-compressed kernel images, as well as support for using such images on
the x86 and ARM architectures, and support for creating and using
LZO-compressed initrd and initramfs images.
Russell King said:
: Testing on a Cortex A9 model:
: - lzo decompressor is 65% of the time gzip takes to decompress a kernel
: - lzo kernel is 9% larger than a gzip kernel
:
: which I'm happy to say confirms your figures when comparing the two.
:
: However, when comparing your new gzip code to the old gzip code:
: - new is 99% of the size of the old code
: - new takes 42% of the time to decompress than the old code
:
: What this means is that for a proper comparison, the results get even better:
: - lzo is 7.5% larger than the old gzip'd kernel image
: - lzo takes 28% of the time that the old gzip code took
:
: So the expense seems definitely worth the effort. The only reason I
: can think of ever using gzip would be if you needed the additional
: compression (eg, because you have limited flash to store the image.)
:
: I would argue that the default for ARM should therefore be LZO.
This patch:
The lzo compressor is worse than gzip at compression, but faster at
extraction. Here are some figures for an ARM board I'm working on:
Uncompressed size: 3.24Mo
gzip 1.61Mo 0.72s
lzo 1.75Mo 0.48s
So for a compression ratio that is still relatively close to gzip, it's
much faster to extract, at least in that case.
This part contains:
- Makefile routine to support lzo compression
- Fixes to the existing lzo compressor so that it can be used in
compressed kernels
- wrapper around the existing lzo1x_decompress, as it only extracts one
block at a time, while we need to extract a whole file here
- config dialog for kernel compression
[akpm@linux-foundation.org: coding-style fixes]
[akpm@linux-foundation.org: cleanup]
Signed-off-by: Albin Tonnerre <albin.tonnerre@free-electrons.com>
Tested-by: Wu Zhangjin <wuzhangjin@gmail.com>
Acked-by: "H. Peter Anvin" <hpa@zytor.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Russell King <rmk@arm.linux.org.uk>
Acked-by: Russell King <rmk@arm.linux.org.uk>
Cc: Ralf Baechle <ralf@linux-mips.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2010-01-09 05:42:42 +07:00
2010-03-11 16:42:14 +07:00
quiet_cmd_lzo = LZO $@
2019-07-09 17:50:50 +07:00
cmd_lzo = { cat $( real-prereqs) | lzop -9; $( size_append) ; } > $@
2010-03-05 23:34:46 +07:00
2013-07-09 06:01:46 +07:00
quiet_cmd_lz4 = LZ4 $@
2019-07-09 17:50:50 +07:00
cmd_lz4 = { cat $( real-prereqs) | lz4c -l -c1 stdin stdout; \
2019-01-17 17:02:44 +07:00
$( size_append) ; } > $@
2013-07-09 06:01:46 +07:00
2012-03-17 04:03:55 +07:00
# U-Boot mkimage
# ---------------------------------------------------------------------------
MKIMAGE := $( srctree) /scripts/mkuboot.sh
# SRCARCH just happens to match slightly more than ARCH (on sparc), so reduces
# the number of overrides in arch makefiles
UIMAGE_ARCH ?= $( SRCARCH)
UIMAGE_COMPRESSION ?= $( if $( 2) ,$( 2) ,none)
UIMAGE_OPTS-y ?=
UIMAGE_TYPE ?= kernel
UIMAGE_LOADADDR ?= arch_must_set_this
UIMAGE_ENTRYADDR ?= $( UIMAGE_LOADADDR)
UIMAGE_NAME ?= 'Linux-$(KERNELRELEASE)'
2018-12-31 08:05:01 +07:00
quiet_cmd_uimage = UIMAGE $@
2019-08-25 20:28:37 +07:00
cmd_uimage = $( BASH) $( MKIMAGE) -A $( UIMAGE_ARCH) -O linux \
2012-03-17 04:03:55 +07:00
-C $( UIMAGE_COMPRESSION) $( UIMAGE_OPTS-y) \
-T $( UIMAGE_TYPE) \
-a $( UIMAGE_LOADADDR) -e $( UIMAGE_ENTRYADDR) \
2018-12-31 08:05:01 +07:00
-n $( UIMAGE_NAME) -d $< $@
2012-03-17 04:03:55 +07:00
2011-01-13 08:01:22 +07:00
# XZ
# ---------------------------------------------------------------------------
# Use xzkern to compress the kernel image and xzmisc to compress other things.
#
# xzkern uses a big LZMA2 dictionary since it doesn't increase memory usage
# of the kernel decompressor. A BCJ filter is used if it is available for
# the target architecture. xzkern also appends uncompressed size of the data
# using size_append. The .xz format has the size information available at
# the end of the file too, but it's in more complex format and it's good to
# avoid changing the part of the boot code that reads the uncompressed size.
# Note that the bytes added by size_append will make the xz tool think that
# the file is corrupt. This is expected.
#
# xzmisc doesn't use size_append, so it can be used to create normal .xz
# files. xzmisc uses smaller LZMA2 dictionary than xzkern, because a very
# big dictionary would increase the memory usage too much in the multi-call
# decompression mode. A BCJ filter isn't used either.
quiet_cmd_xzkern = XZKERN $@
2019-07-09 17:50:50 +07:00
cmd_xzkern = { cat $( real-prereqs) | sh $( srctree) /scripts/xz_wrap.sh; \
2019-01-17 17:02:44 +07:00
$( size_append) ; } > $@
2011-01-13 08:01:22 +07:00
quiet_cmd_xzmisc = XZMISC $@
2019-01-17 17:02:44 +07:00
cmd_xzmisc = cat $( real-prereqs) | xz --check= crc32 --lzma2= dict = 1MiB > $@
2017-04-13 02:43:52 +07:00
# ASM offsets
# ---------------------------------------------------------------------------
# Default sed regexp - multiline due to syntax constraints
2017-04-21 13:21:11 +07:00
#
# Use [:space:] because LLVM's integrated assembler inserts <tab> around
# the .ascii directive whereas GCC keeps the <space> as-is.
2017-04-13 02:43:52 +07:00
d e f i n e s e d - o f f s e t s
2017-04-21 13:21:11 +07:00
' s:^[ [ :space:] ] *\. ascii[ [ :space:] ] *"\(.*\)" .*:\1 :; \
/^->/{ s:->#\( .*\) :/* \1 */:; \
2017-04-13 02:43:52 +07:00
s:^->\( [ ^ ] *\) [ \$ $# ] *\( [ ^ ] *\) \( .*\) :#define \1 \2 /* \3 */:; \
2017-04-21 13:21:11 +07:00
s:->::; p; } '
2017-04-13 02:43:52 +07:00
e n d e f
# Use filechk to avoid rebuilds when a header changes, but the resulting file
# does not
d e f i n e f i l e c h k _ o f f s e t s
echo " #ifndef $2 " ; \
echo " #define $2 " ; \
echo "/*" ; \
echo " * DO NOT MODIFY." ; \
echo " *" ; \
echo " * This file was generated by Kbuild" ; \
echo " */" ; \
echo "" ; \
2018-07-25 12:16:11 +07:00
sed -ne $( sed-offsets) < $<; \
2017-04-13 02:43:52 +07:00
echo "" ; \
2018-12-31 15:24:09 +07:00
echo "#endif"
2017-04-13 02:43:52 +07:00
e n d e f
