Refine RUN_TESTS define's output block for summary and non-summary code
to remove duplicate code and make it readable.
cd `dirname $$TEST` > /dev/null; and cd - > /dev/null; are moved
to common code block and indentation fixed.
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Within run_tests target, the whole script needs to be executed within
the same shell and not as separate subshells, so the initial test_num
variable set to 0 is still present when executing "test_num=`echo
$$test_num+1 | bc`;".
Demonstration of the issue (make run_tests):
TAP version 13
(standard_in) 1: syntax error
selftests: basic_test
========================================
ok 1.. selftests: basic_test [PASS]
(standard_in) 1: syntax error
selftests: basic_percpu_ops_test
========================================
ok 1.. selftests: basic_percpu_ops_test [PASS]
(standard_in) 1: syntax error
selftests: param_test
========================================
ok 1.. selftests: param_test [PASS]
With fix applied:
TAP version 13
selftests: basic_test
========================================
ok 1..1 selftests: basic_test [PASS]
selftests: basic_percpu_ops_test
========================================
ok 1..2 selftests: basic_percpu_ops_test [PASS]
selftests: param_test
========================================
ok 1..3 selftests: param_test [PASS]
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Fixes: 1f87c7c15d ("selftests: lib.mk: change RUN_TESTS to print messages in TAP13 format")
CC: Shuah Khan <shuahkh@osg.samsung.com>
CC: linux-kselftest@vger.kernel.org
Signed-off-by: Shuah Khan (Samsung OSG) <shuah@kernel.org>
Commit fbcab13d2e ("selftests: silence test output by default")
changed the run_tests logic as well as the logic to generate
run_kselftests.sh to redirect test output away from the console.
As discussed on the list and at kernel summit, this is not a desirable
default as it means in order to debug a failure the console output is
not sufficient, you also need access to the test machine to get the
full test logs. Additionally it's impolite to write directly to
/tmp/$TEST_NAME on shared systems.
The change to the run_tests logic was reverted in commit
a323335e62 ("selftests: lib.mk: print individual test results to
console by default"), and instead a summary option was added so that
quiet output could be requested.
However the change to run_kselftests.sh was left as-is.
This commit applies the same logic to the run_kselftests.sh code, ie.
the script now takes a "--summary" option which suppresses the output,
but shows all output by default.
Additionally instead of writing to /tmp/$TEST_NAME the output is
redirected to the directory where the generated test script is
located.
Fixes: fbcab13d2e ("selftests: silence test output by default")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Change run_tests to print individual test results to console by default.
Introduce "summary" option to print individual test results to a file
/tmp/test_name and just print the summary to the console.
This change is necessary to support use-cases where test machines get
rebooted once tests are run and the console log should contain the full
results.
In the following example, individual test results with "summary=1" option
are written to /tmp/kcmp_test
make --silent TARGETS=kcmp kselftest
TAP version 13
selftests: kcmp_test
========================================
pid1: 30126 pid2: 30127 FD: 2 FILES: 2 VM: 1 FS: 2 SIGHAND: 2 IO:
0 SYSVSEM: 0 INV: -1
PASS: 0 returned as expected
PASS: 0 returned as expected
FAIL: 0 expected but -1 returned (Invalid argument)
Pass 2 Fail 1 Xfail 0 Xpass 0 Skip 0 Error 0
1..3
Bail out!
Pass 2 Fail 1 Xfail 0 Xpass 0 Skip 0 Error 0
1..3
Pass 0 Fail 0 Xfail 0 Xpass 0 Skip 0 Error 0
1..0
ok 1..1 selftests: kcmp_test [PASS]
make --silent TARGETS=kcmp summary=1 kselftest
TAP version 13
selftests: kcmp_test
========================================
ok 1..1 selftests: kcmp_test [PASS]
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
For make O=dir run_tests to work, test scripts, test files, and other
dependencies need to be copied over to the object directory. Running
tests from the object directory is necessary to avoid making the source
tree dirty.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Some tests such as sync can't use generic build rules in lib.mk and require
custom rules. Currently there is no provision to allow custom builds and
test such as sync use TEST_PROGS which is reserved for test shell scripts.
Add TEST_CUSTOM_PROGS variable to lib.mk to run and install custom tests
built by individual test make files.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Fix test executable status check to use full path for make O=dir case,m
when tests are relocated to user specified object directory. Without the
full path, this check fails to find the file and fails the test.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
kselftest and kselftest-clean targets fail when object directory is
specified to relocate objects. Main Makefile make O= path clears the
built-in defines LINK.c, COMPILE.S, LINK.S, and RM that are used in
lib.mk to build and clean targets. Define them.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Some of the networking tests are very noisy and make it impossible to
see if we actually passed the tests as they run. Default to suppressing
the output from any tests run in order to make it easier to track what
failed.
Signed-off-by: Josef Bacik <jbacik@fb.com>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Change common RUN_TESTS to print messages in user friendly TAP13 format.
This change add TAP13 header at the start of RUN_TESTS target run, and
prints the resulting pass/fail messages with test number information in
the TAP 13 format for each test in the run tests list.
This change covers test scripts as well as test programs. Test programs
have an option to use ksft_ API, however test scripts won't be able to.
With this change, test scripts can print TAP13 format output without any
changes to individual scripts.
Test programs can provide TAP13 format output as needed as some tests
already do. Tests that haven't been converted will benefit from this
change. Tests that are converted benefit from the test counts for all
the tests in each test directory.
Running firmware tests:
make --silent -C tools/testing/selftests/firmware/ run_tests
Before the change:
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
./fw_filesystem.sh: /sys/devices/virtual/misc/test_firmware not present
You must have the following enabled in your kernel:
CONFIG_TEST_FIRMWARE=y
selftests: fw_filesystem.sh [FAIL]
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
selftests: fw_fallback.sh [FAIL]
After the change:
TAP version 13
selftests: fw_filesystem.sh
========================================
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
./fw_filesystem.sh: /sys/devices/virtual/misc/test_firmware not present
You must have the following enabled in your kernel:
CONFIG_TEST_FIRMWARE=y
not ok 1..1 selftests: fw_filesystem.sh [FAIL]
selftests: fw_fallback.sh
========================================
modprobe: ERROR: could not insert 'test_firmware': Operation not
permitted
not ok 1..2 selftests: fw_fallback.sh [FAIL]
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Change lib.mk RUN_TESTS to take test list as an argument. This will
allow it to be called from individual test makefiles to run additional
tests that aren't suitable for a default kselftest run. As an example,
timers test includes destructive tests that aren't included in the
common run_tests target.
Change times/Makefile to use RUN_TESTS call with destructive test list
as an argument instead of using its own RUN_TESTS target.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Suppress "cd" output from run_tests while running tests to declutter the
test results.
Running efivarfs test:
make --silent -C tools/testing/selftests/efivarfs/ run_tests
Before the change:
skip all tests: must be run as root
selftests: efivarfs.sh [PASS]
/lkml/linux-kselftest/tools/testing/selftests/efivarfs
After the change:
skip all tests: must be run as root
selftests: efivarfs.sh [PASS]
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Executing selftests is fragile as if someone forgot to set a secript
as executable the test will fail, and you won't know for sure if the
failure was caused by the lack of proper permissions or something else.
Setting scripts as executable is required, this also enable folks to
execute selftests as independent units.
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Define CLEAN macro to allow Makefiles to override common clean target
in lib.mk. This will help fix the following failures:
warning: overriding recipe for target 'clean'
../lib.mk:55: warning: ignoring old recipe for target 'clean'
Fixes: 88baa78d1f ("selftests: remove duplicated all and clean target")
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
In commit a8ba798bc8 ("selftests: enable O and KBUILD_OUTPUT"), added
support to generate compile targets in a user specified directory. OUTPUT
variable controls the location which is undefined when tests are built in
the test directory or with "make -C tools/testing/selftests/x86".
make -C tools/testing/selftests/x86/
make: Entering directory '/lkml/linux_4.11/tools/testing/selftests/x86'
Makefile:44: warning: overriding recipe for target 'clean'
../lib.mk:51: warning: ignoring old recipe for target 'clean'
gcc -m64 -o /single_step_syscall_64 -O2 -g -std=gnu99 -pthread -Wall single_step_syscall.c -lrt -ldl
/usr/bin/ld: cannot open output file /single_step_syscall_64: Permission denied
collect2: error: ld returned 1 exit status
Makefile:50: recipe for target '/single_step_syscall_64' failed
make: *** [/single_step_syscall_64] Error 1
make: Leaving directory '/lkml/linux_4.11/tools/testing/selftests/x86'
Same failure with "cd tools/testing/selftests/x86/;make" run.
Fix this with a change to lib.mk to define OUTPUT to be the pwd when
MAKELEVEL is 0. This covers both cases mentioned above.
Reported-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Both these rules incorrectly use $< (first prerequisite) rather than
$^ (all prerequisites), meaning they don't work if we're using more than
one .S file as input. Switch them to using $^.
They also don't include $(CPPFLAGS) and other variables used in the
default rules, which breaks targets that require those. Fix that by
using the builtin $(COMPILE.S) and $(LINK.S) rules.
Fixes: a8ba798bc8 ("selftests: enable O and KBUILD_OUTPUT")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Tested by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Currently we can't build some tests, for example:
$ make -C tools/testing/selftests/ TARGETS=vm
...
gcc -Wall -I ../../../../usr/include -lrt -lpthread ../../../../usr/include/linux/kernel.h userfaultfd.c -o tools/testing/selftests/vm/userfaultfd
/tmp/ccmOkQSM.o: In function `stress':
userfaultfd.c:(.text+0xc60): undefined reference to `pthread_create'
userfaultfd.c:(.text+0xca5): undefined reference to `pthread_create'
userfaultfd.c:(.text+0xcee): undefined reference to `pthread_create'
userfaultfd.c:(.text+0xd30): undefined reference to `pthread_create'
userfaultfd.c:(.text+0xd77): undefined reference to `pthread_join'
userfaultfd.c:(.text+0xe7d): undefined reference to `pthread_join'
userfaultfd.c:(.text+0xe9f): undefined reference to `pthread_cancel'
userfaultfd.c:(.text+0xec6): undefined reference to `pthread_join'
userfaultfd.c:(.text+0xf14): undefined reference to `pthread_join'
/tmp/ccmOkQSM.o: In function `userfaultfd_stress':
userfaultfd.c:(.text+0x13e2): undefined reference to `pthread_attr_setstacksize'
collect2: error: ld returned 1 exit status
This is because the rule for linking .c files to binaries is incorrect.
The first bug is that it uses $< (first prerequisite) instead of $^ (all
preqrequisites), fix it by using ^$.
Secondly the ordering of the prerequisites vs $(LDLIBS) is wrong,
meaning on toolchains that use --as-needed we fail to link (as above).
Fix that by placing $(LDLIBS) *after* ^$.
Finally switch to using the default rule $(LINK.c), so that we get
$(CPPFLAGS) etc. included.
Fixes: a8ba798bc8 ("selftests: enable O and KBUILD_OUTPUT")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Tested by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
In commit 88baa78d1f ("selftests: remove duplicated all and clean
target"), the "all" target was removed from individual Makefiles and
added to lib.mk.
However the "all" target was added to lib.mk *after* the existing
"runtests" target. This means "runtests" becomes the first (default)
target for most of our Makefiles.
This has the effect of causing a plain "make" to build *and run* the
tests. Which is at best rude, but depending on which tests are run could
oops someone's build machine.
$ make -C tools/testing/selftests/
...
make[1]: Entering directory 'tools/testing/selftests/bpf'
gcc -Wall -O2 -I../../../../usr/include test_verifier.c -o tools/testing/selftests/bpf/test_verifier
gcc -Wall -O2 -I../../../../usr/include test_maps.c -o tools/testing/selftests/bpf/test_maps
gcc -Wall -O2 -I../../../../usr/include test_lru_map.c -o tools/testing/selftests/bpf/test_lru_map
#0 add+sub+mul FAIL
Failed to load prog 'Function not implemented'!
#1 unreachable FAIL
Unexpected error message!
#2 unreachable2 FAIL
...
Fix it by moving the "all" target to the start of lib.mk, making it the
default target.
Fixes: 88baa78d1f ("selftests: remove duplicated all and clean target")
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Tested by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Enable O and KBUILD_OUTPUT for kselftest. User could compile kselftest
to another directory by passing O or KBUILD_OUTPUT. And O is high
priority than KBUILD_OUTPUT.
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Some testcases need the clean extra data after running. This patch
introduce the "EXTRA_CLEAN" variable to address this requirement.
After KBUILD_OUTPUT is enabled in later patch, it will be easy to
decide to if we need do the cleanup in the KBUILD_OUTPUT path(if the
testcase ran immediately after compiled).
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
There are difference rules for compiling c source file in different
testcases. In order to enable KBUILD_OUTPUT support in later patch,
this patch introduce the default rules in
"tools/testing/selftest/lib.mk" and remove the existing rules in each
testcase.
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
The TEST_DIRS was introduced in Commit e8c1d7cdf1 ("selftests: copy
TEST_DIRS to INSTALL_PATH") for coping a whole directory in ftrace.
After rsync(with -a) is introduced by Commit 900d65ee11 ("selftests:
change install command to rsync"). Rsync could handle the directory
without the definition of TEST_DIRS.
This patch simply replace TEST_DIRS with TEST_FILES in ftrace and remove
the TEST_DIRS in tools/testing/selftest/lib.mk
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Currently, kselftest use TEST_PROGS, TEST_PROGS_EXTENDED, TEST_FILES to
indicate the test program, extended test program and test files. It is
easy to understand the purpose of these files. But mix of compiled and
uncompiled files lead to duplicated "all" and "clean" targets.
In order to remove the duplicated targets, introduce TEST_GEN_PROGS,
TEST_GEN_PROGS_EXTENDED, TEST_GEN_FILES to indicate the compiled
objects.
Also, the later patch will make use of TEST_GEN_XXX to redirect these
files to output directory indicated by KBUILD_OUTPUT or O.
And add this changes to "Contributing new tests(details)" of
Documentation/kselftest.txt.
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
The command of install could not handle the special files in exec
testcases, change the default rule to rsync to fix this.
The installation is unchanged after this commit.
Suggested-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
When the test cases is not supported by the current architecture
the install files(TEST_PROGS, TEST_PROGS_EXTENDED and TEST_FILES)
will be empty. Check it before installation to dismiss a failure
reported by install program.
Signed-off-by: Bamvor Jian Zhang <bamvor.zhangjian@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Loop over all TEST_DIRS and recursively copy them to the INSTALL_PATH. Tests
such as ftrace require a directory and all of it's contents to execute the
test properly, thus these directories and files need to be copied when we
perform an install.
Acked-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Tyler Baker <tyler.baker@linaro.org>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
This avoids repeating the logic in every Makefile. We mimic the
top-level Makefile and use $(CROSS_COMPILE)gcc.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Change the timers Makefile to make use of shared run and install logic
in lib.mk. Destructive tests are installed but not run by default.
Add a new variable, TEST_PROGS_EXTENDED, which is a list of extra
programs to install, but which are not run by the default run_tests
logic.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
This adds make install support to selftests. The basic usage is:
$ cd tools/testing/selftests
$ make install
That installs into tools/testing/selftests/install, which can then be
copied where ever necessary.
The install destination is also configurable using eg:
$ INSTALL_PATH=/mnt/selftests make install
The implementation uses two targets in the child makefiles. The first
"install" is expected to install all files into $(INSTALL_PATH).
The second, "emit_tests", is expected to emit the test instructions (ie.
bash script) on stdout. Separating this from install means the child
makefiles need no knowledge of the location of the test script.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
This adds a Make include file which most selftests can then include to
get the run_tests logic.
On its own this has the advantage of some reduction in repetition, and
also means the pass/fail message is defined in fewer places.
However the key advantage is it will allow us to implement install very
simply in a subsequent patch.
The default implementation just executes each program in $(TEST_PROGS).
We use a variable to hold the default implementation of $(RUN_TESTS)
because that gives us a clean way to override it if necessary, ie. using
override. The mount, memory-hotplug and mqueue tests use that to provide
a different implementation.
Tests are not run via /bin/bash, so if they are scripts they must be
executable, we add a+x to several.
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
