kernel/tools/testing/kunit/kunit.py, branch linux-6.2.y

kunit: tool: make parser preserve whitespace when printing test log

2022-12-12T21:13:48Z

Currently, kunit_parser.py is stripping all leading whitespace to make parsing easier. But this means we can't accurately show kernel output for failing tests or when the kernel crashes. Embarassingly, this affects even KUnit's own output, e.g. [13:40:46] Expected 2 + 1 == 2, but [13:40:46] 2 + 1 == 3 (0x3) [13:40:46] not ok 1 example_simple_test [13:40:46] [FAILED] example_simple_test After this change, here's what the output in context would look like [13:40:46] =================== example (4 subtests) =================== [13:40:46] # example_simple_test: initializing [13:40:46] # example_simple_test: EXPECTATION FAILED at lib/kunit/kunit-example-test.c:29 [13:40:46] Expected 2 + 1 == 2, but [13:40:46] 2 + 1 == 3 (0x3) [13:40:46] [FAILED] example_simple_test [13:40:46] [SKIPPED] example_skip_test [13:40:46] [SKIPPED] example_mark_skipped_test [13:40:46] [PASSED] example_all_expect_macros_test [13:40:46] # example: initializing suite [13:40:46] # example: pass:1 fail:1 skip:2 total:4 [13:40:46] # Totals: pass:1 fail:1 skip:2 total:4 [13:40:46] ===================== [FAILED] example ===================== This example shows one minor cosmetic defect this approach has. The test counts lines prevent us from dedenting the suite-level output. But at the same time, any form of non-KUnit output would do the same unless it happened to be indented as well. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan

kunit: tool: make --json do nothing if --raw_ouput is set

2022-12-12T21:13:47Z

When --raw_output is set (to any value), we don't actually parse the test results. So asking to print the test results as json doesn't make sense. We internally create a fake test with one passing subtest, so --json would actually print out something misleading. This patch: * Rewords the flag descriptions so hopefully this is more obvious. * Also updates --raw_output's description to note the default behavior is to print out only "KUnit" results (actually any KTAP results) * also renames and refactors some related logic for clarity (e.g. test_result => test, it's a kunit_parser.Test object). Notably, this patch does not make it an error to specify --json and --raw_output together. This is an edge case, but I know of at least one wrapper around kunit.py that always sets --json. You'd never be able to use --raw_output with that wrapper. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan

kunit: tool: rename all_test_uml.config, use it for --alltests

2022-09-30T19:22:55Z

Context: 1. all_tests_uml.config used to be UML specific back when users to manually specify CONFIG_VIRTIO_UML=y to enable CONFIG_PCI=y. 2. --alltests used allyesconfig along with a curated list of options to disable. It's only ever worked for brief periods of time and has perennially been broken due to compile issues. Now all_tests_uml.config should work across ~all architectures. Let's instead use this to implement --alltests. Note: if anyone was using all_tests_uml.config, this change breaks them. I think that's unlikely since it was added in 5.19 and was a lot to type: --kunitconfig=tools/testing/kunit/configs/all_tests_uml.config. We could make it a symlink to the new name, but I don't think the caution is warranted here. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan

kunit: tool: make --raw_output=kunit (aka --raw_output) preserve leading spaces

2022-09-30T19:17:17Z

With $ kunit.py run --raw_output=all ... you get the raw output from the kernel, e.g. something like > TAP version 14 > 1..26 > # Subtest: time_test_cases > 1..1 > ok 1 - time64_to_tm_test_date_range > ok 1 - time_test_cases But --raw_output=kunit or equivalently --raw_output, you get > TAP version 14 > 1..26 > # Subtest: time_test_cases > 1..1 > ok 1 - time64_to_tm_test_date_range > ok 1 - time_test_cases It looks less readable in my opinion, and it also isn't "raw output." This is due to sharing code with kunit_parser.py, which wants to strip leading whitespace since it uses anchored regexes. We could update the kunit_parser.py code to tolerate leaading spaces, but this patch takes the easier way out and adds a bool flag. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Signed-off-by: Shuah Khan

kunit: tool: make --kunitconfig repeatable, blindly concat

2022-07-08T17:22:02Z

It's come up a few times that it would be useful to have --kunitconfig be repeatable [1][2]. This could be done before with a bit of shell-fu, e.g. $ find fs/ -name '.kunitconfig' -exec cat {} + | \ ./tools/testing/kunit/kunit.py run --kunitconfig=/dev/stdin or equivalently: $ cat fs/ext4/.kunitconfig fs/fat/.kunitconfig | \ ./tools/testing/kunit/kunit.py run --kunitconfig=/dev/stdin But this can be fairly clunky to use in practice. And having explicit support in kunit.py opens the door to having more config fragments of interest, e.g. options for PCI on UML [1], UML coverage [2], variants of tests [3]. There's another argument to be made that users can just use multiple --kconfig_add's, but this gets very clunky very fast (e.g. [2]). Note: there's a big caveat here that some kconfig options might be incompatible. We try to give a clearish error message in the simple case where the same option appears multiple times with conflicting values, but more subtle ones (e.g. mutually exclusive options) will be potentially very confusing for the user. I don't know we can do better. Note 2: if you want to combine a --kunitconfig with the default, you either have to do to specify the current build_dir > --kunitconfig=.kunit --kunitconfig=additional.config or > --kunitconfig=tools/testing/kunit/configs/default.config --kunitconifg=additional.config each of which have their downsides (former depends on --build_dir, doesn't work if you don't have a .kunitconfig yet), etc. Example with conflicting values: > $ ./tools/testing/kunit/kunit.py config --kunitconfig=lib/kunit --kunitconfig=/dev/stdin < CONFIG_KUNIT_TEST=n > CONFIG_KUNIT=m > EOF > ... > kunit_kernel.ConfigError: Multiple values specified for 2 options in kunitconfig: > CONFIG_KUNIT=y > vs from /dev/stdin > CONFIG_KUNIT=m > > CONFIG_KUNIT_TEST=y > vs from /dev/stdin > # CONFIG_KUNIT_TEST is not set [1] https://lists.freedesktop.org/archives/dri-devel/2022-June/357616.html [2] https://lore.kernel.org/linux-kselftest/CAFd5g45f3X3xF2vz2BkTHRqOC4uW6GZxtUUMaP5mwwbK8uNVtA@mail.gmail.com/ [3] https://lore.kernel.org/linux-kselftest/CANpmjNOdSy6DuO6CYZ4UxhGxqhjzx4tn0sJMbRqo2xRFv9kX6Q@mail.gmail.com/ Signed-off-by: Daniel Latypov Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: introduce --qemu_args

2022-07-08T00:00:05Z

Example usage: $ ./tools/testing/kunit/kunit.py run --arch=x86_64 \ --kconfig_add=CONFIG_SMP=y --qemu_args='-smp 8' Looking in the test.log, one can see > smp: Bringing up secondary CPUs ... > .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 > smp: Brought up 1 node, 8 CPUs This flag would allow people to make tweaks like this without having to create custom qemu_config files. For consistency with --kernel_args, we allow users to repeat this argument, e.g. you can tack on a --qemu_args='-m 2048', or you could just append it to the first string ('-smp 8 -m 2048'). Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: refactoring printing logic into kunit_printer.py

2022-07-07T23:46:25Z

Context: * kunit_kernel.py is importing kunit_parser.py just to use the print_with_timestamp() function * the parser is directly printing to stdout, which will become an issue if we ever try to run multiple kernels in parallel This patch introduces a kunit_printer.py file and migrates callers of kunit_parser.print_with_timestamp() to call kunit_printer.stdout.print_with_timestamp() instead. Future changes: If we want to support showing results for parallel runs, we could then create new Printer's that don't directly write to stdout and refactor the code to pass around these Printer objects. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: redo how we construct and mock LinuxSourceTree

2022-07-07T23:45:59Z

Our main function currently has an optional `linux` argument which is used to by our unit tests to inject a mock. We currently have the same code copy-pasted several times to do if not linux: linux = MakeRealInstance(cli_args.foo, cli_args.bar, ...) But in python, dependency injection isn't necessary or idiomatic when we can just use mock.patch() to mock things out. This change 1. adds a helper to create a LinuxSourceTree from the cli_args 2. drops the `linux` parameter in favor of mocking the __init__ func. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: misc cleanups

2022-05-16T19:22:36Z

This primarily comes from running pylint over kunit tool code and ignoring some warnings we don't care about. If we ever got a fully clean setup, we could add this to run_checks.py, but we're not there yet. Fix things like * Drop unused imports * check `is None`, not `== None` (see PEP 8) * remove redundant parens around returns * remove redundant `else` / convert `elif` to `if` where appropriate * rename make_arch_qemuconfig() param to base_kunitconfig (this is the name used in the subclass, and it's a better one) * kunit_tool_test: check the exit code for SystemExit (could be 0) Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan

kunit: tool: more descriptive metavars/--help output

2022-04-04T22:22:14Z

Before, our help output contained lines like --kconfig_add KCONFIG_ADD --qemu_config qemu_config --jobs jobs They're not very helpful. The former kind come from the automatic 'metavar' we get from argparse, the uppercase version of the flag name. The latter are where we manually specified metavar as the flag name. After: --build_dir DIR --make_options X=Y --kunitconfig PATH --kconfig_add CONFIG_X=Y --arch ARCH --cross_compile PREFIX --qemu_config FILE --jobs N --timeout SECONDS --raw_output [{all,kunit}] --json [FILE] This patch tries to make the code more clear by specifying the _type_ of input we expect, e.g. --build_dir is a DIR, --qemu_config is a FILE. I also switched it to uppercase since it looked more clearly like placeholder text that way. This patch also changes --raw_output to specify `choices` to make it more clear what the options are, and this way argparse can validate it for us, as shown by the added test case. Signed-off-by: Daniel Latypov Reviewed-by: David Gow Reviewed-by: Brendan Higgins Signed-off-by: Shuah Khan