kernel/tools/testing/selftests/tc-testing/tdc.py, branch linux-rolling-stable

tc-testing: don't access non-existent variable on exception

2024-08-29T15:35:56Z

[ Upstream commit a0c9fe5eecc97680323ee83780ea3eaf440ba1b7 ] Since commit 255c1c7279ab ("tc-testing: Allow test cases to be skipped") the variable test_ordinal doesn't exist in call_pre_case(). So it should not be accessed when an exception occurs. This resolves the following splat: ... During handling of the above exception, another exception occurred: Traceback (most recent call last): File ".../tdc.py", line 1028, in main() File ".../tdc.py", line 1022, in main set_operation_mode(pm, parser, args, remaining) File ".../tdc.py", line 966, in set_operation_mode catresults = test_runner_serial(pm, args, alltests) File ".../tdc.py", line 642, in test_runner_serial (index, tsr) = test_runner(pm, args, alltests) File ".../tdc.py", line 536, in test_runner res = run_one_test(pm, args, index, tidx) File ".../tdc.py", line 419, in run_one_test pm.call_pre_case(tidx) File ".../tdc.py", line 146, in call_pre_case print('test_ordinal is {}'.format(test_ordinal)) NameError: name 'test_ordinal' is not defined Fixes: 255c1c7279ab ("tc-testing: Allow test cases to be skipped") Signed-off-by: Simon Horman Acked-by: Jamal Hadi Salim Link: https://patch.msgid.link/20240815-tdc-test-ordinal-v1-1-0255c122a427@kernel.org Signed-off-by: Jakub Kicinski Signed-off-by: Sasha Levin

selftests: tc-testing: return fail if a test fails in setup/teardown

2024-01-26T00:38:16Z

As of today tests throwing exceptions in setup/teardown phase are treated as skipped but they should really be failures. Signed-off-by: Pedro Tammela Reviewed-by: Davide Caratti Acked-by: Jamal Hadi Salim Reviewed-by: Davide Caratti Link: https://lore.kernel.org/r/20240124181933.75724-6-pctammela@mojatatu.com Signed-off-by: Jakub Kicinski

selftests: tc-testing: cleanup on Ctrl-C

2023-11-28T02:15:43Z

Cleanup net namespaces and other resources if we get a SIGINT (Ctrl-C). As user visible resources are allocated on a per test basis, it's only required to catch this condition when (possibly) running tests. So far calling post_suite is enough to free up anything that might linger. A missing keyword replacement for nsPlugin is also included. Signed-off-by: Pedro Tammela Acked-by: Jamal Hadi Salim Link: https://lore.kernel.org/r/20231124154248.315470-5-pctammela@mojatatu.com Signed-off-by: Jakub Kicinski

selftests: tc-testing: remove unnecessary time.sleep

2023-11-28T02:15:42Z

This operation is redundant and it's not stabilizing nor waiting for anything. Signed-off-by: Pedro Tammela Acked-by: Jamal Hadi Salim Link: https://lore.kernel.org/r/20231124154248.315470-3-pctammela@mojatatu.com Signed-off-by: Jakub Kicinski

selftests: tc-testing: report number of workers in use

2023-11-21T02:06:36Z

Report the number of workers in use to process the test batches. Since the number is now subject to a limit, avoid users getting confused. Signed-off-by: Pedro Tammela Reviewed-by: Simon Horman Acked-by: Jamal Hadi Salim Link: https://lore.kernel.org/r/20231117171208.2066136-7-pctammela@mojatatu.com Signed-off-by: Jakub Kicinski

selftests: tc-testing: cap parallel tdc to 4 cores

2023-11-21T02:06:35Z

We have observed a lot of lock contention and test instability when running with >8 cores. Enough to actually make the tests run slower than with fewer cores. Cap the maximum cores of parallel tdc to 4 which showed in testing to be a reasonable number for efficiency and stability in different kernel config scenarios. Signed-off-by: Pedro Tammela Reviewed-by: Simon Horman Acked-by: Jamal Hadi Salim Link: https://lore.kernel.org/r/20231117171208.2066136-2-pctammela@mojatatu.com Signed-off-by: Jakub Kicinski

selftests/tc-testing: implement tdc parallel test run

2023-09-28T07:51:07Z

Use a Python process pool to run the tests in parallel. Not all tests can run in parallel, for instance tests that are not namespaced and tests that use netdevsim, as they can conflict with one another. The code logic will split the tests into serial and parallel. For the parallel tests, we build batches of 32 tests and queue each batch on the process pool. For the serial tests, they are queued as a whole into the process pool, which in turn executes them concurrently with the parallel tests. Even though the tests serialize on rtnl_lock in the kernel, this feature showed results with a ~3x speedup on the wall time for the entire test suite running in a VM: Before - 4m32.502s After - 1m19.202s Examples: In order to run tdc using 4 processes: ./tdc.py -J4 <...> In order to run tdc using 1 process: ./tdc.py -J1 <...> || ./tdc.py <...> Note that the kernel configuration will affect the speed of the tests, especially if such configuration slows down process creation and/or fork(). Tested-by: Davide Caratti Signed-off-by: Pedro Tammela Acked-by: Jamal Hadi Salim Signed-off-by: Paolo Abeni

selftests/tc-testing: localize test resources

2023-09-28T07:51:07Z

As of today, the current tdc architecture creates one netns and uses it to run all tests. This assumption was embedded into the nsPlugin which carried over as how the tests were written. The tdc tests are by definition self contained and can, theoretically, run in parallel. Even though in the kernel they will serialize over the rtnl lock, we should expect a significant speedup of the total wall time for the entire test suite, which is hitting close to 1100 tests at this point. A first step to achieve this goal is to remove sharing of global resources like veth/dummy interfaces and the netns. In this patch we 'localize' these resources on a per test basis. Each test gets it's own netns, VETH/dummy interfaces. The resources are spawned in the pre_suite phase, where tdc will prepare all netns and interfaces for all tests. This is done in order to avoid concurrency issues with netns / interfaces spawning and commands using them. As tdc progresses, the resources are deleted after each test finishes executing. Tests that don't use the nsPlugin still run under the root namespace, but are now required to manage any external resources like interfaces. These cannot be parallelized as their definition doesn't allow it. On the other hand, when using the nsPlugin, tests don't need to create dummy/veth interfaces as these are handled already. Tested-by: Davide Caratti Signed-off-by: Pedro Tammela Acked-by: Jamal Hadi Salim Signed-off-by: Paolo Abeni

selftests: tc-testing: add "depends_on" property to skip tests

2023-03-31T06:24:24Z

currently, users can skip individual test cases by means of writing "skip": "yes" in the scenario file. Extend this functionality, introducing 'dependsOn': it's optional property like "skip", but the value contains a command (for example, a probe on iproute2 to check if it supports a specific feature). If such property is present, tdc executes that command and skips the test when the return value is non-zero. Reviewed-by: Pedro Tammela Signed-off-by: Davide Caratti Signed-off-by: Jakub Kicinski

selftests: tc-testing: Add matchJSON to tdc

2022-10-27T03:22:33Z

This allows the use of a matchJSON field in tests to match against JSON output from the command under test, if that command outputs JSON. You specify what you want to match against as a JSON array or object in the test's matchJSON field. You can leave out any fields you don't want to match against that are present in the output and they will be skipped. An example matchJSON value would look like this: "matchJSON": [ { "Value": { "neighIP": { "family": 4, "addr": "AQIDBA==", "width": 32 }, "nsflags": 142, "ncflags": 0, "LLADDR": "ESIzRFVm" } } ] The real output from the command under test might have some extra fields that we don't care about for matching, and since we didn't include them in our matchJSON value, those fields will not be attempted to be matched. If everything we included above has the same values as the real command output, the test will pass. The matchJSON field's type must be the same as the command output's type, otherwise the test will fail. So if the command outputs an array, then the value of matchJSON must also be an array. If matchJSON is an array, it must not contain more elements than the command output's array, otherwise the test will fail. Signed-off-by: Jeremy Carter Signed-off-by: Victor Nogueira Acked-by: Jamal Hadi Salim Link: https://lore.kernel.org/r/20221024111603.2185410-1-victor@mojatatu.com Signed-off-by: Jakub Kicinski