Features of popular test runners

Legend:

  • Parallel, possibility to run tests in parallel.
  • Fail fast, stop testing as soon as first test failed.
  • Last failed, possibility to run tests that were failed in the last run.
  • Impact analysis, possibility to choose only those tests that covers new changes.
  • Grouping, possibility to group test by attributes.
  • Shuffle, possibility to run tests in random order.
Tool Language Parallel Fail Fast Last Failed Impact Analysis Grouping Shuffle
Google Test C/C++ Yes1 ? ? ?
CTest (CMake) C/C++ Yes No Yes13 No
Pytest Python Yes2 Yes Yes Yes8 Yes20
nose Python Yes Yes Yes Yes6
go test Go Yes15 Yes14 ? Yes3 Yes18 Yes?19
cargo test Rust Yes Yes (by default) No4
RSpec Ruby Yes Yes16 Yes17 Yes5
PHPUnit PHP Yes11 Yes12 Yes9
JUnit Java Yes10
TestNG Java
C# ? ? Yes7
prove Perl Yes No Yes No
  1. gtest-parallel, gg
  2. pytest-xdist, pytest-parallel
  3. The go test command now caches test results: if the test executable and command line match a previous run and the files and environment variables consulted by that run have not changed either, go test will print the previous test output, replacing the elapsed time with the string “(cached).” Test caching applies only to successful test results; only to go test commands with an explicit list of packages; and only to command lines using a subset of the -cpu, -list, -parallel, -run, -short, and -v test flags. The idiomatic way to bypass test caching is to use -count=1. (Go 1.10+)
  4. https://github.com/rust-lang/cargo/issues/6826
  5. Crystalball, whattorun
  6. nose-knows is a nose plugin for figuring out which unit tests you should run after modifying code. It works by tracing your code while you run unit tests, and creating an output file that can be used later.
  7. https://www.ncrunch.net/
  8. pytest-knows, pytest-incremental, pytest-picked, pytest-cache, pytest-testmon, smother, python-tia, pytest-smartcollect and more
  9. Since PHPUnit v7.3
  10. https://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html#skipAfterFailureCount
  11. fastest, paratest
  12. --stop-on-*, https://phpunit.de/manual/6.5/en/textui.html
  13. The --rerun-failed option was added to CTest in CMake 3.0
  14. The new go test -failfast flag disables running additional tests after any test fails. Note that tests running in parallel with the failing test are allowed to complete. (Go 1.10+)
  15. Allow parallel execution of test functions that call t.Parallel. The value of this flag is the maximum number of tests to run simultaneously; by default, it is set to the value of GOMAXPROCS. Note that -parallel only applies within a single test binary. The ‘go test’ command may run tests for different packages in parallel as well, according to the setting of the -p flag.
  16. –fail-fast
  17. –only-failures
  18. The -short mode for go test allows us to mark any long-running tests to be skipped in this mode. go test and the testing package support this via the t.Skip(), the testing.Short() functions and the -short flag.
  19. https://github.com/golang/go/issues/28592
  20. pytest-random-order, pytest-randomly

Новые

Все посты →