Potential alternatives to python-tia

This section summarizes the evaluation of Python packages (listed in alphabetic order) which could potentially have been used instead of python-tia and packages which could be used to implement python-tia s features. At time of writing (last update: October 2018) existing packages provide a subset but not all required features. The functionality is not provided in a generic way, tightly coupled to the Python language and a specific Python test runner (pytest and nose). The packages implement single features sometimes using very different approaches which are used as design input for the implementation in python-tia. Note because almost all alternatives depend on/are using coverage.py for coverage mapping: At time of project initializations coverage.py did not offer it’s Who tests what (coverage data on a per test basis) feature but accumulated coverage data (coverage on per test run basis).

pytest-incremental

looks for imports recursively to find dependencies (using AST) [caveats](https://pytest-incremental.readthedocs.io/runner.html#caveats): * only modules imported with import are detected * modules not explicitly imported but used at run-time * monkey patched modules are not detected * dependency cycles in imports may affect detection in a bad way

pytest-knows

Sources: ptknows (github)

Package: pytest-knows (pypi)

Evaluation result: No option due to pytest test runner dependency.

pytest-knows is a pytest plugin which makes use of trace and stat.ST_MTIME (https://docs.python.org/2/library/stat.html#stat.ST_MTIME) (time of last file modification). During setup of pytest via the pytest hook pytest_configure() (https://github.com/mapix/ptknows/blob/master/ptknows.py#L47) it opens an UNIX database via the Python 2 dbm interface (in Python 3 the module has been renamed to dbm.ndbm ). Before pytest runs a single test pytest-knows hooks into there via the pytest hook pytest_runtest_call() (https://github.com/mapix/ptknows/blob/master/ptknows.py#L55). It is checked if dependency info for this test (mapping of test to executed production code files) has been stored into the database before. If there is info available and the last modification time of the production code file corresponding to the test has not changed the test is skipped. In case there is no dependency info or the last modification time of one of the tests associated production code files has changed the test is executed. During test execution trace info is gathered and the dependency information for the test (mapping of test to executed production code files) stored in the database. After execution the databaes is closed via pytest hook pytest_unconfigure() (https://github.com/mapix/ptknows/blob/master/ptknows.py#L51).

pytest-picked

Sources: pytest-picked (github)

Package: pytest-picked (pypi)

Evaluation result: No option due to pytest test runner dependency.

pytest-picked is a pytest plugin which makes use of git. It does not create a coverage map and impact map. Instead it uses git status --short (git wrapped via subprocess) to determine test files and folders which have been changed locally.

pytest-smartcollect

Lead developer: Abigail Hahn (Microsoft)

Package: PyPI (pytest-smartcollect 1.0.2)

Source code: GitHub

Documentation: Read The Docs

Change detection

VCS based file and file line scope change detection with gitpython. Test function scope detection of changed lines in function body. “Member” scope change detection with abstract syntax tree parsing using ast. “Members” are any names that can be imported from a module, including assignments, function definitions and class definitions. Test function scope detection based on use of a changed member from another module. Supports detection based on composition (function and class definitions) and inheritance (class definitions).

Semantic mapping

No semantic mapping.

Pipeline execution

No pipeline execution.

Coverage mapping

TODO

Impact mapping

TODO

History

TODO

pytest-testmon

Sources: pytest-testmon (github)

Package: pytest-testmon (pypi)

Evaluation result: No option due to pytest test runner dependency.

pytest-testmon is a pytest plugin which automatically selects and re-executes only tests affected by recent changes. File scope changes are determined based on hash using hashlib and zlib.adler32(). ast is used to generate Python language aware change interpretation. For coverage tracking coveragpy is used. Subprocess coverage tracking requires manual installation of coverage-pth. The standard lib module ast is used to create an abstract representation of the source code as input for pytest-testmon can be used with pytest-watch to trigger its execution automatically.

blog article about why testmon moved from file change time to hash approach and distributed test extecution: https://medium.com/@boxed/vastly-faster-python-integration-tests-9d8106b3693c

smother

Lead developer: Chris Beaumont (Netflix)

Package: PyPI (smother 0.2)

Source code: GitHub

Documentation: Read The Docs

Change detection

VCS based file scope change detection with subprocess wrapped git. VCS diff and AST based Python scope sub-file change detection. (Intervals represent code regions.)

Semantic mapping

No semantic mapping. (Smother is limited to single test runner run.)

Pipeline execution

No pipeline execution. (Smother is a wrapper for coverage.py and doesn’t execute pipelines.)

Coverage mapping

Test runner integration based generation of coverage.py test coverage data with pytest plugin or nose plugin. Per test based coverage data is stored in smother files.

Impact mapping

The coverage mapping is inverted into impact mapping for reporting only. (Smother is a wrapper for coverage.py and doesn’t support pipeline execution.)

History

The initial codebase that smother was designed for took nearly 24 hours of CPU time to run its 11K tests.

—Chris Beaumont (Smother - Why?)

“testimpact” script

Sources: samplemod (github)

Paul Hammant presented a proof of concept script for TIA in his blog post about samplemod “Reducing Test Times by Only Running Impacted Tests - Python Edition”. The script testimpact.sh (https://github.com/paul-hammant/samplemod/blob/master/testimpact.sh) determines the test files using ack (https://github.com/paul-hammant/samplemod/blob/master/testimpact.sh#L7), runs every test with nosetest (https://github.com/paul-hammant/samplemod/blob/master/testimpact.sh#L15), determines which production code is executed by each test and writes the coverage map into meta data directory meta/ (directory meta/tests (https://github.com/paul-hammant/samplemod/tree/master/meta/tests) and meta/tests2 (https://github.com/paul-hammant/samplemod/tree/master/meta/tests2). The resulting impact map (production code vs. test code which executes the production code) ends up in meta/impact-map.txt (https://github.com/paul-hammant/samplemod/blob/master/meta/impact-map.txt).

watchdog

Sources: watchdog (github)

Documentation: watchdog (docs)

Package: watchdog (pypi)

watchdog provides various filesystem observers: an OS native filesystem observer (https://pythonhosted.org/watchdog/api.html#module-watchdog.observers), a polling observer (https://pythonhosted.org/watchdog/api.html#module-watchdog.observers.polling), etc. Observers trigger event handlers (https://pythonhosted.org/watchdog/api.html#event-handler-classes) in case of various filesystem events (https://pythonhosted.org/watchdog/api.html#event-classes).

when-changed

Sources: when-changed (github)

Evaluation result: No option due to limited capabilities.

when-changed is a command line tool which executes commands when a directories or files have changed. File changes are detected continuously using watchdog (github) class watchdog.events.FileSystemEventHandler() (https://github.com/joh/when-changed/blob/master/whenchanged/whenchanged.py#L36) and watchdog.observers.Observer().

nose-knows

Sources: nose-knows (github)

Package: nose-knows (pypi)

Evaluation result: No option due to nose test runner dependency.

nose-knows is a nose plugin with experimental support for pytest. The coverage map (.knows file) maps production code on the file level vs. tests (created in “output mode”, cmd line option --knows-out). In Knows.begin() (https://github.com/eventbrite/nose-knows/blob/master/src/knows/base.py#L58) it makes use of threading.settrace(self.tracer) with the tracer function Knows.tracer() (https://github.com/eventbrite/nose-knows/blob/master/src/knows/base.py#L63) to trace the production code executed during tests. begin() is integrated into the test runner processing procedure for nose via KnowsNosePlugin.begin() (https://github.com/eventbrite/nose-knows/blob/master/src/knows/nose_plugin.py#L105>`_, for pytest via pytest_sessionstart() (https://github.com/eventbrite/nose-knows/blob/master/src/knows/pytest_plugin.py#L94>). The trace context for particular tests is determined via Knows.start_test() (https://github.com/eventbrite/nose-knows/blob/master/src/knows/base.py#L84) which is called in the plugins via the corresponding test runner hooks for nose via KnowsNosePlugin.startTest() (https://github.com/eventbrite/nose-knows/blob/master/src/knows/nose_plugin.py#L108), for pytest via pytest_runtest_protocol() (https://github.com/eventbrite/nose-knows/blob/a647cc1f82984522f728ccc83145c774f4756197/src/knows/pytest_plugin.py#L99). In “input mode” the coverage map (.knows file) is used to generate the impact map dynamically Knows.get_tests_to_run() (https://github.com/eventbrite/nose-knows/blob/3ac3cfc81c7d3bc7beaf2b533ab37a0bbf132779/src/knows/base.py#L26) for a production code file and to selectivelly run tests for it.