pytest-retry

Bug Fixes

• use verbosity count for trace limit rather than unlimited trace depth in verbose mode @str0zzapreti in https://github.com/str0zzapreti/pytest-retry/pull/41
• Code snippets are now highlighted in the readme for supported editors (thanks @Borda)

Full Changelog: https://github.com/str0zzapreti/pytest-retry/compare/1.6.2...1.6.3

Bug Fixes

• fixed Item instantiation error in Pytest 8 by @str0zzapreti in https://github.com/str0zzapreti/pytest-retry/pull/35

Full Changelog: https://github.com/str0zzapreti/pytest-retry/compare/1.6.1...1.6.2

Bug Fixes

• Fixed an OSError exception which could occur due to a socket port assignment conflict when running multiple instances of pytest-retry with pytest-xdist n > 1 simultaneously on the same machine

New Features

• Retry reports will now be displayed when using pytest-xdist with n > 1
• Retry reports now log if a retried test passed on a subsequent attempt. Previously, only failures were logged

Fixes

• Additional test coverage to validate xdist compatability for retry reports
• Additional test coverage to prevent regressions with fixture resets during retries
• Cleaned up some internal naming for clarity and consistency

Full Changelog: https://github.com/str0zzapreti/pytest-retry/compare/1.5.0...1.6.0

New Features

• Global retries can now be configured using .ini files as an alternative to using command line options. This also adds pyproject.toml support for free. See the updated readme for details.

Bug Fixes

• Cumulative timing mode only summed first and final attempt durations when retries > 1.
• Empty flaky mark would default to 0 retries, but still retry once (task failed successfully)

Bug fixes

• Fix for issue #20: Unexpected strict xfail exception

Hotfix Release

Bug Fixes:

• Fixed a module directory which was causing an import error when attempting to retry tests.

New features:

• A new condition argument is available for the flaky mark which provides greater control over when tests are retried. Tests with a condition argument in the flaky mark will only be retried if condition is True. This can be used to, for example, configure a test to only retry when running on a specific operating system or during a specific time of day.

Note: There is no matching command line parameter for condition, but the documentation has been updated to suggest an easy workaround if you need to globally add the same condition to all of your tests at once without marking each individually.

BREAKING CHANGES: Previously, any arguments not specified in a flaky mark would fall back to the default plugin values. From 1.4.0 onwards, unspecified arguments will now fall back to your global defaults if specified. To understand the difference in behavior, consider the following case:

pytest-retry defaults to a value of 0 for retry delay unless specified via the command line or a flaky mark

You run the following test file

import pytest


@pytest.mark.flaky(retries=3)
def test_unreliable_function():
    # testing some unreliable function here
    # assert expected function result

with the following command line arguments:

python -m pytest --retries=2, --retry-delay=5

Since the flaky mark does not specify a delay argument, prior to version 1.4.0, test_unreliable_function would be retried 3 times (flaky mark args override command line) with 0 delay between attempts. From version 1.4.0 onwards, test_unreliable_function would be retried 3 times with a 5 second delay between each attempt.

In other words, flaky marks before 1.4.0 silently ignore all of your command line options, while flaky marks from 1.4.0 onward will respect any command line options that you did not explicitly wish to override

If you have a configuration which relies on the previous behavior and want to upgrade, it should be simple to update your affected tests accordingly. This change was made according to the principle of least surprise (the new behavior appears more intuitive), but please feel free to provide feedback on this change.

Bug Fixes: Fixed a bug where 'excluded' exceptions passed to a flaky mark were not being filtered. Fixed a test which was validating this incorrect behavior by mistake.

New features:

• Test retries can now be made conditional upon a list of filtered or excluded exception types. Filtered exceptions will only initiate retries if the failed test attempt occurred due to an exception in the list. Excluded exceptions will only initiate retries if the failed test attempt does not have an exception matching a listed exception.
• At a global level, this feature is implemented using two new hooks, either of which (but not both) can be defined in conftest.py.
• At a per-test level, two new arguments have been added to the flaky mark. Again, only one of these can be specified at a time.

Hotfix release

Fixes:

• Prevented a possible durations index error if another exception occurred during the pytest_report_teststatus stage. (e.g. if using a custom teststatus hook in conjunction with this plugin).
• Switched back to overwriting the pytest report outcome (despite its Literal typing) because otherwise the counters don't play nice.

New featues:

• Overhauled internal test reporting system to track outcome, attempts, and duration per test
• Instead of a boolean success key, there's now a str outcome key which can be 'passed', 'failed', or 'skipped'
• There is now a duration key (importable from the plugin) which can be used to retrieve test duration from item stashs
• Deprecated success_key

Fixed the following bugs:

• Unhandled exceptions during teardown cause runtest process to fail on a stash KeyError

New featues:

• Final reported test durations are now based on the latest test attempt by default.
• The old behavior, where all attempt durations were summed into the final duration, can still be enabled via the --cumulative-timings=1 flag or via the flaky marker timings arg
• Instead of attaching a full report to a test item as an attr (discouraged by Pytest documentation). The outcome of the test and the number of attempts are now safely saved to each item's stash. Both keys success_key and attempt_key can be imported and used to access these values in other hooks or fixtures.

Changelog:

• Test durations are now overwritten by the most recent attempt by default. This is to provide more consistent data to time-based splitting algorithms, such as CircleCI's.
• new --cumulative-timings flag (defaults to False). Can be used to revert to the previous behavior where the duration of each attempt is summed together in the final report
• Latest report instance is no longer attached to the item as an attribute. The solution encouraged by the Pytest documentation is to use the item's Stash instance. Two new keys are used to store the number of attempts for each item and whether it passed or failed.
• Both stash keys can be imported from the plugin to access the outcome data for other purposes.

Changelog:

• The latest original report is now attached to the test item for easy access

Initial release of pytest-retry