pytest-codspeed

What's Changed

• build: enable free-threaded wheels in cibuildwheel by @art049 in #121

Full Changelog: v5.0.0...v5.0.1

Highlights

MacOS walltime profiling is now available with the codspeed cli v4.16.1 and above.

pytest-codspeed can now be used in free threaded mode. This has been tested with 3.14t and 3.15t.
For this, we have dropped usage of cffi in favor of the native extension support.

What's Changed

• chore: bump instrument-hooks by @not-matthias in #119
• Remove CFFI in favor of native extension to support free-threaded mode by @adriencaccia in #96
• feat(hooks): declare native extension free-thread safe by @art049 in #120
• feat: use instrument-hook markers in walltime by @GuillaumeLagrange in #118

Full Changelog: v4.5.0...v5.0.0

What's Changed

This release adds first support for macOS walltime.

Please note that profiling and other instruments are not yet available on macOS and will come in a later update.

• Support macos walltime without profiling in pytest-codspeed by @GuillaumeLagrange in #117

Full Changelog: v4.4.0...v4.5.0

What's Changed

We now collect buildtime and runtime environment data to warn users about differences in their runtime environment when comparing two runs against one another.

This data includes toolchain metadata like version and build options, as well as a list of dynamically loaded linked libraries.

• fix: fix segfault caused by multiple activate_stack_trampoline by @GuillaumeLagrange in #109
• fix: Exclude setup time from benchmark in walltime mode by @toby-coleman in #114
• feat: collect Python toolchain information via instrument hooks environment API by @GuillaumeLagrange in #108

New Contributors

• @toby-coleman made their first contribution in #114

Full Changelog: v4.3.0...v4.4.0

What's Changed

This release brings support for the memory instrument, which enables you to track memory usage, heap allocations, and memory leaks in your benchmarks.

• Bump uv in CI from 0.5.20 to 0.9.5 by @edgarrmondragon in #95
• feat/improve local walltime by @adriencaccia in #97
• feat: rename instrumentation to simulation by @GuillaumeLagrange in #98
• Pin python in ci integration tests to prevent intermitent codspeed segfaults in walltime by @GuillaumeLagrange in #105
• feat: add memory profiling support by @not-matthias in #103
• feat: add .gitignore to .codspeed folder on creation by @art049 in #107

New Contributors

• @GuillaumeLagrange made their first contribution in #98

Full Changelog: v4.2.0...v4.3.0

What's Changed

• Support python 3.14 by @adriencaccia in #91

Full Changelog: v4.1.1...v4.2.0

What's Changed?

• fix: build on macos with instrument-hooks by @not-matthias in #90

Full Changelog: v4.1.0...v4.1.1

What's New?

🚀 The new version of CodSpeedHQ/instrument-hooks will improve the quality of the walltime profiling!

Details

✨ Features

• Bump instrument-hooks (#87) by @not-matthias in #87

<!-- 1 -->🐛 Bug Fixes

• Use ParamSpec in instrument definition instead of tuple and dict by @art049
• Use ParamSpec in the plugin call definition by @flying-sheep

<!-- 3 -->📚 Documentation

• Add pull_request trigger comment by @adriencaccia

<!-- 7 -->⚙️ Internals

• Exclude beta tags in changelog by @art049
• Switch to sharded upload (#75) by @art049 in #75
• Fix changelog generation by @art049

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v4.0.0...v4.1.0

What's Changed

This release introduces profiling to the walltime instrument and includes several key improvements to the existing benchmark fixture API! 🎉

[!WARNING]
Since we're now using CodSpeedHQ/instrument-hooks to control the instrument state, the performance may slightly change in tiny microbenchmarks when upgrading.

🚀 Features

• Support perf profiling in Python walltime instrument by @not-matthias in https://github.com/CodSpeedHQ/pytest-codspeed/pull/79
• Support pytest-benchmark marker attributes by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/80
• Support the pedantic API by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/81

New Contributors

• @not-matthias made their first contribution in https://github.com/CodSpeedHQ/pytest-codspeed/pull/79

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v3.2.0...v4.0.0

What's Changed

This release improves the reliability of the measurement, primarily focusing on the wall-time instrument.

<!-- 0 -->🚀 Features

• Increase the min round time to a bigger value (+/- 1ms) by @art049

• Fix the random seed while measuring with instruments by @art049 in #48

<!-- 1 -->🐛 Bug Fixes

• Use time per iteration instead of total round time in stats by @art049

<!-- 2 -->🏗️ Refactor

• Replace hardcoded outlier factor for improved readability by @art049 in #67

<!-- 7 -->⚙️ Internals

• Fix self-dependency by @adriencaccia in #66
• Fix uv version in CI by @adriencaccia
• Add benchmarks-walltime job to run additional performance benchmarks by @art049 in #65

⚠️ Since the wall-time instrument is still in beta, this minor release might create unexpected performance changes for this new instrument.

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v3.1.2...v3.2.0

What's Changed

<!-- 1 -->🐛 Bug Fixes

• fix: update package_data to include header and source files for valgrind wrapper by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/64

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v3.1.1...v3.1.2

What's Changed

<!-- 7 -->⚙️ Internals

• Add a py3-none-any fallback wheel by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/61

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v3.1.0...v3.1.1

What's changed

<!-- 0 -->🚀 Features

• Build the native extension during packaging to avoid runtime build issues by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/57

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v3.0.0...v3.1.0

What's Changed

This new release includes a new wall time instrument that allows measurement of real execution timings. This will be very useful for macro benchmarks running on properly isolated machines. You can now use our CodSpeed Macro runners without changing anything to your existing CI workflows.

Find more info on our Walltime instrument here.

Breaking changes

• Dropped support for Python 3.8 since it reached its EOL

Features

• feat: add walltime clock benchmarking by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/41
• feat/add max time and quartiles by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/45
• feat: also save the lower and upper fences in the json data by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/46

Fixes

• fix: pytest-benchmark 5.0.0 compatibility by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/54
• fix: don't override pytest's default protocol by @kenodegard in https://github.com/CodSpeedHQ/pytest-codspeed/pull/32

Internals

• chore: Add TCH, FA, and UP lints by @kenodegard in https://github.com/CodSpeedHQ/pytest-codspeed/pull/29
• Expose type information by @Dreamsorcerer in https://github.com/CodSpeedHQ/pytest-codspeed/pull/53
• ci: bump actions/checkout to v4 by @fargito in https://github.com/CodSpeedHQ/pytest-codspeed/pull/47
• chore: support python 3.13 and drop 3.7 by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/40
• feat: use a true semver version by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/44
• test: add benches for TheAlgorithms/backtracking by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/43
• test: add benches for TheAlgorithms/audio_filters by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/42
• docs(readme): update action version in the CI workflow configuration by @frgfm in https://github.com/CodSpeedHQ/pytest-codspeed/pull/39

New Contributors

• @kenodegard made their first contribution in https://github.com/CodSpeedHQ/pytest-codspeed/pull/29
• @frgfm made their first contribution in https://github.com/CodSpeedHQ/pytest-codspeed/pull/39
• @fargito made their first contribution in https://github.com/CodSpeedHQ/pytest-codspeed/pull/47
• @Dreamsorcerer made their first contribution in https://github.com/CodSpeedHQ/pytest-codspeed/pull/53

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v2.2.1...v3.0.0

<!-- 0 -->🚀 Features

• Send more outlier data by @art049

<!-- 1 -->🐛 Bug Fixes

• Fix display of parametrized tests by @art049
• Reenable gc logic by @art049

<!-- 6 -->🧪 Testing

• Add benches for various syscalls by @art049

<!-- 0 -->🚀 Features

• Also save the lower and upper fences in the json data by @art049 in #46

<!-- 6 -->🧪 Testing

• Refactor the algorithm benches using parametrization and add benches on bit_manipulation by @art049

<!-- 0 -->🚀 Features

• Also save the q1 and q3 in the json data by @art049 in #45
• Add the --codspeed-max-time flag by @art049

<!-- 0 -->🚀 Features

• Send the semver version to cospeed instead of the PEP440 one by @art049 in #44
• Also store the semver version by @art049

<!-- 6 -->🧪 Testing

• Add benches for TheAlgorithms/backtracking by @art049 in #43

<!-- 0 -->🚀 Features

• Improve table style when displaying results by @art049 in #41
• Add the total bench time to the collected stats by @art049
• Add configuration and split tests between instruments by @art049
• Add outlier detection in the walltime instrument by @art049
• Implement the walltime instrument by @art049
• Add bench of various python noop by @art049
• Avoid overriding pytest's default protocol (#32) by @kenodegard in #32

<!-- 1 -->🐛 Bug Fixes

• Use importlib_metadata to keep backward compatibility by @art049
• Properly decide the mode depending on our env variable spec by @art049
• Disable pytest-speed when installed and codspeed is enabled by @art049

<!-- 2 -->🏗️ Refactor

• Differentiate the mode from the underlying instrument by @art049
• Move the instrumentation wrapper directly in the instrument by @art049
• Change Instrumentation to CPUInstrumentation by @art049
• Create an abstraction for each instrument by @art049

<!-- 3 -->📚 Documentation

• Update action version in the CI workflow configuration (#39) by @frgfm in #39
• Bump action versions in README by @adriencaccia

<!-- 6 -->🧪 Testing

• Add benches for TheAlgorithms/audio_filters by @art049 in #42

<!-- 7 -->⚙️ Internals

• Add a test on the walltime instrument by @art049
• Fix utils test using a fake git repo by @art049
• Update readme by @art049
• Support python 3.13 and drop 3.7 by @art049 in #40
• Add TCH, FA, and UP to ruff lints (#29) by @kenodegard in #29

What's Changed

• fix: Loosen runtime requirements by @edgarrmondragon in https://github.com/CodSpeedHQ/pytest-codspeed/pull/21
• fix: support pytest 8.1.1 by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/28

New Contributors

• @edgarrmondragon made their first contribution in https://github.com/CodSpeedHQ/pytest-codspeed/pull/21

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v2.2.0...v2.2.1

What's Changed

pytest-xdist is now fully supported. Thus if you have a lot of benches, you can totally leverage xdist test distribution (on the same CPU) to speed up benchmark execution.

Details

• feat: fix pytest-xdist-compatibility by @art049 in #18

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v2.1.0...v2.2.0

What's Changed

Relative paths are now properly handled when using the working-directory action parameter.

Details

• fix: git relative path with working-directory by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/15

Internals

• fix: fix typo in release.yml by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/14

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v2.0.1...v2.1.0

What's Changed

Added

• Added support for returning values by @patrick91 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/12
• Add a return type to the benchmark fixture by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/13

New Contributors

• @patrick91 made their first contribution in https://github.com/CodSpeedHQ/pytest-codspeed/pull/12

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v2.0.0...v2.0.1

🎉 What's Changed

• Flame graph generation is now supported with Python 3.12+, allowing more granularity in the performance reports. More details can be found in the documentation.

• Each benchmark is now warmed up a single time before each run to reduce the flakiness and the side effects related to the order in which the benchmarks are run.

🚀 Migration Path

A heads-up, this is a breaking change and it might affect your current performance baseline a bit. But here's the exciting part - it's packed with new, cool features and promises improved result stability 🎉!

We recommend you create a standalone PR to update the pytest-codspeed package, acknowledge the potential regressions, and merge it to update the baseline on your default branch.

📚 Details

Fixes

• support benchmark.extra_info from pytest-benchmark by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/10

Internals

• add introspection benchmarks by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/9
• bump isort to 5.12 by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/11

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v1.2.2...v2.0.0

What's Changed

• feat: add library metadata in the profile output by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/5

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v1.2.1...v1.2.2

What's Changed

• fix: support kwargs with the benchmark fixture by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/4

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v1.2.0...v1.2.1

What's Changed

• fix: disable automatic garbage collection to increase stability by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/2

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v1.1.0...v1.2.0

What's Changed

• feat: allow running along with pytest-benchmark by @art049 in https://github.com/CodSpeedHQ/pytest-codspeed/pull/1

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v1.0.4...v1.1.0

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v0.0.3...v1.0.4

Full Changelog: https://github.com/CodSpeedHQ/pytest-codspeed/compare/v1.0.1...v1.0.2

<!-- generated by git-cliff -->