numbagg

Behavior Changes

• group_nanmean now promotes boolean and integer input to float64. It previously computed the mean in the input's integer dtype, so a group of [4, 5] gave 4 rather than 4.5 (#553)
• The moving functions now raise ValueError for a min_count greater than window, which no window can satisfy, instead of returning an all-NaN array (#817)

Bug Fixes

• Offset the covariance accumulators in nancorrmatrix and nancovmatrix (#759) and in the four moving matrix functions (#767), and accumulate in float64. On input far from zero the old form lost the result to cancellation: with standard-normal columns shifted by 1e8, move_covmatrix returned variances of -15 and move_corrmatrix values of 2.4. move_corrmatrix and move_exp_nancorrmatrix now also clip to [-1, 1], as np.corrcoef does
• Fix two label-dtype overflows in the grouped functions: widening a labels array no longer picks an unsigned dtype, which turned the -1 "no group" label into a large positive one, and the num_labels default no longer overflows on a label at the dtype's maximum (#749)
• Accept a tuple axis in the move_exp_* functions (#598)

Type Stubs

• Add stubs for move_corrmatrix, move_covmatrix, move_exp_nancorrmatrix and move_exp_nancovmatrix (#810)
• Annotate the matrix function returns in funcs.pyi (#812)
• Widen the nanquantile and nanmedian return types to np.ndarray, since **kwargs reaches the gufunc and an out= or dtype= argument decides the result dtype (#815)
• Accept a tuple axis in the moving-function stubs (#783)

Internal

• move_corrmatrix and move_covmatrix move out of MOVE_FUNCS into a new MOVE_MATRIX_FUNCS list (#768)
• Type checking moves from mypy to ty, and stubtest now checks the stubs against the runtime in CI (#472, #505, #748)
• Various dependency updates

Full Changelog: v0.9.4...v0.9.5

Bug Fixes

• Fix group_nanargmax and group_nanargmin to support N-dimensional labels (#485)
• Fix move_axes to raise AxisError for out-of-bounds axis indices instead of silently wrapping (#484)
• Fix move_corrmatrix and move_covmatrix to use pairwise statistics, producing correct results when NaN values occur at different positions across variables (#482)
• Fix allnan and anynan to initialize output arrays, preventing incorrect results on certain inputs (#481)
• Fix grouped functions with supports_bool=False (like group_nanvar) to raise clear TypeError for boolean input (#478)

Type Stub Fixes

• Add missing axis parameter to exponential moving window function type stubs (#483)
• Add default value to ddof parameter in nanvar/nanstd type stubs (#479)

Internal

• Fix flaky test failures in moving window functions (#450)
• Various dependency updates

Full Changelog: v0.9.3...v0.9.4

What's Changed

• Fix bfill/ffill to preserve integer dtypes by @max-sixty in #418
• Fix threading layer detection to respect numba configuration by @max-sixty in #413
• Update for Python 3.10 and CI to use correct python by @ianhi in #421

New Contributors

• @ianhi made their first contribution in #421

Full Changelog: v0.9.2...v0.9.3

0.9.2 is a maintenance release that fixes the CI/CD pipeline for publishing to PyPI.

CI/CD Fixes

• Fixed virtual environment setup in the release workflow (#411)
• Corrected GitHub Actions artifact versions (#412)

These changes ensure reliable package publishing but do not affect the functionality of numbagg itself.

0.9.0 implements our own "dynamic" compilation for grouped functions, in lieu of numba's (which currently doesn't work with parallel functions). This allows us to compile a function only for the types of the current function call's arguments, rather than all possible types allowed by the function. The speeds up the JIT compilation of a single function call by ~4x for the grouped functions.

0.8.2 reduces numerical instability in moving (aka rolling) functions with very short windows, as well as slightly improving their performance.

0.8.1 adds an experimental NUMBAGG_FASTMATH env var option (thanks @frazane) which increases performance in some routines at the cost of minor inaccuracy. Feel free to provide feedback in an issue if you find this helpful (or unhelpful!). There's also a change for numpy 2.0 compatibility (thanks @mathause), and some internal improvements.

0.8.0 includes nanmedian, a wrapper of nanquantile with one quantile of 0.5.

0.7.2 raises an error if values outside [0, 1] are passed to nanquantile

0.7.1 removes a stray print statement from the code. Thanks to @mathause for raising and fixing the issue.

0.7.0 adds a ddof argument to std & var aggregation & grouping functions. Internally, there are lots of new benchmarks, which are more clearly presented in the Readme, and added some initial property tests.

0.6.8 contains mostly internal changes — the initial benchmarking approach is expanded to all functions and displayed in the new Readme. The same framework is now used to test all functions. We also ensure the functions don't emit warnings when handling expected inputs in our tests.

0.6.7 removes the temporary patch for the int8 issues we experienced previously in grouping functions, replacing it with something more robust. Specifically, when there are a very large number of items in a group and labels has a very small dtype, labels is cast to a higher dtype.

Following closely on the heels of 0.6.5, 0.6.6 works around another rare but serious bug with int8 types. We now coerce all int8 label arrays to int16.

Many thanks to @dcherian for the report.

0.6.5 works around a rare but serious bug — when a labels array with int8 type is used in a group function, numbagg can return an incorrect result. The bug requires the array to be a specific size. The currently implemented solution is a workaround rather than an understanding of the underlying issue. Check out https://github.com/numbagg/numbagg/issues/211 for more details.

0.6.4 fixes a small bug — the value for the window argument for rolling methods couldn't be equal to the axis length.

Numbagg will now compile withmode="cpu" if it detects that it's being run in a ThreadPoolExecutor. Previously, the default mode="parallel" could cause numba to abort the python program within that context.

Note that running in a multi-process context retains mode="parallel", so the new behavior should only be slower in infrequent cases, such as a local dask multi-threaded executor.

I'm not completely confident this is the globally optimal solution, so this may evolve. https://github.com/numba/numba/issues/9288 has more context.

0.6.2 allows grouping functions to take a wider range of int types as labels. Thanks to @dcherian for the contribution.

0.6.1:

• Enables parallel mode in most functions. This radically improves performance in multi-core systems on multi-dimensional arrays (see benchmarks for details)
• Allows passing an array of alphas in the moving_exp functions, which lets us decay values by different amounts
• Improves nanquantile's compatibility with various axis values
• Extends benchmarks to different shapes, adds bottleneck as a comparison

• Add ffill & bfill, at ~2.7x pandas' performance
• Add standard moving window functions — move_corr, move_cov, move_std, move_sum, move_var, in addition to the existing move_mean. These have 3.5-20x pandas' performance.
• New benchmarks using pytest-benchmark. This includes a script which makes a nice output which we've added to the readme. It currently only covers the moving and moving_exp functions.

• Add a nanquantile function; approximately 4x faster than np.nanquantile when over 2 dimensions. It's slightly slower than np.quantile and pandas' .quantile
• Ensure we don't produce inf values for some exponential moving functions. Numerical values remain unchanged.

• Sets ddof=1 for std & var functions, mirroring the grouped & move_exp functions (but notably different from numpy)
• Adds a move_exp_nancount functions, for exponentially weighted moving counts
• Adds nancount as an alias for count

0.4.5 fixes an issue with our new PyPI release workflow. 0.4.1-4 were not published to PyPI

0.4.0 adds some more exponentially weighted functions:

• move_exp_nanstd
• move_exp_nanvar
• move_exp_nancorr
• move_exp_nancov

Because functions can now take more than one array, the signature of the moving exponential functions has changed slightly to require alpha to be a keyword argument. This is technically a breaking change, though most consumers will be passing alpha as a kwarg already (xarray included).

This release adds a min_weight parameter to the exponential moving functions, so that it's possible to output values if there's a sufficient number of recent valid values — similar to the min_count count parameter to the simple moving functions.

After a lengthy hiatus of development on numbagg, we're back with a big release:

• Lots of new grouping functions, in an attempt to be an engine for flox, a library from @dcherian & others. The functions include:

  • group_nancount
  • group_nanargmax, group_nanargmin
  • group_nanfirst, group_nanlast
  • group_nansum_of_squares
  • group_nanprod
  • group_nanall, group_nanany
  • group_nanvar
  • group_nanstd
  • group_nanmax, group_nanmin

• Lots of performance improvements to existing grouping functions

  • Initial benchmarking shows 2-5x the performance over pandas' equivalent functions (though mostly towards the lower end, and the benchmarks are not as robust as I'd like; feedback and verifications welcome).

• Large test coverage expansion of grouping functions

• Improvements to the exponentially weighted moving functions:

  • A new move_exp_nanvar function
  • Code simplification and modest performance improvements to existing functions
  • Benchmarks show 1-5x the performance of pandas' equivalent functions.

• A modest performance gain to existing moving functions.

• Internally, we've removed some of the original hacks that were initially required. Thanks to numbagg for supporting many of these natively!

The documentation needs a pass — the Readme could be reorganized, and the benchmarks could be more systematically measured and reported. It's possible that these large changes have introduced small bugs — particularly around edge cases, such as unfamiliar dtypes. That said, the main use cases are quite well-tested, and we have pandas & numpy to thank for excellent comparisons to test against.)

Please report any issues or questions. I (@max-sixty) am excited numbagg is back, and will gauge how much to add on the extent to which folks find it useful. And ofc thanks to @shoyer for writing the original library!