dask

Released on September 1, 2023

Bug Fixes

Remove support for np.int64 in keys ( dask#10483 ) crusaderky

Fix _partitions dtype in meta for shuffling ( dask#10462 ) Hendrik Makait

Don’t use exception hooks to shorten tracebacks ( dask#10456 ) crusaderky

Documentation

Add p2p shuffle option to DataFrame docs ( dask#10477 ) Patrick Hoefler

Maintenance

Skip failing tests for pandas=2.1.0 ( dask#10488 ) Patrick Hoefler

Update tests for pandas=2.1.0 ( dask#10439 ) Patrick Hoefler

Enable pytest-timeout ( dask#10482 ) crusaderky

Bump actions/checkout from 3.5.3 to 3.6.0 ( dask#10470 )

Released on August 18, 2023

Enhancements

Adding support for cgroup v2 to cpu_count ( dask#10419 ) Johan Olsson

Support multi-column groupby with sort=True and split_out>1 ( dask#10425 ) Richard (Rick) Zamora

Add DataFrame.enforce_runtime_divisions method ( dask#10404 ) Richard (Rick) Zamora

Enable file mode="x" with a single_file=True for Dask DataFrame to_csv ( dask#10443 ) Genevieve Buckley

Bug Fixes

Fix ValueError when running to_csv in append mode with single_file as True ( dask#10441 ) Ben

Maintenance

Add default types_mapper to from_pyarrow_table_dispatch for pandas ( dask#10446 ) Richard (Rick) Zamora

Released on August 4, 2023

Enhancements

Fix for make_timeseries performance regression ( dask#10428 ) Irina Truong

Documentation

Add distributed.print to debugging docs ( dask#10435 ) James Bourbeau

Documenting compatibility of NumPy functions with Dask functions ( dask#9941 ) Chiara Marmo

Maintenance

Use SPDX in license metadata ( dask#10437 ) John A Kirkham

Require dask[array] in dask[dataframe] ( dask#10357 ) John A Kirkham

Update gpuCI RAPIDS_VER to 23.10 ( dask#10427 )

Simplify compatibility code ( dask#10426 ) Hendrik Makait

Fix compatibility variable naming ( dask#10424 ) Hendrik Makait

Fix a few errors with upstream pandas and pyarrow ( dask#10412 ) Irina Truong

Released on July 20, 2023

Note

This release updates Dask DataFrame to automatically convert text data using object data types to string[pyarrow] if pandas>=2 and pyarrow>=12 are installed.

This should result in significantly reduced memory consumption and increased computation performance in many workflows that deal with text data.

You can disable this change by setting the dataframe.convert-string configuration value to False with

dask . config . set ({ "dataframe.convert-string" : False })

Enhancements

Convert to pyarrow strings if proper dependencies are installed ( dask#10400 ) James Bourbeau

Avoid repartition before shuffle for p2p ( dask#10421 ) Patrick Hoefler

API to generate random Dask DataFrames ( dask#10392 ) Irina Truong

Speed up dask.bag.Bag.random_sample ( dask#10356 ) crusaderky

Raise helpful ValueError for invalid time units ( dask#10408 ) Nat Tabris

Make repartition a no-op when divisions match (divisions provided as a list) ( dask#10395 ) Nicolas Grandemange

Bug Fixes

Use dataframe.convert-string in read_parquet token ( dask#10411 ) James Bourbeau

Category dtype is lost when concatenating MultiIndex ( dask#10407 ) Irina Truong

Fix FutureWarning: The provided callable... ( dask#10405 ) Irina Truong

Enable non-categorical hive-partition columns in read_parquet ( dask#10353 ) Richard (Rick) Zamora

concat ignoring DataFrame withouth columns ( dask#10359 ) Patrick Hoefler

Released on July 7, 2023

Enhancements

Catch exceptions when attempting to load CLI entry points ( dask#10380 ) Jacob Tomlinson

Bug Fixes

Fix typo in _clean_ipython_traceback ( dask#10385 ) Alexander Clausen

Ensure that df is immutable after from_pandas ( dask#10383 ) Patrick Hoefler

Warn consistently for inplace in Series.rename ( dask#10313 ) Patrick Hoefler

Documentation

Add clarification about output shape and reshaping in rechunk documentation ( dask#10377 ) Swayam Patil

Maintenance

Simplify astype implementation ( dask#10393 ) Patrick Hoefler

Fix test_first_and_last to accommodate deprecated last ( dask#10373 ) James Bourbeau

Add level to create_merge_tree ( dask#10391 ) Patrick Hoefler

Do not derive from scipy.stats.chisquare docstring ( dask#10382 ) Doug Davis

Released on June 26, 2023

Enhancements

Remove no longer supported clip_lower and clip_upper ( dask#10371 ) Patrick Hoefler

Support DataFrame.set_index(..., sort=False) ( dask#10342 ) Miles

Cleanup remote tracebacks ( dask#10354 ) Irina Truong

Add dispatching mechanisms for pyarrow.Table conversion ( dask#10312 ) Richard (Rick) Zamora

Choose P2P even if fusion is enabled ( dask#10344 ) Hendrik Makait

Validate that rechunking is possible earlier in graph generation ( dask#10336 ) Hendrik Makait

Bug Fixes

Fix issue with header passed to read_csv ( dask#10355 ) GALI PREM SAGAR

Respect dropna and observed in GroupBy.var and GroupBy.std ( dask#10350 ) Patrick Hoefler

Fix H5FD_lock error when writing to hdf with distributed client ( dask#10309 ) Irina Truong

Fix for total_mem_usage of bag.map() ( dask#10341 ) Irina Truong

Deprecations

Deprecate DataFrame.fillna / Series.fillna with method ( dask#10349 ) Irina Truong

Deprecate DataFrame.first and Series.first ( dask#10352 ) Irina Truong

Maintenance

Deprecate numpy.compat ( dask#10370 ) Irina Truong

Fix annotations and spans leaking between threads ( dask#10367 ) Irina Truong

Use general kwargs in pyarrow_table_dispatch functions ( dask#10364 ) Richard (Rick) Zamora

Remove unnecessary try / except in isna ( dask#10363 ) Patrick Hoefler

mypy support for numpy 1.25 ( dask#10362 ) crusaderky

Bump actions/checkout from 3.5.2 to 3.5.3 ( dask#10348 )

Restore numba in upstream build ( dask#10330 ) James Bourbeau

Update nightly wheel index for pandas / numpy / scipy ( dask#10346 ) Matthew Roeschke

Add rechunk config values to yaml ( dask#10343 ) Hendrik Makait

Released on June 9, 2023

Enhancements

Add missing not in predicate support to read_parquet ( dask#10320 ) Richard (Rick) Zamora

Bug Fixes

Fix for incorrect value_counts ( dask#10323 ) Irina Truong

Update empty describe top and freq values ( dask#10319 ) James Bourbeau

Documentation

Fix hetzner typo ( dask#10332 ) Sarah Charlotte Johnson

Maintenance

Test with numba and sparse on Python 3.11 ( dask#10329 ) Thomas Grainger

Remove numpy.find_common_type warning ignore ( dask#10311 ) James Bourbeau

Update gpuCI RAPIDS_VER to 23.08 ( dask#10310 )

Released on May 26, 2023

Note

This release drops support for Python 3.8. As of this release Dask supports Python 3.9, 3.10, and 3.11. See this community issue for more details.

Enhancements

Drop Python 3.8 support ( dask#10295 ) Thomas Grainger

Change Dask Bag partitioning scheme to improve cluster saturation ( dask#10294 ) Jacob Tomlinson

Generalize dd.to_datetime for GPU-backed collections, introduce get_meta_library utility ( dask#9881 ) Charles Blackmon-Luca

Add na_action to DataFrame.map ( dask#10305 ) Patrick Hoefler

Raise TypeError in DataFrame.nsmallest and DataFrame.nlargest when columns is not given ( dask#10301 ) Patrick Hoefler

Improve sizeof for pd.MultiIndex ( dask#10230 ) Patrick Hoefler

Support duplicated columns in a bunch of DataFrame methods ( dask#10261 ) Patrick Hoefler

Add numeric_only support to DataFrame.idxmin and DataFrame.idxmax ( dask#10253 ) Patrick Hoefler

Implement numeric_only support for DataFrame.quantile ( dask#10259 ) Patrick Hoefler

Add support for numeric_only=False in DataFrame.std ( dask#10251 ) Patrick Hoefler

Implement numeric_only=False for GroupBy.cumprod and GroupBy.cumsum ( dask#10262 ) Patrick Hoefler

Implement numeric_only for skew and kurtosis ( dask#10258 ) Patrick Hoefler

mask and where should accept a callable ( dask#10289 ) Irina Truong

Fix conversion from Categorical to pa.dictionary in read_parquet ( dask#10285 ) Patrick Hoefler

Bug Fixes

Spurious config on nested annotations ( dask#10318 ) crusaderky

Fix rechunking behavior for dimensions with known and unknown chunk sizes ( dask#10157 ) Hendrik Makait

Enable drop to support mismatched partitions ( dask#10300 ) James Bourbeau

Fix divisions construction for to_timestamp ( dask#10304 ) Patrick Hoefler

pandas ExtensionDtype raising in Series reduction operations ( dask#10149 ) Patrick Hoefler

Fix regression in da.random interface ( dask#10247 ) Eray Aslan

da.coarsen doesn’t trim an empty chunk in meta ( dask#10281 ) Irina Truong

Fix dtype inference for engine="pyarrow" in read_csv ( dask#10280 ) Patrick Hoefler

Documentation

Add meta_from_array to API docs ( dask#10306 ) Ruth Comer

Update Coiled links ( dask#10296 ) Sarah Charlotte Johnson

Add docs for demo day ( dask#10288 ) Matthew Rocklin

Maintenance

Explicitly install anaconda-client from conda-forge when uploading conda nightlies ( dask#10316 ) Charles Blackmon-Luca

Configure isort to add from future import annotations ( dask#10314 ) Thomas Grainger

Avoid pandas Series.getitem deprecation in tests ( dask#10308 ) James Bourbeau

Ignore numpy.find_common_type warning from pandas ( dask#10307 ) James Bourbeau

Add test to check that DataFrame.setitem does not modify df inplace ( dask#10223 ) Patrick Hoefler

Clean up default value of dropna in value_counts ( dask#10299 ) Patrick Hoefler

Add pytest-cov to test extra ( dask#10271 ) James Bourbeau

Released on May 12, 2023

Enhancements

Implement numeric_only=False for GroupBy.corr and GroupBy.cov ( dask#10264 ) Patrick Hoefler

Add support for numeric_only=False in DataFrame.var ( dask#10250 ) Patrick Hoefler

Add numeric_only support to DataFrame.mode ( dask#10257 ) Patrick Hoefler

Add DataFrame.map to dask.DataFrame API ( dask#10246 ) Patrick Hoefler

Adjust for DataFrame.applymap deprecation and all NA concat behaviour change ( dask#10245 ) Patrick Hoefler

Enable numeric_only=False for DataFrame.count ( dask#10234 ) Patrick Hoefler

Disallow array input in mask/where ( dask#10163 ) Irina Truong

Support numeric_only=True in GroupBy.corr and GroupBy.cov ( dask#10227 ) Patrick Hoefler

Add numeric_only support to GroupBy.median ( dask#10236 ) Patrick Hoefler

Support mimesis=9 in dask.datasets ( dask#10241 ) James Bourbeau

Add numeric_only support to min , max and prod ( dask#10219 ) Patrick Hoefler

Add numeric_only=True support for GroupBy.cumsum and GroupBy.cumprod ( dask#10224 ) Patrick Hoefler

Add helper to unpack numeric_only keyword ( dask#10228 ) Patrick Hoefler

Bug Fixes

Fix clone + from_array failure ( dask#10211 ) crusaderky

Fix dataframe reductions for ea dtypes ( dask#10150 ) Patrick Hoefler

Avoid scalar conversion deprecation warning in numpy=1.25 ( dask#10248 ) James Bourbeau

Make sure transform output has the same index as input ( dask#10184 ) Irina Truong

Fix corr and cov on a single-row partition ( dask#9756 ) Irina Truong

Fix test_groupby_numeric_only_supported and test_groupby_aggregate_categorical_observed upstream errors ( dask#10243 ) Irina Truong

Documentation

Clean up futures docs ( dask#10266 ) Matthew Rocklin

Add Index API reference ( dask#10263 ) hotpotato

Maintenance

Warn when meta is passed to apply ( dask#10256 ) Patrick Hoefler

Remove imageio version restriction in CI ( dask#10260 ) Patrick Hoefler

Remove unused DataFrame variance methods ( dask#10252 ) Patrick Hoefler

Un- xfail test_categories with pyarrow strings and pyarrow>=12 ( dask#10244 ) Irina Truong

Bump gpuCI PYTHON_VER 3.8->3.9 ( dask#10233 ) Charles Blackmon-Luca

Released on April 28, 2023

Enhancements

Implement numeric_only support for DataFrame.sum ( dask#10194 ) Patrick Hoefler

Add support for numeric_only=True in GroupBy operations ( dask#10222 ) Patrick Hoefler

Avoid deep copy in DataFrame.setitem for pandas 1.4 and up ( dask#10221 ) Patrick Hoefler

Avoid calling Series.apply with _meta_nonempty ( dask#10212 ) Patrick Hoefler

Unpin sqlalchemy and fix compatibility issues ( dask#10140 ) Patrick Hoefler

Bug Fixes

Partially revert default client discovery ( dask#10225 ) Florian Jetter

Support arrow dtypes in Index meta creation ( dask#10170 ) Patrick Hoefler

Repartitioning raises with extension dtype when truncating floats ( dask#10169 ) Patrick Hoefler

Adjust empty Index from fastparquet to object dtype ( dask#10179 ) Patrick Hoefler

Documentation

Update Kubernetes docs ( dask#10232 ) Jacob Tomlinson

Add DataFrame.reduction to API docs ( dask#10229 ) James Bourbeau

Add DataFrame.persist to docs and fix links ( dask#10231 ) Patrick Hoefler

Add documentation for GroupBy.transform ( dask#10185 ) Irina Truong

Fix formatting in random number generation docs ( dask#10189 ) Eray Aslan

Maintenance

Pin imageio to <2.28 ( dask#10216 ) Patrick Hoefler

Add note about importlib_metadata backport ( dask#10207 ) James Bourbeau

Add xarray back to Python 3.11 CI builds ( dask#10200 ) James Bourbeau

Add mindeps build with all optional dependencies ( dask#10161 ) Charles Blackmon-Luca

Provide proper like value for array_safe in percentiles_summary ( dask#10156 ) Charles Blackmon-Luca

Avoid re-opening hdf file multiple times in read_hdf ( dask#10205 ) Thomas Grainger

Add merge tests on nullable columns ( dask#10071 ) Charles Blackmon-Luca

Fix coverage configuration ( dask#10203 ) Thomas Grainger

Remove is_period_dtype and is_sparse_dtype ( dask#10197 ) Patrick Hoefler

Bump actions/checkout from 3.5.0 to 3.5.2 ( dask#10201 )

Avoid deprecated is_categorical_dtype from pandas ( dask#10180 ) Patrick Hoefler

Adjust for deprecated is_interval_dtype and is_datetime64tz_dtype ( dask#10188 ) Patrick Hoefler

Released on April 14, 2023

Enhancements

Override old default values in update_defaults ( dask#10159 ) Gabe Joseph

Add a CLI command to list and get a value from dask config ( dask#9936 ) Irina Truong

Handle string-based engine argument to read_json ( dask#9947 ) Richard (Rick) Zamora

Avoid deprecated GroupBy.dtypes ( dask#10111 ) Irina Truong

Bug Fixes

Revert grouper -related changes ( dask#10182 ) Irina Truong

GroupBy.cov raising for non-numeric grouping column ( dask#10171 ) Patrick Hoefler

Updates for Index supporting numpy numeric dtypes ( dask#10154 ) Irina Truong

Preserve dtype for partitioning columns when read with pyarrow ( dask#10115 ) Patrick Hoefler

Fix annotations for to_hdf ( dask#10123 ) Hendrik Makait

Handle None column name when checking if columns are all numeric ( dask#10128 ) Lawrence Mitchell

Fix valid_divisions when passed a tuple ( dask#10126 ) Brian Phillips

Maintain annotations in DataFrame.categorize ( dask#10120 ) Hendrik Makait

Fix handling of missing min/max parquet statistics during filtering ( dask#10042 ) Richard (Rick) Zamora

Deprecations

Deprecate use_nullable_dtypes= and add dtype_backend= ( dask#10076 ) Irina Truong

Deprecate convert_dtype in Series.apply ( dask#10133 ) Irina Truong

Documentation

Document Generator based random number generation ( dask#10134 ) Eray Aslan

Maintenance

Update dataframe.convert_string to dataframe.convert-string ( dask#10191 ) Irina Truong

Add python-cityhash to CI environments ( dask#10190 ) Charles Blackmon-Luca

Temporarily pin scikit-image to fix Windows CI ( dask#10186 ) Patrick Hoefler

Handle pandas deprecation warnings for to_pydatetime and apply ( dask#10168 ) Patrick Hoefler

Drop bokeh<3 restriction ( dask#10177 ) James Bourbeau

Fix failing tests under copy-on-write ( dask#10173 ) Patrick Hoefler

Allow pyarrow CI to fail ( dask#10176 ) James Bourbeau

Switch to Generator for random number generation in dask.array ( dask#10003 ) Eray Aslan

Bump peter-evans/create-pull-request from 4 to 5 ( dask#10166 )

Fix flaky modf operation in test_arithmetic ( dask#10162 ) Irina Truong

Temporarily remove xarray from CI with pandas 2.0 ( dask#10153 ) James Bourbeau

Fix update_graph counting logic in test_default_scheduler_on_worker ( dask#10145 ) James Bourbeau

Fix documentation build with pandas 2.0 ( dask#10138 ) James Bourbeau

Remove dask/gpu from gpuCI update reviewers ( dask#10135 ) Charles Blackmon-Luca

Update gpuCI RAPIDS_VER to 23.06 ( dask#10129 )

Bump actions/stale from 6 to 8 ( dask#10121 )

Use declarative setuptools ( dask#10102 ) Thomas Grainger

Relax assert_eq checks on Scalar -like objects ( dask#10125 ) Matthew Rocklin

Upgrade readthedocs config to ubuntu 22.04 and Python 3.11 ( dask#10124 ) Thomas Grainger

Bump actions/checkout from 3.4.0 to 3.5.0 ( dask#10122 )

Fix test_null_partition_pyarrow in pyarrow CI build ( dask#10116 ) Irina Truong

Drop distributed pack ( dask#9988 ) Florian Jetter

Make dask.compatibility private ( dask#10114 ) Jacob Tomlinson

Released on March 24, 2023

Enhancements

Deprecate observed=False for groupby with categoricals ( dask#10095 ) Irina Truong

Deprecate axis= for some groupby operations ( dask#10094 ) James Bourbeau

The axis keyword in DataFrame.rolling/Series.rolling is deprecated ( dask#10110 ) Irina Truong

DataFrame._data deprecation in pandas ( dask#10081 ) Irina Truong

Use importlib_metadata backport to avoid CLI UserWarning ( dask#10070 ) Thomas Grainger

Port option parsing logic from dask.dataframe.read_parquet to to_parquet ( dask#9981 ) Anton Loukianov

Bug Fixes

Avoid using dd.shuffle in groupby-apply ( dask#10043 ) Richard (Rick) Zamora

Enable null hive partitions with pyarrow parquet engine ( dask#10007 ) Richard (Rick) Zamora

Support unknown shapes in *_like functions ( dask#10064 ) Doug Davis

Documentation

Add to_backend methods to API docs ( dask#10093 ) Lawrence Mitchell

Remove broken gpuCI link in developer docs ( dask#10065 ) Charles Blackmon-Luca

Maintenance

Configure readthedocs sphinx warnings as errors ( dask#10104 ) Thomas Grainger

Un- xfail test_division_or_partition with pyarrow strings active ( dask#10108 ) Irina Truong

Un- xfail test_different_columns_are_allowed with pyarrow strings active ( dask#10109 ) Irina Truong

Restore Entrypoints compatibility ( dask#10113 ) Jacob Tomlinson

Un- xfail test_to_dataframe_optimize_graph with pyarrow strings active ( dask#10087 ) Irina Truong

Only run test_development_guidelines_matches_ci on editable install ( dask#10106 ) Charles Blackmon-Luca

Un- xfail test_dataframe_cull_key_dependencies_materialized with pyarrow strings active ( dask#10088 ) Irina Truong

Install mimesis in CI environments ( dask#10105 ) Charles Blackmon-Luca

Fix for no module named ipykernel ( dask#10101 ) Irina Truong

Fix docs builds by installing ipykernel ( dask#10103 ) Thomas Grainger

Allow pyarrow build to continue on failures ( dask#10097 ) James Bourbeau

Bump actions/checkout from 3.3.0 to 3.4.0 ( dask#10096 )

Fix test_set_index_on_empty with pyarrow strings active ( dask#10054 ) Irina Truong

Un- xfail pyarrow pickling tests ( dask#10082 ) James Bourbeau

CI environment file cleanup ( dask#10078 ) James Bourbeau

Un- xfail more pyarrow tests ( dask#10066 ) Irina Truong

Temporarily skip pyarrow_compat tests with p`andas 2.0 ( dask#10063 ) James Bourbeau

Fix test_melt with pyarrow strings active ( dask#10052 ) Irina Truong

Fix test_str_accessor with pyarrow strings active ( dask#10048 ) James Bourbeau

Fix test_better_errors_object_reductions with pyarrow strings active ( dask#10051 ) James Bourbeau

Fix test_loc_with_non_boolean_series with pyarrow strings active ( dask#10046 ) James Bourbeau

Fix test_values with pyarrow strings active ( dask#10050 ) James Bourbeau

Temporarily xfail test_upstream_packages_installed ( dask#10047 ) James Bourbeau

Released on March 10, 2023

Enhancements

Support pyarrow strings in MultiIndex ( dask#10040 ) Irina Truong

Improved support for pyarrow strings ( dask#10000 ) Irina Truong

Fix flaky RuntimeWarning during array reductions ( dask#10030 ) James Bourbeau

Extend complete extras ( dask#10023 ) James Bourbeau

Raise an error with dataframe.convert-string=True and pandas<2.0 ( dask#10033 ) Irina Truong

Rename shuffle/rechunk config option/kwarg to method ( dask#10013 ) James Bourbeau

Add initial support for converting pandas extension dtypes to arrays ( dask#10018 ) James Bourbeau

Remove randomgen support ( dask#9987 ) Eray Aslan

Bug Fixes

Skip rechunk when rechunking to the same chunks with unknown sizes ( dask#10027 ) Hendrik Makait

Custom utility to convert parquet filters to pyarrow expression ( dask#9885 ) Richard (Rick) Zamora

Consider numpy scalars and 0d arrays as scalars when padding ( dask#9653 ) Justus Magin

Fix parquet overwrite behavior after an adaptive read_parquet operation ( dask#10002 ) Richard (Rick) Zamora

Documentation

Add and update docs for Data Transfer section ( dask#10022 ) Miles

Maintenance

Remove stale hive-partitioning code from pyarrow parquet engine ( dask#10039 ) Richard (Rick) Zamora

Increase minimum supported pyarrow to 7.0 ( dask#10024 ) James Bourbeau

Revert “Prepare drop packunpack ( dask#9994 ) ( dask#10037 ) Florian Jetter

Have codecov wait for more builds before reporting ( dask#10031 ) James Bourbeau

Prepare drop packunpack ( dask#9994 ) Florian Jetter

Add CI job with pyarrow strings turned on ( dask#10017 ) James Bourbeau

Fix test_groupby_dropna_with_agg for pandas 2.0 ( dask#10001 ) Irina Truong

Fix test_pickle_roundtrip for pandas 2.0 ( dask#10011 ) James Bourbeau

Released on March 1, 2023

Bug Fixes

Bag must not pick p2p as shuffle default ( dask#10005 ) Florian Jetter

Documentation

Minor follow-up to P2P by default ( dask#10008 ) James Bourbeau

Maintenance

Add minimum version to optional jinja2 dependency ( dask#9999 ) Charles Blackmon-Luca

Released on February 24, 2023

Note

This release changes the default DataFrame shuffle algorithm to p2p to improve stability and performance. Learn more here and please provide any feedback on this discussion .

If you encounter issues with this new algorithm, please see the documentation for more information, and how to switch back to the old mode.

Enhancements

Enable P2P shuffling by default ( dask#9991 ) Florian Jetter

P2P rechunking ( dask#9939 ) Hendrik Makait

Efficient dataframe.convert-string support for read_parquet ( dask#9979 ) Irina Truong

Allow p2p shuffle kwarg for DataFrame merges ( dask#9900 ) Florian Jetter

Change split_row_groups default to “infer” ( dask#9637 ) Richard (Rick) Zamora

Add option for converting string data to use pyarrow strings ( dask#9926 ) James Bourbeau

Add support for multi-column sort_values ( dask#8263 ) Charles Blackmon-Luca

Generator based random-number generation indask.array ( dask#9038 ) Eray Aslan

Support numeric_only for simple groupby aggregations for pandas 2.0 compatibility ( dask#9889 ) Irina Truong

Bug Fixes

Fix profilers plot not being aligned to context manager enter time ( dask#9739 ) David Hoese

Relax dask.dataframe assert_eq type checks ( dask#9989 ) Matthew Rocklin

Restore describe compatibility for pandas 2.0 ( dask#9982 ) James Bourbeau

Documentation

Improving deploying Dask docs ( dask#9912 ) Sarah Charlotte Johnson

More docs for DataFrame.partitions ( dask#9976 ) Tom Augspurger

Update docs with more information on default Delayed scheduler ( dask#9903 ) Guillaume Eynard-Bontemps

Deployment Considerations documentation ( dask#9933 ) Gabe Joseph

Maintenance

Temporarily rerun flaky tests ( dask#9983 ) James Bourbeau

Update parsing of FULL_RAPIDS_VER/FULL_UCX_PY_VER ( dask#9990 ) Charles Blackmon-Luca

Increase minimum supported versions to pandas=1.3 and numpy=1.21 ( dask#9950 ) James Bourbeau

Fix std to work with numeric_only for pandas 2.0 ( dask#9960 ) Irina Truong

Temporarily xfail test_roundtrip_partitioned_pyarrow_dataset ( dask#9977 ) James Bourbeau

Fix copy on write failure in test_idxmaxmin ( dask#9944 ) Patrick Hoefler

Bump pre-commit versions ( dask#9955 ) crusaderky

Fix test_groupby_unaligned_index for pandas 2.0 ( dask#9963 ) Irina Truong

Un- xfail test_set_index_overlap_2 for pandas 2.0 ( dask#9959 ) James Bourbeau

Fix test_merge_by_index_patterns for pandas 2.0 ( dask#9930 ) Irina Truong

Bump jacobtomlinson/gha-find-replace from 2 to 3 ( dask#9953 ) James Bourbeau

Fix test_rolling_agg_aggregate for pandas 2.0 compatibility ( dask#9948 ) Irina Truong

Bump black to 23.1.0 ( dask#9956 ) crusaderky

Run GPU tests on python 3.8 & 3.10 ( dask#9940 ) Charles Blackmon-Luca

Fix test_to_timestamp for pandas 2.0 ( dask#9932 ) Irina Truong

Fix an error with groupby value_counts for pandas 2.0 compatibility ( dask#9928 ) Irina Truong

Config converter: replace all dashes with underscores ( dask#9945 ) Jacob Tomlinson

CI: use nightly wheel to install pyarrow in upstream test build ( dask#9873 ) Joris Van den Bossche

Released on February 10, 2023

Enhancements

Update numeric_only default in quantile for pandas 2.0 ( dask#9854 ) Irina Truong

Make repartition a no-op when divisions match ( dask#9924 ) James Bourbeau

Update datetime_is_numeric behavior in describe for pandas 2.0 ( dask#9868 ) Irina Truong

Update value_counts to return correct name in pandas 2.0 ( dask#9919 ) Irina Truong

Support new axis=None behavior in pandas 2.0 for certain reductions ( dask#9867 ) James Bourbeau

Filter out all-nan RuntimeWarning at the chunk level for nanmin and nanmax ( dask#9916 ) Julia Signell

Fix numeric meta_nonempty index creation for pandas 2.0 ( dask#9908 ) James Bourbeau

Fix DataFrame.info() tests for pandas 2.0 ( dask#9909 ) James Bourbeau

Bug Fixes

Fix GroupBy.value_counts handling for multiple groupby columns ( dask#9905 ) Charles Blackmon-Luca

Documentation

Fix some outdated information/typos in development guide ( dask#9893 ) Patrick Hoefler

Add note about keep=False in drop_duplicates docstring ( dask#9887 ) Jayesh Manani

Add meta details to dask Array ( dask#9886 ) Jayesh Manani

Clarify task stream showing more rows than threads ( dask#9906 ) Gabe Joseph

Maintenance

Fix test_numeric_column_names for pandas 2.0 ( dask#9937 ) Irina Truong

Fix dask/dataframe/tests/test_utils_dataframe.py tests for pandas 2.0 ( dask#9788 ) James Bourbeau

Replace index.is_numeric with is_any_real_numeric_dtype for pandas 2.0 compatibility ( dask#9918 ) Irina Truong

Avoid pd.core import in dask utils ( dask#9907 ) Matthew Roeschke

Use label for upstream build on pull requests ( dask#9910 ) James Bourbeau

Broaden exception catching for sqlalchemy.exc.RemovedIn20Warning ( dask#9904 ) James Bourbeau

Temporarily restrict sqlalchemy < 2 in CI ( dask#9897 ) James Bourbeau

Update isort version to 5.12.0 ( dask#9895 ) Lawrence Mitchell

Remove unused skiprows variable in read_csv ( dask#9892 ) Patrick Hoefler

Released on January 27, 2023

Enhancements

Add to_backend method to Array and _Frame ( dask#9758 ) Richard (Rick) Zamora

Small fix for timestamp index divisions in pandas 2.0 ( dask#9872 ) Irina Truong

Add numeric_only to DataFrame.cov and DataFrame.corr ( dask#9787 ) James Bourbeau

Fixes related to group_keys default change in pandas 2.0 ( dask#9855 ) Irina Truong

infer_datetime_format compatibility for pandas 2.0 ( dask#9783 ) James Bourbeau

Bug Fixes

Fix serialization bug in BroadcastJoinLayer ( dask#9871 ) Richard (Rick) Zamora

Satisfy broadcast argument in DataFrame.merge ( dask#9852 ) Richard (Rick) Zamora

Fix pyarrow parquet columns statistics computation ( dask#9772 ) aywandji

Documentation

Fix “duplicate explicit target name” docs warning ( dask#9863 ) Chiara Marmo

Fix code formatting issue in “Defining a new collection backend” docs ( dask#9864 ) Chiara Marmo

Update dashboard documentation for memory plot ( dask#9768 ) Jayesh Manani

Add docs section about no-worker tasks ( dask#9839 ) Florian Jetter

Maintenance

Additional updates for detecting a distributed scheduler ( dask#9890 ) James Bourbeau

Update gpuCI RAPIDS_VER to 23.04 ( dask#9876 )

Reverse precedence between collection and distributed default ( dask#9869 ) Florian Jetter

Update xarray-contrib/issue-from-pytest-log to version 1.2.6 ( dask#9865 ) James Bourbeau

Dont require dask config shuffle default ( dask#9826 ) Florian Jetter

Un- xfail datetime64 Parquet roundtripping tests for new fastparquet ( dask#9811 ) James Bourbeau

Add option to manually run upstream CI build ( dask#9853 ) James Bourbeau

Use custom timeout in CI builds ( dask#9844 ) James Bourbeau

Remove kwargs from make_blockwise_graph ( dask#9838 ) Florian Jetter

Ignore warnings on persist call in test_setitem_extended_API_2d_mask ( dask#9843 ) Charles Blackmon-Luca

Fix running S3 tests locally ( dask#9833 ) James Bourbeau

Released on January 13, 2023

Enhancements

Use distributed default clients even if no config is set ( dask#9808 ) Florian Jetter

Implement ma.where and ma.nonzero ( dask#9760 ) Erik Holmgren

Update zarr store creation functions ( dask#9790 ) Ryan Abernathey

iteritems compatibility for pandas 2.0 ( dask#9785 ) James Bourbeau

Accurate sizeof for pandas string[python] dtype ( dask#9781 ) crusaderky

Deflate sizeof() of duplicate references to pandas object types ( dask#9776 ) crusaderky

GroupBy.getitem compatibility for pandas 2.0 ( dask#9779 ) James Bourbeau

append compatibility for pandas 2.0 ( dask#9750 ) James Bourbeau

get_dummies compatibility for pandas 2.0 ( dask#9752 ) James Bourbeau

is_monotonic compatibility for pandas 2.0 ( dask#9751 ) James Bourbeau

numpy=1.24 compatability ( dask#9777 ) James Bourbeau

Documentation

Remove duplicated encoding kwarg in docstring for to_json ( dask#9796 ) Sultan Orazbayev

Mention SubprocessCluster in LocalCluster documentation ( dask#9784 ) Hendrik Makait

Move Prometheus docs to dask/distributed ( dask#9761 ) crusaderky

Maintenance

Temporarily ignore RuntimeWarning in test_setitem_extended_API_2d_mask ( dask#9828 ) James Bourbeau

Fix flaky test_threaded.py::test_interrupt ( dask#9827 ) Hendrik Makait

Update xarray-contrib/issue-from-pytest-log in upstream report ( dask#9822 ) James Bourbeau

pip install dask on gpuCI builds ( dask#9816 ) Charles Blackmon-Luca

Bump actions/checkout from 3.2.0 to 3.3.0 ( dask#9815 )

Resolve sqlalchemy import failures in mindeps testing ( dask#9809 ) Charles Blackmon-Luca

Ignore sqlalchemy.exc.RemovedIn20Warning ( dask#9801 ) Thomas Grainger

xfail datetime64 Parquet roundtripping tests for pandas 2.0 ( dask#9786 ) James Bourbeau

Remove sqlachemy 1.3 compatibility ( dask#9695 ) McToel

Reduce size of expected DoK sparse matrix ( dask#9775 ) Elliott Sales de Andrade

Remove executable flag from dask/dataframe/io/orc/utils.py ( dask#9774 ) Elliott Sales de Andrade

Released on December 16, 2022

Enhancements

Support dtype_backend="pandas|pyarrow" configuration ( dask#9719 ) James Bourbeau

Support cupy.ndarray to cudf.DataFrame dispatching in dask.dataframe ( dask#9579 ) Richard (Rick) Zamora

Make filesystem-backend configurable in read_parquet ( dask#9699 ) Richard (Rick) Zamora

Serialize all pyarrow extension arrays efficiently ( dask#9740 ) James Bourbeau

Bug Fixes

Fix bug when repartitioning with tz -aware datetime index ( dask#9741 ) James Bourbeau

Partial functions in aggs may have arguments ( dask#9724 ) Irina Truong

Add support for simple operation with pyarrow -backed extension dtypes ( dask#9717 ) James Bourbeau

Rename columns correctly in case of SeriesGroupby ( dask#9716 ) Lawrence Mitchell

Documentation

Fix url link typo in collection backend doc ( dask#9748 ) Shawn

Update Prometheus docs ( dask#9696 ) Hendrik Makait

Maintenance

Add zarr to Python 3.11 CI environment ( dask#9771 ) James Bourbeau

Add support for Python 3.11 ( dask#9708 ) Thomas Grainger

Bump actions/checkout from 3.1.0 to 3.2.0 ( dask#9753 )

Avoid np.bool8 deprecation warning ( dask#9737 ) James Bourbeau

Make sure dev packages aren’t overwritten in upstream CI build ( dask#9731 ) James Bourbeau

Avoid adding data.h5 and mydask.html files during tests ( dask#9726 ) Thomas Grainger

Released on December 2, 2022

Enhancements

Remove statistics-based set_index logic from read_parquet ( dask#9661 ) Richard (Rick) Zamora

Add support for use_nullable_dtypes to dd.read_parquet ( dask#9617 ) Ian Rose

Fix map_overlap in order to accept pandas arguments ( dask#9571 ) Fabien Aulaire

Fix pandas 1.5+ FutureWarning in .str.split(..., expand=True) ( dask#9704 ) Jacob Hayes

Enable column projection for groupby slicing ( dask#9667 ) Richard (Rick) Zamora

Support duplicate column cum-functions ( dask#9685 ) Ben

Improve error message for failed backend dispatch call ( dask#9677 ) Richard (Rick) Zamora

Bug Fixes

Revise meta creation in arrow parquet engine ( dask#9672 ) Richard (Rick) Zamora

Fix da.fft.fft for array-like inputs ( dask#9688 ) James Bourbeau

Fix groupby -aggregation when grouping on an index by name ( dask#9646 ) Richard (Rick) Zamora

Maintenance

Avoid PytestReturnNotNoneWarning in test_inheriting_class ( dask#9707 ) Thomas Grainger

Fix flaky test_dataframe_aggregations_multilevel ( dask#9701 ) Richard (Rick) Zamora

Bump mypy version ( dask#9697 ) crusaderky

Disable dashboard in test_map_partitions_df_input ( dask#9687 ) James Bourbeau

Use latest xarray-contrib/issue-from-pytest-log in upstream build ( dask#9682 ) James Bourbeau

xfail ttest_1samp for upstream scipy ( dask#9670 ) James Bourbeau

Update gpuCI RAPIDS_VER to 23.02 ( dask#9678 )

Released on November 18, 2022

Enhancements

Restrict bokeh=3 support ( dask#9673 ) Gabe Joseph

Updates for fastparquet evolution ( dask#9650 ) Martin Durant

Maintenance

Update ga-yaml-parser step in gpuCI updating workflow ( dask#9675 ) Charles Blackmon-Luca

Revert importlib.metadata workaround ( dask#9658 ) James Bourbeau

Fix mindeps-distributed CI build to handle numpy / pandas not being installed ( dask#9668 ) James Bourbeau

Released on November 15, 2022

Enhancements

Generalize from_dict implementation to allow usage from other backends ( dask#9628 ) GALI PREM SAGAR

Bug Fixes

Avoid pandas constructors in dask.dataframe.core ( dask#9570 ) Richard (Rick) Zamora

Fix sort_values with Timestamp data ( dask#9642 ) James Bourbeau

Generalize array checking and remove pd.Index call in _get_partitions ( dask#9634 ) Benjamin Zaitlen

Fix read_csv behavior for header=0 and names ( dask#9614 ) Richard (Rick) Zamora

Documentation

Update dashboard docs for queuing ( dask#9660 ) Gabe Joseph

Remove import dask as d from docstrings ( dask#9644 ) Matthew Rocklin

Fix link to partitions docs in read_parquet docstring ( dask#9636 ) qheuristics

Add API doc links to array/bag/dataframe sections ( dask#9630 ) Matthew Rocklin

Maintenance

Use conda-incubator/setup-miniconda@v2.2.0 ( dask#9662 ) John A Kirkham

Allow bokeh=3 ( dask#9659 ) James Bourbeau

Run upstream build with Python 3.10 ( dask#9655 ) James Bourbeau

Pin pyyaml version in mindeps testing ( dask#9640 ) Charles Blackmon-Luca

Add pre-commit to catch breakpoint() ( dask#9638 ) James Bourbeau

Bump xarray-contrib/issue-from-pytest-log from 1.1 to 1.2 ( dask#9635 )

Remove blosc references ( dask#9625 ) Naty Clementi

Upgrade mypy and drop unused comments ( dask#9616 ) Hendrik Makait

Harden test_repartition_npartitions ( dask#9585 ) Richard (Rick) Zamora

Released on October 31, 2022

This was a hotfix and has no changes in this repository. The necessary fix was in dask/distributed, but we decided to bump this version number for consistency.

Released on October 28, 2022

Enhancements

Enable named aggregation syntax ( dask#9563 ) ChrisJar

Add extension dtype support to set_index ( dask#9566 ) James Bourbeau

Redesigning the array HTML repr for clarity ( dask#9519 ) Shingo OKAWA

Bug Fixes

Fix merge with emtpy left DataFrame ( dask#9578 ) Ian Rose

Documentation

Add note about limiting thread oversubscription by default ( dask#9592 ) James Bourbeau

Use sphinx-click for dask CLI ( dask#9589 ) James Bourbeau

Fix Semaphore API docs ( dask#9584 ) James Bourbeau

Render meta description in map_overlap docstring ( dask#9568 ) James Bourbeau

Maintenance

Require Click 7.0+ in Dask ( dask#9595 ) John A Kirkham

Temporarily restrict bokeh<3 ( dask#9607 ) James Bourbeau

Resolve importlib -related failures in upstream CI ( dask#9604 ) Charles Blackmon-Luca

Improve upstream CI report ( dask#9603 ) James Bourbeau

Fix upstream CI report ( dask#9602 ) James Bourbeau

Remove setuptools host dep, add CLI entrypoint ( dask#9600 ) Charles Blackmon-Luca

More Backend dispatch class type annotations ( dask#9573 ) Ian Rose

Released on October 14, 2022

New Features

Backend library dispatching for IO in Dask-Array and Dask-DataFrame ( dask#9475 ) Richard (Rick) Zamora

Add new CLI that is extensible ( dask#9283 ) Doug Davis

Enhancements

Groupby median ( dask#9516 ) Ian Rose

Fix array copy not being a no-op ( dask#9555 ) David Hoese

Add support for string timedelta in map_overlap ( dask#9559 ) Nicolas Grandemange

Shuffle-based groupby for single functions ( dask#9504 ) Ian Rose

Make datetime.datetime tokenize idempotantly ( dask#9532 ) Martin Durant

Support tokenizing datetime.time ( dask#9528 ) Tim Paine

Bug Fixes

Avoid race condition in lazy dispatch registration ( dask#9545 ) James Bourbeau

Do not allow setitem to np.nan for int dtype ( dask#9531 ) Doug Davis

Stable demo column projection ( dask#9538 ) Ian Rose

Ensure pickle -able binops in delayed ( dask#9540 ) Ian Rose

Fix project CSV columns when selecting ( dask#9534 ) Martin Durant

Documentation

Update Parquet best practice ( dask#9537 ) Matthew Rocklin

Maintenance

Restrict tiledb-py version to avoid CI failures ( dask#9569 ) James Bourbeau

Bump actions/github-script from 3 to 6 ( dask#9564 )

Bump actions/stale from 4 to 6 ( dask#9551 )

Bump peter-evans/create-pull-request from 3 to 4 ( dask#9550 )

Bump actions/checkout from 2 to 3.1.0 ( dask#9552 )

Bump codecov/codecov-action from 1 to 3 ( dask#9549 )

Bump the-coding-turtle/ga-yaml-parser from 0.1.1 to 0.1.2 ( dask#9553 )

Move dependabot configuration file ( dask#9547 ) James Bourbeau

Add dependabot for GitHub actions ( dask#9542 ) James Bourbeau

Run mypy on Windows and Linux ( dask#9530 ) crusaderky

Update gpuCI RAPIDS_VER to 22.12 ( dask#9524 )

Released on September 30, 2022

Enhancements

Remove factorization logic from array auto chunking ( dask#9507 ) James Bourbeau

Documentation

Add docs on running Dask in a standalone Python script ( dask#9513 ) James Bourbeau

Clarify custom-graph multiprocessing example ( dask#9511 ) nouman

Maintenance

Groupby sort upstream compatibility ( dask#9486 ) Ian Rose

Released on September 16, 2022

New Features

Add DataFrame and Series median methods ( dask#9483 ) James Bourbeau

Enhancements

Shuffle groupby default ( dask#9453 ) Ian Rose

Filter by list ( dask#9419 ) Greg Hayes

Added distributed.utils.key_split functionality to dask.utils.key_split ( dask#9464 ) Luke Conibear

Bug Fixes

Fix overlap so that set_index doesn’t drop rows ( dask#9423 ) Julia Signell

Fix assigning pandas Series to column when ddf.columns.min() raises ( dask#9485 ) Erik Welch

Fix metadata comparison stack_partitions ( dask#9481 ) James Bourbeau

Provide default for split_out ( dask#9493 ) Lawrence Mitchell

Deprecations

Allow split_out to be None , which then defaults to 1 in groupby().aggregate() ( dask#9491 ) Ian Rose

Documentation

Fixing enforce_metadata documentation, not checking for dtypes ( dask#9474 ) Nicolas Grandemange

Fix it's –> its typo ( dask#9484 ) Nat Tabris

Maintenance

Workaround for parquet writing failure using some datetime series but not others ( dask#9500 ) Ian Rose

Filter out numeric_only warnings from pandas ( dask#9496 ) James Bourbeau

Avoid set_index(..., inplace=True) where not necessary ( dask#9472 ) James Bourbeau

Avoid passing groupby key list of length one ( dask#9495 ) James Bourbeau

Update test_groupby_dropna_cudf based on cudf support for group_keys ( dask#9482 ) James Bourbeau

Remove dd.from_bcolz ( dask#9479 ) James Bourbeau

Added flake8-bugbear to pre-commit hooks ( dask#9457 ) Luke Conibear

Bind loop variables in function definitions ( B023 ) ( dask#9461 ) Luke Conibear

Added assert for comparisons ( B015 ) ( dask#9459 ) Luke Conibear

Set top-level default shell in CI workflows ( dask#9469 ) James Bourbeau

Removed unused loop control variables ( B007 ) ( dask#9458 ) Luke Conibear

Replaced getattr calls for constant attributes ( B009 ) ( dask#9460 ) Luke Conibear

Pin libprotobuf to allow nightly pyarrow in the upstream CI build ( dask#9465 ) Joris Van den Bossche

Replaced mutable data structures for default arguments ( B006 ) ( dask#9462 ) Luke Conibear

Changed flake8 mirror and updated version ( dask#9456 ) Luke Conibear

Released on September 2, 2022

Enhancements

Enable automatic column projection for groupby aggregations ( dask#9442 ) Richard (Rick) Zamora

Accept superclasses in NEP-13/17 dispatching ( dask#6710 ) Gabe Joseph

Bug Fixes

Rename by columns internally for cumulative operations on the same by columns ( dask#9430 ) Pavithra Eswaramoorthy

Fix get_group with categoricals ( dask#9436 ) Pavithra Eswaramoorthy

Fix caching-related MaterializedLayer.cull performance regression ( dask#9413 ) Richard (Rick) Zamora

Documentation

Add maintainer documentation page ( dask#9309 ) James Bourbeau

Maintenance

Revert skipped fastparquet test ( dask#9439 ) Pavithra Eswaramoorthy

tmpfile does not end files with period on empty extension ( dask#9429 ) Hendrik Makait

Skip failing fastparquet test with latest release ( dask#9432 ) James Bourbeau

Released on August 19, 2022

New Features

Implement ma.*_like functions ( dask#9378 ) Ruth Comer

Enhancements

Fuse compatible annotations ( dask#9402 ) Ian Rose

Shuffle-based groupby aggregation for high-cardinality groups ( dask#9302 ) Richard (Rick) Zamora

Unpack namedtuple ( dask#9361 ) Hendrik Makait

Bug Fixes

Fix SeriesGroupBy cumulative functions with axis=1 ( dask#9377 ) Pavithra Eswaramoorthy

Sparse array reductions ( dask#9342 ) Ian Rose

Fix make_meta while using categorical column with index ( dask#9348 ) Pavithra Eswaramoorthy

Don’t allow incompatible keywords in DataFrame.dropna ( dask#9366 ) Naty Clementi

Make set_index handle entirely empty dataframes ( dask#8896 ) Julia Signell

Improve dataclass handling in unpack_collections ( dask#9345 ) Hendrik Makait

Fix bag sampling when there are some smaller partitions ( dask#9349 ) Ian Rose

Add support for empty partitions to da.min / da.max functions ( dask#9268 ) geraninam

Documentation

Clarify that bind() etc. regenerate the keys ( dask#9385 ) crusaderky

Consolidate dashboard diagnostics documentation ( dask#9357 ) Sarah Charlotte Johnson

Remove outdated meta information Pavithra Eswaramoorthy

Maintenance

Use entry_points utility in sizeof ( dask#9390 ) James Bourbeau

Add entry_points compatibility utility ( dask#9388 ) Jacob Tomlinson

Upload environment file artifact for each CI build ( dask#9372 ) James Bourbeau

Remove werkzeug pin in CI ( dask#9371 ) James Bourbeau

Fix type annotations for dd.from_pandas and dd.from_delayed ( dask#9362 ) Jordan Yap

Released on August 5, 2022

Enhancements

Ensure make_meta doesn’t hold ref to data ( dask#9354 ) Jim Crist-Harif

Revise divisions logic in from_pandas ( dask#9221 ) Richard (Rick) Zamora

Warn if user sets index with existing index ( dask#9341 ) Julia Signell

Add keepdims keyword for da.average ( dask#9332 ) Ruth Comer

Change repr methods to avoid Layer materialization ( dask#9289 ) Richard (Rick) Zamora

Bug Fixes

Make sure order kwarg will not crash the astype method ( dask#9317 ) Genevieve Buckley

Fix bug for cumsum on cupy chunked dask arrays ( dask#9320 ) Genevieve Buckley

Match input and output structure in _sample_reduce ( dask#9272 ) Pavithra Eswaramoorthy

Include meta in array serialization ( dask#9240 ) Frédéric BRIOL

Fix Index.memory_usage ( dask#9290 ) James Bourbeau

Fix division calculation in dask.dataframe.io.from_dask_array ( dask#9282 ) Jordan Yap

Documentation

Fow to use kwargs with custom task graphs ( dask#9322 ) Genevieve Buckley

Add note to da.from_array about how the order is not preserved ( dask#9346 ) Julia Signell

Add I/O info for async functions ( dask#9326 ) Logan Norman

Tidy up docs snippet for futures IO functions ( dask#9340 ) Julia Signell

Use consistent variable names for pandas df and Dask ddf in dataframe-groupby.rst ( dask#9304 ) ivojuroro

Switch js-yaml for yaml.js in config converter ( dask#9306 ) Jacob Tomlinson

Maintenance

Update da.linalg.solve for SciPy 1.9.0 compatibility ( dask#9350 ) Pavithra Eswaramoorthy

Update test_getitem_avoids_large_chunks_missing ( dask#9347 ) Pavithra Eswaramoorthy

Fix docs title formatting for “Extend sizeof ” Doug Davis

Import loop_in_thread fixture in tests ( dask#9337 ) James Bourbeau

Temporarily xfail test_solve_sym_pos ( dask#9336 ) Pavithra Eswaramoorthy

Fix small typo in 10 minutes to Dask page ( dask#9329 ) Shaghayegh

Temporarily pin werkzeug in CI to avoid test suite hanging ( dask#9325 ) James Bourbeau

Add tests for cupy.angle() ( dask#9312 ) Peter Andreas Entschev

Update gpuCI RAPIDS_VER to 22.10 ( dask#9314 )

Add pandas[test] to test extra ( dask#9110 ) Ben Beasley

Add bokeh and scipy to upstream CI build ( dask#9265 ) James Bourbeau

Released on July 22, 2022

Enhancements

Return Dask array if all axes are squeezed ( dask#9250 ) Pavithra Eswaramoorthy

Make cycle reported by toposort shorter ( dask#9068 ) Erik Welch

Unknown chunk slicing - raise informative error ( dask#9285 ) Naty Clementi

Bug Fixes

Fix bug in HighLevelGraph.cull ( dask#9267 ) Richard (Rick) Zamora

Sort categories ( dask#9264 ) Pavithra Eswaramoorthy

Use max (instead of sum ) for calculating warnsize ( dask#9235 ) Pavithra Eswaramoorthy

Fix bug when filtering on partitioned column with pyarrow ( dask#9252 ) Richard (Rick) Zamora

Documentation

Updated repartition documentation to add note about partition_size ( dask#9288 ) Dylan Stewart

Don’t include docs in Array methods, just refer to module docs ( dask#9244 ) Julia Signell

Remove outdated reference to scheduler and worker dashboards ( dask#9278 ) Pavithra Eswaramoorthy

Fix a few typos ( dask#9270 ) Tim Gates

Adds an custom aggregate example using numpy methods ( dask#9260 ) geraninam

Maintenance

Add type annotations to dd.from_pandas and dd.from_delayed ( dask#9237 ) Michael Milton

Update calculate_divisions docstring ( dask#9275 ) Tom Augspurger

Update test_plot_multiple for upcoming bokeh release ( dask#9261 ) James Bourbeau

Add typing to common array properties ( dask#9255 ) Illviljan

Released on July 8, 2022

Enhancements

Support pathlib.PurePath in normalize_token ( dask#9229 ) Angus Hollands

Add AttributeNotImplementedError for properties so IPython glob search works ( dask#9231 ) Erik Welch

map_overlap : multiple dataframe handling ( dask#9145 ) Fabien Aulaire

Read entrypoints in dask.sizeof ( dask#7688 ) Angus Hollands

Bug Fixes

Fix TypeError: 'Serialize' object is not subscriptable when writing parquet dataset with Client(processes=False) ( dask#9015 ) Lucas Miguel Ponce

Correct dtypes when concat with an empty dataframe ( dask#9193 ) Pavithra Eswaramoorthy

Documentation

Highlight note about persist ( dask#9234 ) Pavithra Eswaramoorthy

Update release-procedure to include more detail and helpful commands ( dask#9215 ) Julia Signell

Better SEO for Futures and Dask vs. Spark pages ( dask#9217 ) Sarah Charlotte Johnson

Maintenance

Use math.prod instead of np.prod on lists, tuples, and iters ( dask#9232 ) crusaderky

Only import IPython if type checking ( dask#9230 ) Florian Jetter

Tougher mypy checks ( dask#9206 ) crusaderky

Released on June 24, 2022

Enhancements

Dask in pyodide ( dask#9053 ) Ian Rose

Create dask.utils.show_versions ( dask#9144 ) Sultan Orazbayev

Better error message for unsupported numpy operations on dask.dataframe objects. ( dask#9201 ) Julia Signell

Add allow_rechunk kwarg to dask.array.overlap function ( dask#7776 ) Genevieve Buckley

Add minutes and hours to dask.utils.format_time ( dask#9116 ) Matthew Rocklin

More retries when writing parquet to remote filesystem ( dask#9175 ) Ian Rose

Bug Fixes

Timedelta deterministic hashing ( dask#9213 ) Fabien Aulaire

Enum deterministic hashing ( dask#9212 ) Fabien Aulaire

shuffle_group() : avoid converting to arrays ( dask#9157 ) Mads R. B. Kristensen

Deprecations

Deprecate extra format_time utility ( dask#9184 ) James Bourbeau

Documentation

Better SEO for 10 Minutes to Dask ( dask#9182 ) Sarah Charlotte Johnson

Better SEO for Delayed and Best Practices ( dask#9194 ) Sarah Charlotte Johnson

Include known inconsistency in DataFrame str.split accessor docstring ( dask#9177 ) Richard Pelgrim

Add inconsistencies keyword to derived_from ( dask#9192 ) Richard Pelgrim

Add missing append in delayed best practices example ( dask#9202 ) Ben

Fix indentation in Best Practices ( dask#9196 ) Sarah Charlotte Johnson

Add link to Genevieve Buckley ’s blog on chunk sizes ( dask#9199 ) Pavithra Eswaramoorthy

Update to_csv docstring ( dask#9094 ) Sarah Charlotte Johnson

Maintenance

Update versioneer: change from using SafeConfigParser to ConfigParser ( dask#9205 ) Thomas A Caswell

Remove ipython hack in CI( dask#9200 ) crusaderky

Released on June 10, 2022

Enhancements

Add feature to show names of layer dependencies in HLG JupyterLab repr ( dask#9081 ) Angelos Omirolis

Add arrow schema extraction dispatch ( dask#9169 ) GALI PREM SAGAR

Add sort_results argument to assert_eq ( dask#9130 ) Pavithra Eswaramoorthy

Add weeks to parse_timedelta ( dask#9168 ) Matthew Rocklin

Warn that cloudpickle is not always deterministic ( dask#9148 ) Pavithra Eswaramoorthy

Switch parquet default engine ( dask#9140 ) Jim Crist-Harif

Use deterministic hashing with _iLocIndexer / _LocIndexer ( dask#9108 ) Fabien Aulaire

Enfore consistent schema in to_parquet pyarrow ( dask#9131 ) Jim Crist-Harif

Bug Fixes

Fix pyarrow.StringArray pickle ( dask#9170 ) Jim Crist-Harif

Fix parallel metadata collection in pyarrow engine ( dask#9165 ) Richard (Rick) Zamora

Improve pyarrow partitioning logic ( dask#9147 ) James Bourbeau

pyarrow 8.0 partitioning fix ( dask#9143 ) James Bourbeau

Documentation

Better SEO for Installing Dask and Dask DataFrame Best Practices ( dask#9178 ) Sarah Charlotte Johnson

Update logos page in docs ( dask#9167 ) Sarah Charlotte Johnson

Add example using pandas Series to map_partition doctring ( dask#9161 ) Alex-JG3

Update docs theme for rebranding ( dask#9160 ) Sarah Charlotte Johnson

Better SEO for docs on Dask DataFrames ( dask#9128 ) Sarah Charlotte Johnson

Maintenance

Remove ensure_file from recommended practice for downstream libraries ( dask#9171 ) Matthew Rocklin

Test round-tripping DataFrame parquet I/O including pyspark ( dask#9156 ) Ian Rose

Try disabling HDF5 locking ( dask#9154 ) Ian Rose

Link best practices to DataFrame-parquet ( dask#9150 ) Tom Augspurger

Fix typo in map_partitions func parameter description ( dask#9149 ) Christopher Akiki

Un- xfail test_groupby_grouper_dispatch ( dask#9139 ) GALI PREM SAGAR

Temporarily import cleanup fixture from distributed ( dask#9138 ) James Bourbeau

Simplify partitioning logic in pyarrow parquet engine ( dask#9041 ) Richard (Rick) Zamora

Released on May 26, 2022

Enhancements

Add a dispatch for non-pandas Grouper objects and use it in GroupBy ( dask#9074 ) brandon-b-miller

Error if read_parquet & to_parquet files intersect ( dask#9124 ) Jim Crist-Harif

Visualize task graphs using ipycytoscape ( dask#9091 ) Ian Rose

Documentation

Fix various typos ( dask#9126 ) Ryan Russell

Maintenance

Fix flaky test_filter_nonpartition_columns ( dask#9127 ) Pavithra Eswaramoorthy

Update gpuCI RAPIDS_VER to 22.08 ( dask#9120 )

Include conftest.py` in sdists ( dask#9115 ) Ben Beasley

Released on May 24, 2022

New Features

Add DataFrame.from_dict classmethod ( dask#9017 ) Matthew Powers

Add from_map function to Dask DataFrame ( dask#8911 ) Richard (Rick) Zamora

Enhancements

Improve to_parquet error for appended divisions overlap ( dask#9102 ) Jim Crist-Harif

Enabled user-defined process-initializer functions ( dask#9087 ) ParticularMiner

Mention align_dataframes=False option in map_partitions error ( dask#9075 ) Gabe Joseph

Add kwarg enforce_ndim to dask.array.map_blocks() ( dask#8865 ) ParticularMiner

Implement Series.GroupBy.fillna / DataFrame.GroupBy.fillna methods ( dask#8869 ) Pavithra Eswaramoorthy

Allow fillna with Dask DataFrame ( dask#8950 ) Pavithra Eswaramoorthy

Update error message for assignment with 1-d dask array ( dask#9036 ) Pavithra Eswaramoorthy

Collection Protocol ( dask#8674 ) Doug Davis

Patch around pandas ArrowStringArray pickling ( dask#9024 ) Jim Crist-Harif

Band-aid for compute_as_if_collection ( dask#8998 ) Ian Rose

Add p2p shuffle option ( dask#8836 ) Matthew Rocklin

Bug Fixes

Fixup column projection with no columns ( dask#9106 ) Jim Crist-Harif

Blockwise cull NumPy dtype ( dask#9100 ) Ian Rose

Fix column-projection bug in from_map ( dask#9078 ) Richard (Rick) Zamora

Prevent nulls in index for non-numeric dtypes ( dask#8963 ) Jorge López

Fix is_monotonic methods for more than 8 partitions ( dask#9019 ) Julia Signell

Handle enumerate and generator inputs to from_map ( dask#9066 ) Richard (Rick) Zamora

Revert is_dask_collection ; back to previous implementation ( dask#9062 ) Doug Davis

Fix Blockwise.clone does not handle iterable literal arguments correctly ( dask#8979 ) JSKenyon

Array setitem hardmask ( dask#9027 ) David Hassell

Fix overlapping divisions error on append ( dask#8997 ) Ian Rose

Deprecations

Add pre-deprecation warnings for read_parquet kwargs chunksize and aggregate_files ( dask#9052 ) Richard (Rick) Zamora

Documentation

Document map_partitions handling of args vs kwargs , usage of partition_info ( dask#9084 ) Charles Blackmon-Luca

Update custom collection documentation (leverage new collection protocol) ( dask#9097 ) Doug Davis

Better SEO for docs on creating and storing Dask DataFrames ( dask#9098 ) Sarah Charlotte Johnson

Clarify chunking in imread docstring ( dask#9082 ) Genevieve Buckley

Rearrange docs TOC ( dask#9001 ) Matthew Rocklin

Corrected map_blocks() docstring for kwarg enforce_ndim ( dask#9071 ) ParticularMiner

Update DataFrame SQL docs references to other libraries ( dask#9077 ) Charles Blackmon-Luca

Update page on creating and storing Dask DataFrames ( dask#9025 ) Sarah Charlotte Johnson

Maintenance

Include NUMPY_LICENSE.txt in license files ( dask#9113 ) Ben Beasley

Increase retries when installing nightly pandas ( dask#9103 ) James Bourbeau

Force nightly pyarrow in the upstream build ( dask#9095 ) Joris Van den Bossche

Improve object handling & testing of ensure_unicode ( dask#9059 ) John A Kirkham

Force nightly pyarrow in the upstream build ( dask#8993 ) Joris Van den Bossche

Additional check on is_dask_collection ( dask#9054 ) Doug Davis

Update ensure_bytes ( dask#9050 ) John A Kirkham

Add end of file pre-commit hook ( dask#9045 ) James Bourbeau

Add codespell pre-commit hook ( dask#9040 ) James Bourbeau

Remove the HDFS tests ( dask#9039 ) Jim Crist-Harif

Fix flaky test_reductions_2D ( dask#9037 ) Jim Crist-Harif

Prevent codecov from notifying of failure too soon ( dask#9031 ) Jim Crist-Harif

Only test on Python 3.9 on macos ( dask#9029 ) Jim Crist-Harif

Update to_timedelta default unit ( dask#9010 ) Pavithra Eswaramoorthy

Released on May 2, 2022

Highlights

This is a bugfix release for this issue .

Documentation

Add highlights section to 2022.04.2 release notes ( dask#9012 ) James Bourbeau

Released on April 29, 2022

Highlights

This release includes several deprecations/breaking API changes to dask.dataframe.read_parquet and dask.dataframe.to_parquet :

to_parquet no longer writes _metadata files by default. If you want to write a _metadata file, you can pass in write_metadata_file=True .

read_parquet now defaults to split_row_groups=False , which results in one Dask dataframe partition per parquet file when reading in a parquet dataset. If you’re working with large parquet files you may need to set split_row_groups=True to reduce your partition size.

read_parquet no longer calculates divisions by default. If you require read_parquet to return dataframes with known divisions, please set calculate_divisions=True .

read_parquet has deprecated the gather_statistics keyword argument. Please use the calculate_divisions keyword argument instead.

read_parquet has deprecated the require_extensions keyword argument. Please use the parquet_file_extension keyword argument instead.

New Features

Add removeprefix and removesuffix as StringMethods ( dask#8912 ) Jorge López

Enhancements

Call fs.invalidate_cache in to_parquet ( dask#8994 ) Jim Crist-Harif

Change to_parquet default to write_metadata_file=None ( dask#8988 ) Jim Crist-Harif

Let arg reductions pass keepdims ( dask#8926 ) Julia Signell

Change split_row_groups default to False in read_parquet ( dask#8981 ) Richard (Rick) Zamora

Improve NotImplementedError message for da.reshape ( dask#8987 ) Jim Crist-Harif

Simplify to_parquet compute path ( dask#8982 ) Jim Crist-Harif

Raise an error if you try to use vindex with a Dask object ( dask#8945 ) Julia Signell

Avoid pre_buffer=True when a precache method is specified ( dask#8957 ) Richard (Rick) Zamora

from_dask_array uses blockwise instead of merging graphs ( dask#8889 ) Bryan Weber

Use pre_buffer=True for “pyarrow” Parquet engine ( dask#8952 ) Richard (Rick) Zamora

Bug Fixes

Handle dtype=None correctly in da.full ( dask#8954 ) Tom White

Fix dask-sql bug caused by blockwise fusion ( dask#8989 ) Richard (Rick) Zamora

to_parquet errors for non-string column names ( dask#8990 ) Jim Crist-Harif

Make sure da.roll works even if shape is 0 ( dask#8925 ) Julia Signell

Fix recursion error issue with set_index ( dask#8967 ) Paul Hobson

Stringify BlockwiseDepDict mapping values when produces_keys=True ( dask#8972 ) Richard (Rick) Zamora

Use DataFram`eIOLayer in DataFrame.from_delayed ( dask#8852 ) Richard (Rick) Zamora

Check that values for the in predicate in read_parquet are correct ( dask#8846 ) Bryan Weber

Fix bug for reduction of zero dimensional arrays ( dask#8930 ) Tom White

Specify dtype when deciding division using np.linspace in read_sql_query ( dask#8940 ) Cheun Hong

Deprecations

Deprecate gather_statistics from read_parquet ( dask#8992 ) Richard (Rick) Zamora

Change require_extension to top-level parquet_file_extension read_parquet kwarg ( dask#8935 ) Richard (Rick) Zamora

Documentation

Update write_metadata_file discussion in documentation ( dask#8995 ) Richard (Rick) Zamora

Update DataFrame.merge docstring ( dask#8966 ) Pavithra Eswaramoorthy

Added description for parameter align_arrays in array.blockwise() ( dask#8977 ) ParticularMiner

ecommend not to use map_block(drop_axis=...) on chunked axes of an array ( dask#8921 ) ParticularMiner

Add copy button to code snippets in docs ( dask#8956 ) James Bourbeau

Maintenance

Pandas 1.5.0 compatibility ( dask#8961 ) Ian Rose

Add pytest-timeout to distributed envs on CI ( dask#8986 ) Julia Signell

Improve read_parquet docstring formatting ( dask#8971 ) Bryan Weber

Remove pytest.warns(None) ( dask#8924 ) Pavithra Eswaramoorthy

Document Python 3.10 as supported ( dask#8976 ) Eray Aslan

parse_timedelta option to enforce explicit unit ( dask#8969 ) crusaderky

mypy compatibility ( dask#8854 ) Paul Hobson

Add a docs page for Dask & Parquet ( dask#8899 ) Jim Crist-Harif

Adds configuration to ignore revs in blame ( dask#8933 ) Bryan Weber

Released on April 15, 2022

New Features

Add missing NumPy ufuncs: abs , left_shift , right_shift , positive . ( dask#8920 ) Tom White

Enhancements

Avoid collecting parquet metadata in pyarrow when write_metadata_file=False ( dask#8906 ) Richard (Rick) Zamora

Better error for failed wildcard path in dd.read_csv() (fixes #8878) ( dask#8908 ) Roger Filmyer

Return da.Array rather than dd.Series for non-ufunc elementwise functions on dd.Series ( dask#8558 ) Julia Signell

Let get_dummies use meta computation in map_partitions ( dask#8898 ) Julia Signell

Masked scalars input to da.from_array ( dask#8895 ) David Hassell

Raise ValueError in merge_asof for duplicate kwargs ( dask#8861 ) Bryan Weber

Bug Fixes

Make is_monotonic work when some partitions are empty ( dask#8897 ) Julia Signell

Fix custom getter in da.from_array when inline_array=False ( dask#8903 ) Ian Rose

Correctly handle dict-specification for rechunk. ( dask#8859 ) Richard

Fix merge_asof : drop index column if left_on == right_on ( dask#8874 ) Gil Forsyth

Deprecations

Warn users that engine='auto' will change in future ( dask#8907 ) Jim Crist-Harif

Remove pyarrow-legacy engine from parquet API ( dask#8835 ) Richard (Rick) Zamora

Documentation

Add note on missing parameter out for dask.array.dot ( dask#8913 ) Francesco Andreuzzi

Update DataFrame.query docstring ( dask#8890 ) Pavithra Eswaramoorthy

Maintenance

Don’t test da.prod on large integer data ( dask#8893 ) Jim Crist-Harif

Add network marks to tests that fail without an internet connection ( dask#8881 ) Paul Hobson

Fix gpuCI GHA version ( dask#8891 ) Charles Blackmon-Luca

xfail / skip some flaky distributed tests ( dask#8887 ) Jim Crist-Harif

Remove unused (deprecated) code from ArrowDatasetEngine ( dask#8885 ) Richard (Rick) Zamora

Add mild typing to common utils functions, part 2 ( dask#8867 ) crusaderky

Documentation of Limitation of sample() ( dask#8858 ) Nadiem Sissouno

Released on April 1, 2022

Note

This is the first release with support for Python 3.10

New Features

Add Python 3.10 support ( dask#8566 ) James Bourbeau

Enhancements

Add check on dtype.itemsize in order to produce a useful error ( dask#8860 ) Davide Gavio

Add mild typing to common utils functions ( dask#8848 ) Matthew Rocklin

Add sanity checks to divisions setter ( dask#8806 ) Jim Crist-Harif

Use Blockwise and map_partitions for more tasks ( dask#8831 ) Bryan Weber

Bug Fixes

Fix dataframe.merge_asof to preserve right_on column ( dask#8857 ) Sarah Charlotte Johnson

Fix “Buffer dtype mismatch” for pandas >= 1.3 on 32bit ( dask#8851 ) Ben Greiner

Fix slicing fusion by altering SubgraphCallable getter ( dask#8827 ) Ian Rose

Deprecations

Remove support for PyPy ( dask#8863 ) James Bourbeau

Drop setuptools at runtime ( dask#8855 ) crusaderky

Remove dataframe.tseries.resample.getnanos ( dask#8834 ) Sarah Charlotte Johnson

Documentation

Organize diagnostic and performance docs ( dask#8871 ) Naty Clementi

Add image to explain drop_axis option of map_blocks ( dask#8868 ) ParticularMiner

Maintenance

Update gpuCI RAPIDS_VER to 22.06 ( dask#8828 )

Restore test_parquet in http ( dask#8850 ) Bryan Weber

Simplify gpuCI updating workflow ( dask#8849 ) Charles Blackmon-Luca

Released on March 18, 2022

New Features

Bag: add implementation for reservoir sampling ( dask#7636 ) Daniel Mesejo-León

Add ma.count to Dask array ( dask#8785 ) David Hassell

Change to_parquet default to compression="snappy" ( dask#8814 ) Jim Crist-Harif

Add weights parameter to dask.array.reduction ( dask#8805 ) David Hassell

Add ddf.compute_current_divisions to get divisions on a sorted index or column ( dask#8517 ) Julia Signell

Enhancements

Pass name and doc through on DelayedLeaf ( dask#8820 ) Leo Gao

Raise exception for not implemented merge how option ( dask#8818 ) Naty Clementi

Move Bag.map_partitions to Blockwise ( dask#8646 ) Richard (Rick) Zamora

Improve error messages for malformed config files ( dask#8801 ) Jim Crist-Harif

Revise column-projection optimization to capture common dask-sql patterns ( dask#8692 ) Richard (Rick) Zamora

Useful error for empty divisions ( dask#8789 ) Pavithra Eswaramoorthy

Scipy 1.8.0 compat: copy private classes into dask/array/stats.py ( dask#8694 ) Julia Signell

Raise warning when using multiple types of schedulers where one is distributed ( dask#8700 ) Pedro Silva

Bug Fixes

Fix bug in applying != filter in read_parquet ( dask#8824 ) Richard (Rick) Zamora

Fix set_index when directly passed a dask Index ( dask#8680 ) Paul Hobson

Quick fix for unbounded memory usage in tensordot ( dask#7980 ) Genevieve Buckley

If hdf file is empty, don’t fail on meta creation ( dask#8809 ) Julia Signell

Update clone_key("x") to retain prefix ( dask#8792 ) crusaderky

Fix “physical” column bug in pyarrow-based read_parquet ( dask#8775 ) Richard (Rick) Zamora

Fix groupby.shift bug caused by unsorted partitions after shuffle ( dask#8782 ) kori73

Fix serialization bug ( dask#8786 ) Richard (Rick) Zamora

Deprecations

Bump diagnostics bokeh dependency to 2.4.2 ( dask#8791 ) Charles Blackmon-Luca

Deprecate bcolz support ( dask#8754 ) Pavithra Eswaramoorthy

Finish making map_overlap default boundary kwarg 'none' ( dask#8743 ) Genevieve Buckley

Documentation

Custom collection example docs fix ( dask#8807 ) Doug Davis

Add Series.str , Series.dt , and Series.cat accessors to docs ( dask#8757 ) Sarah Charlotte Johnson

Fix docstring for ddf.compute_current_divisions ( dask#8793 ) Julia Signell

Dashboard docs on /status page ( dask#8648 ) Naty Clementi

Clarify divisions kwarg in repartition docstring ( dask#8781 ) Sarah Charlotte Johnson

Update Docker images to use ghcr.io ( dask#8774 ) Jacob Tomlinson

Maintenance

Reduce gpuci pytest parallelism ( dask#8826 ) GALI PREM SAGAR

absolufy-imports - No relative imports - PEP8 ( dask#8796 ) Julia Signell

Tidy up assert_eq calls in array tests ( dask#8812 ) Julia Signell

Avoid pytest.warns(None) ( dask#8718 ) LSturtew

Fix test_describe_empty to work without global -Werror ( dask#8291 ) Michał Górny

Temporarily xfail graphviz tests on windows ( dask#8794 ) Jim Crist-Harif

Use packaging.parse for md5 compatibility ( dask#8763 ) James Bourbeau

Make tokenize work in a FIPS 140-2 environment ( dask#8762 ) Jim Crist-Harif

Label issues and PRs on open with ‘needs triage’ ( dask#8761 ) Julia Signell

Add some extra test coverage ( dask#8302 ) lrjball

Specify action version and change from pull_request_target to pull_request ( dask#8767 ) Julia Signell

Make scheduler kwarg pass though to sub functions in da.assert_eq ( dask#8755 ) Julia Signell

Released on February 25, 2022

New Features

Add aggregate functions first and last to dask.dataframe.pivot_table ( dask#8649 ) Knut Nordanger

Add std() support for datetime64 dtype for pandas-like objects ( dask#8523 ) Ben Glossner

Add materialized task counts to HighLevelGraph and Layer html reprs ( dask#8589 ) kori73

Enhancements

Do not allow iterating a DataFrameGroupBy ( dask#8696 ) Bryan Weber

Fix missing newline after info() call on empty DataFrame ( dask#8727 ) Naty Clementi

Add groupby.compute as a not implemented method ( dask#8734 ) Dranaxel

Improve multi dataframe join performance ( dask#8740 ) Holden Karau

Include bool type for Index ( dask#8732 ) Naty Clementi

Allow ArrowDatasetEngine subclass to override pandas->arrow conversion also for partitioned write ( dask#8741 ) Joris Van den Bossche

Increase performance of k-diagonal extraction in da.diag() and da.diagonal() ( dask#8689 ) ParticularMiner

Change linspace creation to match numpy when num equal to 0 ( dask#8676 ) Peter

Tokenize dataclasses ( dask#8557 ) Gabe Joseph

Update tokenize to treat dict and kwargs differently ( dask#8655 ) James Bourbeau

Bug Fixes

Fix bug in dask.array.roll() for roll-shifts that match the size of the input array ( dask#8723 ) ParticularMiner

Fix for normalize_function dataclass methods ( dask#8527 ) Sarah Charlotte Johnson

Fix rechunking with zero-size-chunks ( dask#8703 ) ParticularMiner

Move creation of sqlalchemy connection for picklability ( dask#8745 ) Julia Signell

Deprecations

Drop Python 3.7 ( dask#8572 ) James Bourbeau

Deprecate iteritems ( dask#8660 ) James Bourbeau

Deprecate dataframe.tseries.resample.getnanos ( dask#8752 ) Sarah Charlotte Johnson

Add deprecation warning for pyarrow-legacy engine ( dask#8758 ) Richard (Rick) Zamora

Documentation

Update link typos in changelog ( dask#8717 ) James Bourbeau

Clarify dask.visualize docstring ( dask#8710 ) Dranaxel

Update Docker example to use current best practices ( dask#8731 ) Jacob Tomlinson

Update docs to include distributed.Client.preload ( dask#8679 ) Bryan Weber

Document monthly social meeting ( dask#8595 ) Thomas Grainger

Add docs for Gen2 access with RBAC/ACL i.e. security principal ( dask#8748 ) Martin Thøgersen

Use Dask configuration extension from dask-sphinx-theme ( dask#8751 ) Benjamin Zaitlen

Maintenance

Unpin coverage in CI ( dask#8690 ) James Bourbeau

Add manual trigger for running test suite ( dask#8716 ) James Bourbeau

Xfail scheduler_HLG_unpack_import ; flaky test ( dask#8724 ) Mike McCarty

Temporarily remove scipy upstream CI build ( dask#8725 ) James Bourbeau

Bump pre-release version to be greater than stable releases ( dask#8728 ) Charles Blackmon-Luca

Move custom sort function logic to internal sort_values ( dask#8571 ) Charles Blackmon-Luca

Pin cloudpickle and scipy in docs requirements ( dask#8737 ) Julia Signell

Make the labeler not delete labels, and look for the docs at the right spot ( dask#8746 ) Julia Signell

Fix docs build warnings ( dask#8432 ) Kristopher Overholt

Update test status badge ( dask#8747 ) James Bourbeau

Fix parquet test_pandas_timestamp_overflow_pyarrow test ( dask#8733 ) Joris Van den Bossche

Only run PR builds on changes to relevant files ( dask#8756 ) Charles Blackmon-Luca

Released on February 11, 2022

Note

This is the last release with support for Python 3.7

New Features

Add region to to_zarr when using existing array ( dask#8590 ) Chris Roat

Add engine_kwargs support to dask.dataframe.to_sql ( dask#8609 ) Amir Kadivar

Add include_path_column arg to read_json ( dask#8603 ) Bryan Weber

Add expand_dims to Dask array ( dask#8687 ) Tom White

Enhancements

Add scheduler option to assert_eq utilities ( dask#8610 ) Xinrong Meng

Fix eye inconsistency with NumPy for dtype=None ( dask#8685 ) Tom White

Fix concatenate inconsistency with NumPy for axis=None ( dask#8686 ) Tom White

Type annotations, part 1 ( dask#8295 ) crusaderky

Really allow any iterable to be passed as a meta ( dask#8629 ) Julia Signell

Use map_partitions (Blockwise) in to_parquet ( dask#8487 ) Richard (Rick) Zamora

Bug Fixes

Result of reducing an array should not depend on its chunk-structure ( dask#8637 ) ParticularMiner

Pass place-holder metadata to map_partitions in ACA code path ( dask#8643 ) Richard (Rick) Zamora

Deprecations

Deprecate is_monotonic ( dask#8653 ) James Bourbeau

Remove some deprecations ( dask#8605 ) James Bourbeau

Documentation

Add Domino Data Lab to Hosted / managed Dask clusters ( dask#8675 ) Ray Bell

Fix inter-linking and remove deprecated function ( dask#8715 ) Julia Signell

Fix imbalanced backticks. ( dask#8693 ) Matthias Bussonnier

Add documentation for high level graph visualization ( dask#8483 ) Genevieve Buckley

Update documentation of ProgressBar out parameter ( dask#8604 ) Pedro Silva

Improve documentation of dask.config.set ( dask#8705 ) crusaderky

Revert mention to mypy among type checkers ( dask#8699 ) crusaderky

Maintenance

Update warning handling in get_dummies tests ( dask#8651 ) James Bourbeau

Add a github changelog template ( dask#8714 ) Julia Signell

Update year in LICENSE.txt ( dask#8665 ) David Hoese

Update pre-commit version ( dask#8691 ) James Bourbeau

Include scipy in upstream CI build ( dask#8681 ) James Bourbeau

Temporarily pin scipy < 1.8.0 in CI ( dask#8683 ) James Bourbeau

Pin scipy to less than 1.8.0 in GPU CI ( dask#8698 ) Julia Signell

Avoid pytest.warns(None) in test_multi.py ( dask#8678 ) James Bourbeau

Update GHA concurrent job cancellation ( dask#8652 ) James Bourbeau

Make test__get_paths robust to site.PREFIXES being set ( dask#8644 ) James Bourbeau

Bump gpuCI PYTHON_VER to 3.9 ( dask#8642 ) Charles Blackmon-Luca

Released on January 28, 2022

New Features

Add dask.dataframe.series.view() ( dask#8533 ) Pavithra Eswaramoorthy

Enhancements

Update tz for fastparquet + pandas 1.4.0 ( dask#8626 ) Martin Durant

Cleaning up misc tests for pandas compat ( dask#8623 ) Julia Signell

Moving to SQLAlchemy >= 1.4 ( dask#8158 ) McToel

Pandas compat: Filter sparse warnings ( dask#8621 ) Julia Signell

Fail if meta is not a pandas object ( dask#8563 ) Julia Signell

Use fsspec.parquet module for better remote-storage read_parquet performance ( dask#8339 ) Richard (Rick) Zamora

Move DataFrame ACA aggregations to HLG ( dask#8468 ) Richard (Rick) Zamora

Add optional information about originating function call in DataFrameIOLayer ( dask#8453 ) Richard (Rick) Zamora

Blockwise array creation redux ( dask#7417 ) Ian Rose

Refactor config default search path retrieval ( dask#8573 ) James Bourbeau

Add optimize_graph flag to Bag.to_dataframe function ( dask#8486 ) Maxim Lippeveld

Make sure that delayed output operations still return lists of paths ( dask#8498 ) Julia Signell

Pandas compat: Fix to_frame name to not pass None ( dask#8554 ) Julia Signell

Pandas compat: Fix axis=None warning ( dask#8555 ) Julia Signell

Expand Dask YAML config search directories ( dask#8531 ) abergou

Bug Fixes

Fix groupby.cumsum with series grouped by index ( dask#8588 ) Julia Signell

Fix derived_from for pandas methods ( dask#8612 ) Thomas J. Fan

Enforce boolean ascending for sort_values ( dask#8440 ) Charles Blackmon-Luca

Fix parsing of setitem indices ( dask#8601 ) David Hassell

Avoid divide by zero in slicing ( dask#8597 ) Doug Davis

Deprecations

Downgrade meta error in ( dask#8563 ) to warning ( dask#8628 ) Julia Signell

Pandas compat: Deprecate append when pandas >= 1.4.0 ( dask#8617 ) Julia Signell

Documentation

Replace outdated columns argument with meta in DataFrame constructor ( dask#8614 ) kori73

Refactor deploying docs ( dask#8602 ) Jacob Tomlinson

Maintenance

Pin coverage in CI ( dask#8631 ) James Bourbeau

Move cached_cumsum imports to be from dask.utils ( dask#8606 ) James Bourbeau

Update gpuCI RAPIDS_VER to 22.04 ( dask#8600 )

Update cocstring for from_delayed function ( dask#8576 ) Kirito1397

Handle plot_width / plot_height deprecations ( dask#8544 ) Bryan Van de Ven

Remove unnecessary pyyaml importorskip ( dask#8562 ) James Bourbeau

Specify scheduler in DataFrame assert_eq ( dask#8559 ) Gabe Joseph

Released on January 14, 2022

New Features

Add groupby.shift method ( dask#8522 ) kori73

Add DataFrame.nunique ( dask#8479 ) Sarah Charlotte Johnson

Add da.ndim to match np.ndim ( dask#8502 ) Julia Signell

Enhancements

Only show percentile interpolation= keyword warning if NumPy version >= 1.22 ( dask#8564 ) Julia Signell

Raise PerformanceWarning when limit and "array.slicing.split-large-chunks" are None ( dask#8511 ) Julia Signell

Define normalize_seq function at import time ( dask#8521 ) Illviljan

Ensure that divisions are alway tuples ( dask#8393 ) Charles Blackmon-Luca

Allow a callable scheduler for bag.groupby ( dask#8492 ) Julia Signell

Save Zarr arrays with dask-on-ray scheduler ( dask#8472 ) TnTo

Make byte blocks more even in read_bytes ( dask#8459 ) Martin Durant

Improved the efficiency of matmul() by completely removing concatenation ( dask#8423 ) ParticularMiner

Limit max chunk size when reshaping dask arrays ( dask#8124 ) Genevieve Buckley

Changes for fastparquet superthrift ( dask#8470 ) Martin Durant

Bug Fixes

Fix boolean indices in array assignment ( dask#8538 ) David Hassell

Detect default dtype on array-likes ( dask#8501 ) aeisenbarth

Fix optimize_blockwise bug for duplicate dependency names ( dask#8542 ) Richard (Rick) Zamora

Update warnings for DataFrame.GroupBy.apply and transform ( dask#8507 ) Sarah Charlotte Johnson

Track HLG layer name in Delayed ( dask#8452 ) Gabe Joseph

Fix single item nanmin and nanmax reductions ( dask#8484 ) Julia Signell

Make read_csv with comment kwarg work even if there is a comment in the header ( dask#8433 ) Julia Signell

Deprecations

Replace interpolation with method and method with internal_method ( dask#8525 ) Julia Signell

Remove daily stock demo utility ( dask#8477 ) James Bourbeau

Documentation

Add a join example in docs that be run with copy/paste ( dask#8520 ) kori73

Mention dashboard link in config ( dask#8510 ) Ray Bell

Fix changelog section hyperlinks ( dask#8534 ) Aneesh Nema

Hyphenate “single-machine scheduler” for consistency ( dask#8519 ) Deepyaman Datta

Normalize whitespace in doctests in slicing.py ( dask#8512 ) Maren Westermann

Best practices storage line typo ( dask#8529 ) Michael Delgado

Update figures ( dask#8401 ) Sarah Charlotte Johnson

Remove pyarrow -only reference from split_row_groups in read_parquet docstring ( dask#8490 ) Naty Clementi

Maintenance

Remove obsolete LocalFileSystem tests that fail for fsspec>=2022.1.0 ( dask#8565 ) Richard (Rick) Zamora

Tweak: “RuntimeWarning: invalid value encountered in reciprocal” ( dask#8561 ) crusaderky

Fix skipna=None for DataFrame.sem ( dask#8556 ) Julia Signell

Fix PANDAS_GT_140 ( dask#8552 ) Julia Signell

Collections with HLG must always implement dask_layers ( dask#8548 ) crusaderky

Work around race condition in import llvmlite ( dask#8550 ) crusaderky

Set a minimum version for pyyaml ( dask#8545 ) Gaurav Sheni

Adding nodefaults to environments to fix tiledb + mac issue ( dask#8505 ) Julia Signell

Set ceiling for setuptools ( dask#8509 ) Julia Signell

Add workflow / recipe to generate Dask nightlies ( dask#8469 ) Charles Blackmon-Luca

Bump gpuCI CUDA_VER to 11.5 ( dask#8489 ) Charles Blackmon-Luca

Released on December 10, 2021

New Features

Add Series and Index is_monotonic* methods ( dask#8304 ) Daniel Mesejo-León

Enhancements

Blockwise map_partitions with partition_info ( dask#8310 ) Gabe Joseph

Better error message for length of array with unknown chunk sizes ( dask#8436 ) Doug Davis

Use by instead of index internally on the Groupby class ( dask#8441 ) Julia Signell

Allow custom sort functions for sort_values ( dask#8345 ) Charles Blackmon-Luca

Add warning to read_parquet when statistics and partitions are misaligned ( dask#8416 ) Richard (Rick) Zamora

Support where argument in ufuncs ( dask#8253 ) mihir

Make visualize more consistent with compute ( dask#8328 ) JSKenyon

Bug Fixes

Fix map_blocks not using own arguments in name generation ( dask#8462 ) David Hoese

Fix for index error with reading empty parquet file ( dask#8410 ) Sarah Charlotte Johnson

Fix nullable-dtype error when writing partitioned parquet data ( dask#8400 ) Richard (Rick) Zamora

Fix CSV header bug ( dask#8413 ) Richard (Rick) Zamora

Fix empty chunk causes exception in nanmin / nanmax ( dask#8375 ) Boaz Mohar

Deprecations

Deprecate token keyword argument to map_blocks ( dask#8464 ) James Bourbeau

Deprecation warning for default value of boundary kwarg in map_overlap ( dask#8397 ) Genevieve Buckley

Documentation

Clarify block_info documentation ( dask#8425 ) Genevieve Buckley

Output from alt text sprint ( dask#8456 ) Sarah Charlotte Johnson

Update talks and presentations ( dask#8370 ) Naty Clementi

Update Anaconda link in “Paid support” section of docs ( dask#8427 ) Martin Durant

Fixed broken dask-gateway link in ecosystem.rst ( dask#8424 ) ofirr

Fix CuPy doctest error ( dask#8412 ) Genevieve Buckley

Maintenance

Bump Bokeh min version to 2.1.1 ( dask#8431 ) Bryan Van de Ven

Fix following fsspec=2021.11.1 release ( dask#8428 ) Martin Durant

Add dask/ml.py to pytest exclude list ( dask#8414 ) Genevieve Buckley

Update gpuCI RAPIDS_VER to 22.02 ( dask#8394 )

Unpin graphviz and improve package management in environment-3.7 ( dask#8411 ) Julia Signell