distributed

Released on September 1, 2023

Enhancements

• Don't capture functools frames in code (8146) `crusaderky`_

• Ignore asyncio when scraping code (8122) `Matthew Rocklin`_

• Remove stringification (8083) `Florian Jetter`_

Bug Fixes

• Ensure NannyPlugin are always installed (8107) `Florian Jetter`_

• Don't use exception hooks to shorten tracebacks (8127) `crusaderky`_

• Fix P2P shuffle with LocalCluster(..., processes=False) (8125) `Hendrik Makait`_

• Merge with P2P shuffle fails if left_index or right_index is True (8121) `Patrick Hoefler`_

Documentation

• Do not advertise @span decorator (8120) `crusaderky`_

Maintenance

• Assert DF equality in P2P tests (8117) `Hendrik Makait`_

• Fix warnings from pandas=2.1.0 (8145) `Patrick Hoefler`_

• Enforce dtypes early in P2P shuffling (8131) `Hendrik Makait`_

• Bump actions/checkout from 3.5.3 to 3.6.0 (8139)

• Fix flakiness in test_spans (8132) `crusaderky`_

• Remove duplicated teardown logic in Server.close, Nanny.close, and Server.stop (8129) `David Gold`_

• Fix flaky test_task_counter (8134) `crusaderky`_

• Use asyncio.timeout() in ConnectionPool where available (8109) `Thomas Grainger`_

Released on August 18, 2023

New Features

• Add memray integration (8044) `Florian Jetter`_

Enhancements

• Await async listener.stop in Worker.close (8118) `Hendrik Makait`_

• Minor fixes in memray (8113) `Florian Jetter`_

• Enable basic p2p shuffle for dask-cudf (7743) `Richard (Rick) Zamora`_

• Don't shut down unresponsive workers on gather() (8101) `crusaderky`_

• Propagate CancelledError in gather_from_workers (8089) `crusaderky`_

• Better logging for anomalous task termination (8082) `crusaderky`_

Bug Fixes

• Handle null partitions in P2P shuffling (8116) `Hendrik Makait`_

• Handle CancelledError properly in ConnectionPool (8110) `Florian Jetter`_

• Fix additional race condition that can cause P2P restart to deadlock (8094) `Hendrik Makait`_

• Ensure x-axis is uniform when plotting (8093) `Florian Jetter`_

• Fix deadlock in P2P restarts (8091) `Hendrik Makait`_

Documentation

• Add memray integration to API docs (8115) `James Bourbeau`_

• Fix default in description of LocalCluster s scheduler_port (8073) `Danferno`_

Maintenance

• Remove types_mapper arg now that it is captured in from_pyarrow_table_dispatch (8114) `Richard (Rick) Zamora`_

• Make P2P shuffle extensible (8096) `Hendrik Makait`_

• Make PreloadManager a Sequence (8112) `Hendrik Makait`_

• Introduce PreloadManager to handle failures in preload setup/teardown (8078) `Hendrik Makait`_

• Restructure P2P code (8098) `Hendrik Makait`_

• Make ToPickle a Generic (8097) `Hendrik Makait`_

• Dedicated job for memray tests (8104) `Florian Jetter`_

• Fix test_task_groups_update_start_stop, again (8102) `crusaderky`_

• Remove dumps_task (8067) `Florian Jetter`_

• Simplify usage of queues in nanny (6655) `Florian Jetter`_

• Fix flakiness in tests caused by WindowsTime (8087) `crusaderky`_

• Overhaul gather() (7997) `crusaderky`_

• Fix flaky test_asyncprocess.py::test_simple (8085) `crusaderky`_

• Skip test_client.py::test_file_descriptors_dont_leak on Mac OS (8080) `Hendrik Makait`_

• Reorder operations in Worker.close (8076) `Hendrik Makait`_

Released on August 4, 2023

Enhancements

• Offload CPU intensive sections of update graph to unblock event loop (8049) `Florian Jetter`_

• Log worker close reason in events (8042) `Florian Jetter`_

• Exclude comm handshake from connect timeout (7698) `Florian Jetter`_

• Automatically restart P2P shuffles when output worker leaves (7970) `Hendrik Makait`_

• Add Client.unregister_scheduler_plugin method (7968) `Brian Phillips`_

• Fix log message (8029) `Hendrik Makait`_

• Send shards grouped by input chunk in P2P rechunking (8010) `Hendrik Makait`_

Bug Fixes

• Close state machine and add-ins first in Worker.close (8066) `Hendrik Makait`_

• Fix decide_worker picking a closing worker (8032) `crusaderky`_

• Raise CommClosedError in get_stream_address (8020) `jochenott`_

• Respect average nthreads in adaptive (8041) `Matthew Rocklin`_

• Use queued tasks in adaptive target (8037) `Matthew Rocklin`_

• Restore support for yield unsafe Client context managers and deprecate that support (7987) `Thomas Grainger`_

Documentation

• Change worker_saturation default value to 1.1 in the documention (8040) `minhnguyenxuan60`_

• Clarified concurrent.futures section in client.rst (8048) `mercyo12`_

Maintenance

• Fix flaky test_worker_metrics (8069) `crusaderky`_

• Use SPDX in license metadata (8065) `jakirkham`_

• Rebalance ci1 markers (8061) `Florian Jetter`_

• Ensure stream messages are always ordered (8059) `Florian Jetter`_

• Simplify update graph (8047) `Florian Jetter`_

• Provide close reason when signal is caught (8045) `Florian Jetter`_

• Allow unclosed comms in tests (8057) `Florian Jetter`_

• Cosmetic tweak to adaptive_target (8052) `crusaderky`_

• Fix linting (8046) `Florian Jetter`_

• Update gpuCI RAPIDS_VER to 23.10 (8033)

• Test against more recent pyarrow versions (8021) `James Bourbeau`_

• Add a test for GraphLayout with scatter (8025) `Irina Truong`_

• Fix compatibility variable naming (8030) `Hendrik Makait`_

Released on July 20, 2023

Enhancements

• gather_dep should handle CancelledError (8013) `crusaderky`_

• Pass stimulus_id to SchedulerPlugin.remove_worker and SchedulerPlugin.transition (7974) `Hendrik Makait`_

• Log stimulus_id in retire_worker (8003) `crusaderky`_

• Use BufferOutputStream in P2P (7991) `Florian Jetter`_

• Add Coiled to ignored modules for code sniffing (7986) `Matthew Rocklin`_

• Progress bar can group tasks by span (7952) `Irina Truong`_

• Improved error messages for P2P shuffling (7979) `Hendrik Makait`_

• Reduce removing comms log to debug level (7972) `Florian Jetter`_

Bug Fixes

• Fix for TypeError: '<' not supported in graph dashboard (8017) `Irina Truong`_

• Fix shuffle code to work with pyarrow 13 (8009) `Joris Van den Bossche`_

Documentation

• Add some top-level exposition to the p2p rechunking code (7978) `Lawrence Mitchell`_

Maintenance

• Add test when not repartitioning for p2p in set_index (8016) `Patrick Hoefler`_

• Bump JamesIves/github-pages-deploy-action from 4.4.2 to 4.4.3 (8008)

• Configure asyncio loop using loop_factory kwarg rather than using the set_event_loop_policy (7969) `Thomas Grainger`_

• Fix P2P worker cleanup (7981) `Hendrik Makait`_

• Skip click v8.1.4 in mypy pre-commit hook (7989) `Thomas Grainger`_

• Remove accidental duplicated conversion of pyarrow Table to pandas (7983) `Joris Van den Bossche`_

Released on July 7, 2023

Enhancements

• Propagate spans to tasks (7898) `crusaderky`_

• Make Fine Performance Metrics bar graph horizontal (7966) `crusaderky`_

• Don't pile up context_meter callbacks (7961) `crusaderky`_

• Polish Fine Performance Metrics plot (7963) `crusaderky`_

• Sign task-erred with run_id and reject outdated responses (7933) `Hendrik Makait`_

• Set Client.as_current when entering ctx (6527) `Florian Jetter`_

• Re-run erred task on ComputeTaskEvent (7967) `Hendrik Makait`_

Bug Fixes

• Fix crash in spans when time() is not monotonic (7960) `crusaderky`_

Documentation

• Documentation for Fine Performance Metrics and Spans (7945) `crusaderky`_

• Update client.py to be consistent with the docstring (7705) `Sultan Orazbayev`_

Maintenance

• Use distributed.wait_for in test_close_async_task_handles_cancellation (7955) `Thomas Grainger`_

• Fix flaky UCX tests (7950) `Peter Andreas Entschev`_

Released on June 26, 2023

Enhancements

• Add idle time to fine performance metrics (7938) `crusaderky`_

• Spans: capture code snippets (7930) `crusaderky`_

• Improve memory footprint of P2P rechunking (7897) `Hendrik Makait`_

• Improve error message on invalid state in _handle_remove_replicas (7920) `Hendrik Makait`_

• Make ShuffleSchedulerExtension.remove_worker more robust (7921) `Hendrik Makait`_

• Provide more information if occupancy drops below zero (7924) `Hendrik Makait`_

• Improved conversion between pyarrow and pandas in P2P shuffling (7896) `Hendrik Makait`_

Bug Fixes

• Add Cluster.called_from_running_loop and fix Cluster.asynchronous (7941) `Jacob Tomlinson`_

• Fix annotations and spans leaking between threads (7935) `Irina Truong`_

• Handle null partitions in P2P shuffling (7922) `Jonathan De Troye`_

• Fix race condition in Fine Performance Metrics sync (7927) `crusaderky`_

• Avoid (7923) by starting run_id at 1 (7925) `Hendrik Makait`_

• Fix glitches in Fine Performance Metrics stacked graph (7919) `crusaderky`_

Maintenance

• Wipe the cache after (7935) (7946) `crusaderky`_

• Remove grace period for unclosed comms in gen_cluster (7937) `Thomas Grainger`_

• raise pytest.skip is redundant (7939) `crusaderky`_

• Fix test_rechunk_with_{fully|partially}_unknown_dimension on CI (7934) `Hendrik Makait`_

• Fix compatibility with numpy 1.25 (7932) `crusaderky`_

• Spans: refactor sums of mappings (7918) `crusaderky`_

• Fix flaky test_send_metrics_to_scheduler (7931) `crusaderky`_

• Avoid calls to make_current() and make_clear() by using asyncio.run in LoopRunner (7467) `Thomas Grainger`_

• Add needs triage label to re/opened PRs and issues (7916) `Miles`_

• Remove span_id from global metrics on scheduler (7917) `crusaderky`_

• Add spans to Fine Performance Metrics bokeh dashboard (7911) `crusaderky`_

• FinePerformanceMetrics dashboard overhaul (7910) `crusaderky`_

• Check for skip-caching label (7907) `Miles`_

• Fix CI changes from (7902) (7905) `Hendrik Makait`_

• Rename get_default_shuffle_algorithm to get_default_shuffle_method (7902) `Hendrik Makait`_

• Bump actions/checkout from 3.5.2 to 3.5.3 (7904)

• Refactor P2P rechunk validation (7890) `Hendrik Makait`_

Released on June 9, 2023

Enhancements

• Post fine performance metrics to spans (7885) `crusaderky`_

• Unique Spans (7882) `crusaderky`_

• Add a timeout to client.as_completed that mirrors concurrent.futures.as_completed timeout (7811) `Thomas Grainger`_

• Enforce dtypes in P2P shuffle (7879) `Hendrik Makait`_

• Support load= keyword for Client.upload_file (7873) `James Bourbeau`_

• Support get_worker() and worker_client() in async tasks (7844) `Thomas Grainger`_

• Capture line number for code frames (7786) `Miles`_

Bug Fixes

• Avoid meta roundtrip in P2P shuffle (7895) `Hendrik Makait`_

• Fix Fine Performance Metrics mis-aligned ColumnData lengths (7893) `Miles`_

• Fix Fine Performance Metrics spilling crash (7878) `Miles`_

• Fix spans bug when scatter or client_desires_new_key creates a task (7886) `crusaderky`_

• Fix Fine Performance Metrics w/ Bokeh 3 (7874) `Miles`_

• TaskGroup.start can move backwards (7867) `crusaderky`_

• Use properly imported MatDescriptor for cupy dispatch registration (7868) `Charles Blackmon-Luca`_

• Ensure retire_workers works if AMM extension hasn't been loaded (7863) `crusaderky`_

Maintenance

• Review user-defined fine performance metrics (7894) `crusaderky`_

• Fix tests that disable the shuffle extension (7883) `crusaderky`_

• Refactor Scheduler.is_idle (7881) `crusaderky`_

• Link TaskGroups to Spans (7869) `crusaderky`_

• Spans skeleton (7862) `crusaderky`_

• Update gpuCI RAPIDS_VER to 23.08 (7855)

• Bump JamesIves/github-pages-deploy-action from 4.4.1 to 4.4.2 (7865)

Released on May 26, 2023

Note

This release drops support for Python 3.8. As of this release Dask supports Python 3.9, 3.10, and 3.11. See this community issue for more details.

Enhancements

• Exclude IPython code from computations (7788) `Miles`_

• Drop Python 3.8 support (7840) `Thomas Grainger`_

• Add storage_options to performance_report (7636) `ypogorelova`_

• Don't warn about mismatched msgpack (7839) `Irina Truong`_

• Clean up sys.path on Server shutdown (7838) `James Bourbeau`_

• Dashboard: Fine Performance Metrics (7725) `Miles`_

Bug Fixes

• Properly handle unknown chunk sizes in P2P rechunking (7856) `Hendrik Makait`_

• Minimal change to work around (7726) / support for UCX (7851) `Benjamin Zaitlen`_

• Don't end computations until cluster is truly idle (7790) `crusaderky`_

Maintenance

• Explicitly install anaconda-client from conda-forge when uploading conda nightlies (7861) `Charles Blackmon-Luca`_

• Fix is_idle docs build (7854) `James Bourbeau`_

• Add tests for P2P barrier fusion (7845) `Hendrik Makait`_

• Avoid DeprecationWarning in cupy dispatch registration (7836) `Lawrence Mitchell`_

Released on May 12, 2023

Enhancements

• Client.upload_file send to both Workers and Scheduler and rename scratch directory (7802) `Miles`_

• Allow dashboard to be used with bokeh prereleases (7814) `James Bourbeau`_

Bug Fixes

• Ensure log_event of non-msgpack serializable object do not kill servers (7472) `Florian Jetter`_

• Fix test_nanny.py duplicated pytestmark definitions (7819) `Thomas Grainger`_

• Fix flaky test_dask_worker.py::test_single_executable_deprecated (7817) `Thomas Grainger`_

Maintenance

• Annotation-less P2P shuffling (7801) `Hendrik Makait`_

• Fix docstring for batch_size in client.map (7833) `David Chudzicki`_

• Refactor test_protocol.py (7829) `crusaderky`_

• Lint #6496 (7828) `crusaderky`_

• Remove hardcoded 60s timeout (6496) `Florian Jetter`_

• Add __init__.py files to template and static directories (7809) `Thomas Grainger`_

• Disable compression for fast comms (7768) `crusaderky`_

• Avoid deprecated pd.api.types.is_sparse (7813) `James Bourbeau`_

• Bump gpuCI PYTHON_VER from 3.8 to 3.9 (7812) `Charles Blackmon-Luca`_

Released on April 28, 2023

Enhancements

• Enable GIL monitoring when gilknocker installed (7730) `Miles`_

• By default only set logging handler if no other handler has been set to avoid double logging (7750) `Thomas Grainger`_

• Cluster wait (6700) `Iain Dorrington`_

• Add Prometheus counter for SystemMonitor.last_time (7785) `Miles`_

Bug Fixes

• Partial revert defaultclient config setting (7803) `Florian Jetter`_

• Delay awaiting async SchedulerPlugin.{add|remove}_worker hooks in order to immediately execute all sync ones (7799) `Hendrik Makait`_

• Fix check_idle not returning the correct value if no change to idleness (7781) `Jacob Tomlinson`_

Maintenance

• Avoid warning when gilknocker not installed (7808) `James Bourbeau`_

• Only set worker/nanny to Status.running if it is in Status.init (7773) `Thomas Grainger`_

• Add --cov-config=pyproject.toml so config is always correctly loaded by pytest-cov (7793) `Thomas Grainger`_

• gilknocker from conda-forge (7791) `James Bourbeau`_

• Minor zict cleanup (7783) `crusaderky`_

• Bump actions/checkout from 3.5.0 to 3.5.2 (7784)

• Fix typing now that code is tuple of frame(s) (7778) `Nat Tabris`_

Released on April 14, 2023

Note

With this release we are making a change which will require the Dask scheduler to have consistent software and hardware capabilities as the client and workers.

It's always been recommended that your client and workers have a consistent software and hardware environment so that data structures and dependencies can be pickled and passed between them. However recent changes to the Dask scheduler mean that we now also require your scheduler to have the same consistent environment as everything else.

Enhancements

• Meter queue time to the offload executor (7758) `crusaderky`_

• Add GIL contention metric to Prometheus (7651) `Miles`_

• Add methods Client.forward_logging() and Client.unforward_logging(). (7276) `Max Bane`_

• Optionally capture more frames in computations (7656) `Gabe Joseph`_

• Consider Jupyter activity in idle timeout (7687) `Gabe Joseph`_

• Add a dashboard component that displays RMM memory (7718) `Peter Andreas Entschev`_

• Improve error message if shuffle/rechunk lost annotations (7707) `Hendrik Makait`_

• Exception chaining in P2P shuffling (7706) `Hendrik Makait`_

• Use pickle for graph submissions from client to scheduler (7564) `Florian Jetter`_

Bug Fixes

• Fix crash on missing env var in dashboard link formatting (7729) `Miles`_

• Fix randbytes() on Python 3.8 (7771) `crusaderky`_

• Run scheduler of SubprocessCluster in subprocess (7727) `Hendrik Makait`_

• Drop id from RMM dashboard component (7739) `James Bourbeau`_

Maintenance

• Bump peter-evans/create-pull-request from 4 to 5 (7766)

• Fix flaky test_malloc_trim_threshold in CI (7764) `crusaderky`_

• Minor polish in spill and worker_memory_manager (7752) `crusaderky`_

• Merge identical tool.mypy.overrides sections (7749) `Thomas Grainger`_

• Add changelog section for 2023.3.2.1 (7755) `Charles Blackmon-Luca`_

• Specify ts resolution explicitly in test_processing_chain (7744) `Patrick Hoefler`_

• Unignore Sphinx ref.python (7713) `Thomas Grainger`_

• Temporary fix for test_merge_by_multiple_columns with pandas 2.0 (7747) `James Bourbeau`_

• Remove dask/gpu from gpuCI update reviewers (7741) `Charles Blackmon-Luca`_

• Update gpuCI RAPIDS_VER to 23.06 (7728)

• Remove test for DataFrame.to_hdf (7735) `Hendrik Makait`_

• Test P2P shuffling with DataFrame.to_hdf (7720) `Hendrik Makait`_

• scheduler.py typing - remove allow_incomplete_defs (7721) `Florian Jetter`_

• Remove bokeh upper bound (7413) `James Bourbeau`_

• Use declarative setuptools (7629) `Thomas Grainger`_

• Store performance metrics on scheduler (7701) `Miles`_

• Upgrade readthedocs config to ubuntu 22.04 and Python 3.11 (7722) `Thomas Grainger`_

• Clean up legacy cruft from worker reconnection (7712) `crusaderky`_

• Bump actions/checkout from 3.4.0 to 3.5.0 (7711)

• Drop support for zict 2.1.0 (7709) `crusaderky`_

• Fix mypy warning in test_client.py (7710) `crusaderky`_

• Test P2P shuffling with DataFrame.categorize (7708) `Hendrik Makait`_

Released on March 24, 2023

Enhancements

• Enhanced thread-safety in zict.File (7691) `crusaderky`_

• Future deserialization without available client (7580) `Florian Jetter`_

• Support adjusting GIL monitoring interval (7650) `Miles`_

• Gracefully stop GIL monitoring if running (7652) `Miles`_

• Fine performance metrics for execute, gather_dep, etc. (7586) `crusaderky`_

• Add GIL metric to dashboard (7646) `Miles`_

• Expose scheduler idle via RPC and HTTP API (7642) `Jacob Tomlinson`_

• Add full dashboard link in scheduler logs (7631) `Miles`_

Bug Fixes

• Tell workers when their peers have left (so they don't hang fetching data from them) (7574) `Thomas Grainger`_

• Fix regression in dashboard after (7586) (7683) `crusaderky`_

• Fix OverflowError in Cluster._sync_cluster_info() (7648) `Hendrik Makait`_

• Ensure that serialized data is measured correctly (7593) `Florian Jetter`_

Documentation

• Fix unexpected indentation in Client.cancel docstring (7694) `Thomas Grainger`_

• Improve plugin API documentation (7653) `Florian Jetter`_

Maintenance

• Configure sphinx warnings as errors (7697) `Thomas Grainger`_

• Fix naming comparison in test-report workflow script (7695) `Miles`_

• Temporarily restrict ipywidgets<8.0.5 (7693) `crusaderky`_

• Bump actions/checkout from 3.3.0 to 3.4.0 (7685)

• Temporarily restrict ipykernel<6.22.0 (7689) `James Bourbeau`_

• Fix typo in CODEOWNERS (7670) `Hendrik Makait`_

• Avoid bool object has no attribute close in @gen_cluster (7657) `Thomas Grainger`_

• Fix failing test_server_close_stops_gil_monitoring (7659) `James Bourbeau`_

• Add CODEOWNERS file (7645) `Jacob Tomlinson`_

• Remove weakref finalizer for Offload Executor (7644) `Florian Jetter`_

Released on March 10, 2023

Enhancements

• Add Jupyter link to dashboard menu if --jupyter flag is set (7638) `Jacob Tomlinson`_

• Bump minimum click version from 7.0 to 8.0 (7637) `Miles`_

• Extend dask metapackage dependencies (7630) `James Bourbeau`_

• Further improvements to Client.restart_workers (7620) `Miles`_

• P2P offload get_output_partition (7587) `Florian Jetter`_

• Initial integration of GIL contention metric (7624) `Miles`_

• Add dashboard documentation links (7610) `Miles`_

• Rename shuffle/rechunk config option/kwarg to method (7623) `Hendrik Makait`_

• Return results in restart_workers (7606) `Miles`_

• Ensure client key cancellation uses ordered messages (7583) `Florian Jetter`_

Bug Fixes

• Fix undefined async_wait_for -> async_poll_for (7627) `Miles`_

• Don't send client heartbeat without a scheduler_comm (7612) `James Bourbeau`_

• Do not unspill on free-keys (7607) `crusaderky`_

Documentation

• Add notes to Client.submit, Client.map, and Client.scatter with the description of the current task graph resolution algorithm limitations (7588) `Eugene Druzhynin`_

Maintenance

• Use range with pickle protocol versions (7635) `jakirkham`_

• Share thread pool among P2P shuffle runs (7621) `Hendrik Makait`_

• Replace psutil suspend with BlockedGatherDep in test_failing_worker_with_additional_replicas_on_cluster (7633) `Thomas Grainger`_

• Ignore pkg_resources DeprecationWarning for mindeps (7626) `Miles`_

• Implement wait_for using asyncio.timeout() on 3.11 (7571) `Thomas Grainger`_

• Use tmp_path fixture instead of outdated tmpdir fixture (7582) `ypogorelova`_

• Only one crick callback (7614) `crusaderky`_

• Add mindeps + numpy job to tests CI (7609) `Miles`_

• Do not xfail whole tests due to (6705) (7611) `crusaderky`_

Released on March 1, 2023

Bug Fixes

• Remove pyarrow dependency for rechunking (7604) `Florian Jetter`_

• Update rechunk_transfer and rechunk_unpack errors (7600) `James Bourbeau`_

Maintenance

• Remove dead code and document arguments to ShardBuffer constructors (7590) `Lawrence Mitchell`_

• Fix tests for p2p by default (7595) `Florian Jetter`_

• Remove obsolete cast (7596) `Florian Jetter`_

Released on February 24, 2023

Enhancements

• P2P for array rechunking (7534) `Hendrik Makait`_

• P2P HashJoin (7514) `Florian Jetter`_

• Unpickle Events, Variables, Queues and Semaphore safely without Client context (7579) `Florian Jetter`_

• Allow pickle to fall back to dask_serialize (7567) `Florian Jetter`_

• make ConnectionPool.remove cancel connection attempts (7547) `Thomas Grainger`_

• Meter how long each task prefix stays in each state (7560) `crusaderky`_

Bug Fixes

• Avoid parsing sys.argv when starting jupyter server (7573) `Brett Naul`_

• str/bytes compatibility for PyNVML device name (7563) `James Bourbeau`_

• metrics.monotonic() is not monotonic on Windows (7558) `crusaderky`_

• Fix for bytes/str discrepancy after PyNVML update (7544) `Peter Andreas Entschev`_

Maintenance

• Raise when attempting P2P with active fuse optimization (7585) `Hendrik Makait`_

• Fix test_shuffling (7581) `Hendrik Makait`_

• P2P: raise RuntimeError if pyarrow version is not sufficient (7578) `Florian Jetter`_

• Check for dtype support in p2p (7425) `Hendrik Makait`_

• Update parsing of FULL_RAPIDS_VER/FULL_UCX_PY_VER (7568) `Charles Blackmon-Luca`_

• move retry from get_data_from_worker to gather_from_workers (7546) `Thomas Grainger`_

• Increase numpy and pandas version pins for nightlies (7562) `James Bourbeau`_

• Set validate=True in all tests (7557) `crusaderky`_

• Remove dead code from _get_task_finished_msg (7561) `crusaderky`_

• Mark tests that take >2s as slow (7556) `crusaderky`_

• Fix test_scatter_no_workers on slow CI (7559) `crusaderky`_

• Unskip test_delete_some_results (7508) `Hendrik Makait`_

• scatter() should not sidestep the worker transition machinery (7545) `crusaderky`_

• pre-commit bump (7541) `crusaderky`_

• Better assertions in Worker.validate_state() (7549) `crusaderky`_

• Bump jacobtomlinson/gha-find-replace from 2 to 3 (7540) `James Bourbeau`_

• Bump black to 23.1.0 (7542) `crusaderky`_

• Run GPU tests on python 3.8 & 3.10 (7537) `Charles Blackmon-Luca`_

Released on February 10, 2023

Enhancements

• Rate limit the worker memory logs (7529) `Florian Jetter`_

• Move P2P barrier logic to scheduler extension (7519) `Hendrik Makait`_

Maintenance

• Use PEP 673 Self type (7530) `Thomas Grainger`_

• Tentatively fix test_pause_while_spilling (7517) `crusaderky`_

• Annotate asyncio_tcp.py (7522) `crusaderky`_

• Use dask git tip for mypy (7516) `crusaderky`_

• Upgrade to mypy v1 (7525) `Thomas Grainger`_

• Clean up calls to captured_logger (7521) `crusaderky`_

• Update isort version to 5.12.0 (7513) `Lawrence Mitchell`_

Released on January 27, 2023

Enhancements

• P2P shuffle deduplicates data and can be run several times (7486) `Hendrik Makait`_

• Reverse order of get_logs() and get_worker_logs() (7475) `Nicholas R. Knezek`_

• Add prometheus metric for time and memory used per task prefix (7406) `Thomas Grainger`_

• Additive worker counts in Prometheus (7468) `crusaderky`_

• Add help tool for taskstream (7478) `Florian Jetter`_

Bug Fixes

• Do not allow for a worker to reject a drop replica request (7490) `Hendrik Makait`_

• Fix un/packing for namedtuples with custom constructors (7465) `antonymayi`_

Documentation

• Remove timeout= from docstring example for worker_client (7497) `Florian Jetter`_

Maintenance

• Ignore get_default_shuffle_algorithm linting issue (7506) `Florian Jetter`_

• Remove set_config when using default client (7482) `Florian Jetter`_

• Update gpuCI RAPIDS_VER to 23.04 (7501)

• Fix test_balance_expensive_tasks and improve helper functions in test_steal.py (7253) `Hendrik Makait`_

• Sign every compute task with run ID to correlate response (7463) `Hendrik Makait`_

Released on January 13, 2023

New Features

• Add local SubprocessCluster that runs workers in separate processes (7431) `Hendrik Makait`_

Enhancements

• Ensure client session is quiet after cluster.close() or client.shutdown() (7429) `James Bourbeau`_

• Set lifetime-stagger default value to None (7445) `bstadlbauer`_

• Memory thresholds should never be exactly 0.0 (7458) `Stuart Berg`_

• Remove the Incorrect-Sizeof-Warning (7450) `Mads R. B. Kristensen`_

• Log exceptions in P2P shuffle tasks (7442) `Hendrik Makait`_

• Add support for packing namedtuple and add test for future resolution in submit (7292) `Andrew`_

• Avoid deep copy on lz4 decompression (7437) `crusaderky`_

• Avoid deep copy of numpy buffers on unspill (7435) `crusaderky`_

• Don't error when clicking on empty task stream plot (7432) `James Bourbeau`_

• Do not count spilled memory when comparing vs. process memory (7430) `crusaderky`_

• Stop Client periodic callbacks during shutdown() (7428) `James Bourbeau`_

• Add dask spec CLI (7427) `Matthew Rocklin`_

• Create new zstd (de)compressor for each compression call (7404) `Dylan Wragge`_

• Rename managed_in_memory etc. to match GUI (7418) `crusaderky`_

• Warn users when sizeof() returns inflated output (7419) `crusaderky`_

Bug Fixes

• Ensure dicts are properly recognized as msgpack serializable (7473) `Florian Jetter`_

• Reset state of ShuffleSchedulerExtension on restart (7446) `Hendrik Makait`_

• Reject non-string column names in P2P shuffle (7447) `Hendrik Makait`_

• Avoid int32 in dashboard (7443) `Matthew Rocklin`_

• Fix P2PShuffle serialization for categorical data (7410) `Hendrik Makait`_

• WorkerPorcess blocks on kill if still starting (7424) `Matthew Rocklin`_

Documentation

• Move Prometheus docs from dask/dask (7405) `crusaderky`_

Maintenance

• Various cleanups in semaphore (5885) `Florian Jetter`_

• test_rlimit fails on MacOSX (7457) `crusaderky`_

• Bump actions/checkout from 3.2.0 to 3.3.0 (7464)

• Remove conditional imports of psutil (7462) `crusaderky`_

• Drop support for zict < 2.1.0 (7456) `crusaderky`_

• Fix flaky test_digests (7454) `crusaderky`_

• Add minimum dependency testing to CI (7285) `Charles Blackmon-Luca`_

• Avoid overflow in statitics.mean (7426) `Matthew Rocklin`_

• Ignore numpy bool8 deprecation (7423) `Matthew Rocklin`_

• Add missing skips for pyarrow (7416) `Elliott Sales de Andrade`_

• Be more permissive about expected ciphers in tests (7417) `Elliott Sales de Andrade`_

• Revert "TST: Fetch executables from build root (2551)" (7415) `Elliott Sales de Andrade`_

Released on December 16, 2022

Enhancements

• SpillBuffer metrics (7368) `crusaderky`_

• Prometheus: measure how much spilling blocks the event loop (7370) `crusaderky`_

• Add transfer_outgoing_bytes_total metric (7388) `Gabe Joseph`_

• Fail P2PShuffle gracefully upon worker failure (7326) `Hendrik Makait`_

Bug Fixes

• Select queued tasks in stimuli, not transitions (7402) `Gabe Joseph`_

• Check ContextVar in default_client (7369) `Matthew Rocklin`_

• Fix sending event messages to non-subscribers (7014) `Laurence Watts`_

• Set sizing mode on Tabs to avoid layout collapse (7365) `Mateusz Paprocki`_

Maintenance

• Restructure P2PShuffle extensions (7390) `Hendrik Makait`_

• Add Python 3.11 classifier (7408) `James Bourbeau`_

• Add support for Python 3.11 (7249) `Thomas Grainger`_

• Add test for using annotations with client.submit and client.map (7399) `James Bourbeau`_

• Bump actions/checkout from 3.1.0 to 3.2.0 (7393)

• Remove superfluous ShuffleSchedulerExtension.barriers (7389) `Hendrik Makait`_

• Remove ignore annotation-unchecked (7379) `crusaderky`_

• Remove tornado max version from nightly recipe (7376) `Charles Blackmon-Luca`_

• Remove the experimental feature warning for Semaphore (7373) `Florian Jetter`_

Released on December 2, 2022

Enhancements

• Expose event loop health metrics in Prometheus (7360) `Hendrik Makait`_

• Allow log propagation by default (5669) `Florian Jetter`_

• Clean up of unpack_remotedata() (7322) `Mads R. B. Kristensen`_

• Upgrade to tornado 6.2 (7286) `Thomas Grainger`_

• Introduce Server level comm counters (7332) `Florian Jetter`_

• Prometheus debug log (7302) `Florian Jetter`_

Bug Fixes

• Catch BaseException s from user tasks (5997) `Gabe Joseph`_

• Impossible use case of erred deps in transition to waiting (7354) `crusaderky`_

• Fix a deadlock when queued tasks are resubmitted quickly in succession (7348) `Florian Jetter`_

Documentation

• Editorial changes to Prometheus documentation (7350) `Hendrik Makait`_

Maintenance

• Fetch all artifacts (7355) `Enrico Minack`_

• Delay fsspec and urllib3 import time (6659) `Florian Jetter`_

• Bump mypy (7349) `crusaderky`_

• Bump mypy and remove win specific run (7344) `Florian Jetter`_

• Finish overhaul of SchedulerState annotations (7333) `crusaderky`_

• Fix flaky test_pause_while_spilling (7334) `Gabe Joseph`_

• Update gpuCI RAPIDS_VER to 23.02 (7337)

Released on November 18, 2022

Enhancements

• Restrict bokeh=3 support (7329) `Gabe Joseph`_

• Respect death timeout when waiting for scheduler file (7296) `Florian Jetter`_

• Always raise exception if P2PShuffle s send fails (7317) `Hendrik Makait`_

Documentation

• Fix typo in client.run() docstring (7315) `Richard Pelgrim`_

• Note queuing default change in changelog (7314) `Gabe Joseph`_

Maintenance

• Update ga-yaml-parser step in gpuCI updating workflow (7335) `Charles Blackmon-Luca`_

• Remove exception handling from transitions (7316) `crusaderky`_

• Turn private functions into private SchedulerState methods (7260) `Hendrik Makait`_

• Bump toolz minimum version to 0.10.0 (7309) `Sam Grayson`_

Released on November 15, 2022

Note

This release changes the default scheduling mode to use queuing. This will significantly reduce cluster memory use in most cases, and generally improve stability and performance. Learn more here and please provide any feedback on this discussion.

In rare cases, this could make some workloads slower. See the documentation for more information, and how to switch back to the old mode.

New Features

• Add ForwardOutput worker plugin to forward stdout and stderr to client. (7297) `Hendrik Makait`_

• Duration counters on prefix level (7288) `Florian Jetter`_

• Include button for launching JupyterLab layout in repr (7218) `Ian Rose`_

Enhancements

• Support MIG parsing during CUDA context creation in UCX initialization (6720) `Peter Andreas Entschev`_

• Handle /metrics endpoint without prometheus-client installed (7234) `Hendrik Makait`_

• Add support for unpacking namedtuples in remote data (7282) `Andrew`_

• Enable queuing by default (7279) `Florian Jetter`_

• Fix exists->``exist`` typo in scheduler error messages (7281) `Matthew Plough`_

• If there's an exception in the Client async context manager body then close fast (6920) `Thomas Grainger`_

Documentation

• Copyediting typos + codespell pre-commit hook for docs (7294) `Matthew Plough`_

• Queuing docs (7203) `Gabe Joseph`_

Bug Fixes

• Ensure category is optional when logging "warn" events (7169) `James Bourbeau`_

• Edge and impossible transitions to memory (7205) `crusaderky`_

Maintenance

• Use conda-incubator/setup-miniconda@v2.2.0 (7310) `jakirkham`_

• Allow bokeh=3 (5648) `James Bourbeau`_

• Fix typos in P2P shuffle code (7304) `Hendrik Makait`_

• Reenable test_bad_disk (7300) `Florian Jetter`_

• Reduce max-runs in test reports (7299) `Florian Jetter`_

• Revert idle classification when worker-saturation is set (7278) `Florian Jetter`_

• Fix flaky deadline_expiration (7287) `Florian Jetter`_

• Rewrite of P2P control flow (7268) `Florian Jetter`_

• Add codecov token (7277) `Florian Jetter`_

• Bump minimum bokeh version to 2.4.2 (7271) `James Bourbeau`_

• Remove deprecated code calls to IOLoop.make_current() (7240) `Thomas Grainger`_

• Improved test for balancing expensive tasks (7272) `Hendrik Makait`_

• Refactor semaphore._Watch into general-purpose Deadline utility (7238) `Hendrik Makait`_

• Coverage report fixing (7270) `Tom Hu`_

• Require Click 7.0+ (7226) `jakirkham`_

• Drop tests (7269) `Hendrik Makait`_

• Replace test_(do_not_)steal_communication_heavy_tasks tests with more robust versions (7243) `Hendrik Makait`_

• xfail test_bad_disk (7265) `crusaderky`_

• Move transition_log from Scheduler to SchedulerState (7254) `crusaderky`_

• Remove Scheduler.log (7258) `crusaderky`_

• Use latest pickle (5826) `jakirkham`_

• Polish parsing of worker-saturation from config (7255) `crusaderky`_

• Avoid expensive occupancy calculation when unused (7257) `Gabe Joseph`_

• Un-skip test_nested_compute (7247) `Gabe Joseph`_

• Review test_do_not_steal_communication_heavy_tasks (7250) `crusaderky`_

• Fix test_stress_creation_and_deletion (7215) `crusaderky`_

• Raise exceptions in Server.handle_stream instead of swallowing/logging (7162) `Hendrik Makait`_

• Upgrade to mypy v0.982 (7241) `Thomas Grainger`_

• Fix _update_scheduler_info hanging failed tests (7225) `Gabe Joseph`_

• Bump xarray-contrib/ci-trigger from 1.1 to 1.2 (7232)

Released on October 31, 2022

• Reverted a bug where Bokeh was accidentally made non-optional (7230) `Oliver Holworthy`_

• Schedule a queued task when a task secedes (7224) `Gabe Joseph`_

This was a hotfix release

Released on October 28, 2022

New Features

• Add Client.restart_workers method (7154) `James Bourbeau`_

• Implement PackageInstall plugin for pip and conda (7126) `Hendrik Makait`_

Enhancements

• Add prometheus collector for work-stealing (7206) `Hendrik Makait`_

• Track reason of workers closing and restarting (7166) `Hendrik Makait`_

• Show no-worker on task progress bar (7171) `Florian Jetter`_

• Set OPENBLAS_NUM_THREADS by default (7177) `James Bourbeau`_

• Optionally provide local directory to data constructor (7153) `Lawrence Mitchell`_

• Introduce distributed.comm.ucx.environment config slot (7164) `Lawrence Mitchell`_

• Log information about memory limit (7160) `Florian Jetter`_

• Improve log messages on scheduler for restart (7150) `Florian Jetter`_

• More comprehensive WorkerState task counters (7167) `crusaderky`_

Documentation

• Add note to changelog about new CLI (7178) `James Bourbeau`_

• Update AMM docs (7158) `Benjamin Zaitlen`_

• Add CondaInstall to plugins doc (7149) `James Bourbeau`_

Bug Fixes

• Update minimum bokeh version message (7172) `James Bourbeau`_

• Revamped implementations of remote print() and warn(), fixing #7095 (7129) `Max Bane`_

Maintenance

• Temporarily restrict bokeh<3 (7219) `James Bourbeau`_

• Make Scheduler.reschedule private (7216) `crusaderky`_

• Fix decide_worker_rootish_queuing_disabled assert (7065) `Gabe Joseph`_

• Fix flaky test_include_communication_in_occupancy (7212) `Gabe Joseph`_

• Do not raise on leaked websockets (7199) `Florian Jetter`_

• Update nightly recipes with CLI tests, dependency changes (7201) `Charles Blackmon-Luca`_

• Make p2p shuffle submodules private (7186) `Florian Jetter`_

• Backport tornado PeriodicCallback (7165) `Florian Jetter`_

• Fix mypy failure on CI (7198) `Florian Jetter`_

• User a layer for p2p shuffle (7180) `Florian Jetter`_

• Type annotations for shuffle (7185) `Florian Jetter`_

• Do not close worker on comm error in heartbeat (7163) `Hendrik Makait`_

• Errors when setting TCP timeouts log as error (7161) `Florian Jetter`_

• Remove incorrect advice from pre-commit config (7159) `crusaderky`_

• Bump the-coding-turtle/ga-yaml-parser from 0.1.1 to 0.1.2 (7146)

• Bump JamesIves/github-pages-deploy-action from 4.1.7 to 4.4.1 (7145)

• Use functionalities network for codecov uploader (7148) `Florian Jetter`_

• Use counter metric type where appropriate, incoming_count was reporting bytes (7125) `Nat Tabris`_

Released on October 14, 2022

Note

This release deprecates dask-scheduler, dask-worker, and dask-ssh CLIs in favor of dask scheduler, dask worker, and dask ssh, respectively. The old-style CLIs will continue to work for a time, but will be removed in a future release.

As part of this migration the --reconnect, --nprocs, --bokeh, --bokeh-port CLI options have also been removed for both the old- and new-style CLIs. These options had already previously been deprecated.

Enhancements

• Use of new dask CLI (6735) `Doug Davis`_

• Refactor occupancy (7075) `Hendrik Makait`_

• Expose managed/unmanaged/spilled memory to Prometheus (7112) `crusaderky`_

• Round up saturation-factor (7116) `Gabe Joseph`_

• Return default on KeyError at any level in get_metadata (7109) `Hendrik Makait`_

• Count task states per task prefix and expose to Prometheus (7088) `Nat Tabris`_

• Add scheduler-sni option for dask workers (6290) `Burt Holzman`_

Bug Fixes

• Improve exception catching in UCX communication (7132) `Peter Andreas Entschev`_

• Improve robustness of PipInstall plugin (7111) `Hendrik Makait`_

Maintenance

• Fix dependencies that should point to dask/dask (7138) `James Bourbeau`_

• Hold on to z.sum() until test completes (7136) `Lawrence Mitchell`_

• Bump peter-evans/create-pull-request from 3 to 4 (7120)

• Update typing for system_monitor after python/typeshed#8829 (7131) `Lawrence Mitchell`_

• Fix two potentially flaky queuing tests (7124) `Gabe Joseph`_

• Bump EnricoMi/publish-unit-test-result-action from 1 to 2 (7121)

• Bump actions/checkout from 2 to 3.1.0 (7119)

• Revamp SystemMonitor (7097) `crusaderky`_

• Bump actions/cache from 2 to 3 (7118)

• Bump actions/upload-artifact from 2 to 3 (7117)

• Move dependabot configuration file (7115) `James Bourbeau`_

• Enable dependabot for GitHub Actions (7101) `Florian Jetter`_

• Update coverage upload action (7100) `Florian Jetter`_

• Adjust hardware benchmarks bokeh test (7096) `Florian Jetter`_

• Multi-platform mypy checks (7094) `crusaderky`_

• Update gpuCI RAPIDS_VER to 22.12 (7084)

Released on September 30, 2022

Enhancements

• Smarter stealing with dependencies (7024) `Hendrik Makait`_

• Enable Active Memory Manager by default (7042) `crusaderky`_

• Allow timeout strings in distributed.wait (7081) `James Bourbeau`_

• Make AMM memory measure configurable (7062) `crusaderky`_

• AMM support for actors (7072) `crusaderky`_

• Expose message-bytes-limit in config (7074) `Hendrik Makait`_

• Detect mismatching Python version in scheduler (7018) `Hendrik Makait`_

• Improve KilledWorker message users see (7043) `James Bourbeau`_

• Support for cgroups v2 and respect soft limits (7051) `Samantha Hughes`_

Bug Fixes

• Catch BaseException on UCX read error (6996) `Peter Andreas Entschev`_

• Fix transfer limiting in _select_keys_for_gather (7071) `Hendrik Makait`_

• Parse worker-saturation if a string (7064) `Gabe Joseph`_

• Nanny(config=...) parameter overlays global dask config (7069) `crusaderky`_

• Ensure default clients don't propagate to subprocesses (7028) `Florian Jetter`_

Documentation

• Improve documentation of message-bytes-limit (7077) `Hendrik Makait`_

• Minor tweaks to Sphinx documentation (7041) `crusaderky`_

• Improve upload_file API documentation (7040) `Florian Jetter`_

Maintenance

• test_serialize_numba: Workaround issue with np.empty_like in NP 1.23 (7089) `Graham Markall`_

• Type platform constants for mypy (7091) `jakirkham`_

• dask-worker-space (7054) `crusaderky`_

• Remove failing test case (7087) `Hendrik Makait`_

• test_default_client (7058) `crusaderky`_

• Fix pre-commit fails with recent versions of mypy and pandas (7068) `crusaderky`_

• Add factorization utility (7048) `James Bourbeau`_

Released on September 16, 2022

Enhancements

• Add dashboard component for size of open data transfers (6982) `Hendrik Makait`_

• Allow very fast keys and very expensive transfers as stealing candidates (7022) `Florian Jetter`_

Bug Fixes

• No longer double count transfer cost in stealing (7036) `Hendrik Makait`_

Maintenance

• Make test_wait_first_completed robust (7039) `Florian Jetter`_

• Partial annotations for SchedulerState (7023) `crusaderky`_

• Add more type annotations to stealing.py (7009) `Florian Jetter`_

• Update codecov settings (7015) `Florian Jetter`_

• Speed up test_balance (7008) `Florian Jetter`_

• Fix test report after queuing job added (7012) `Gabe Joseph`_

• Clean up env variables in Gihub Actions (7001) `crusaderky`_

• Make test_steal_reschedule_reset_in_flight_occupancy non timing dependent (7010) `Florian Jetter`_

• Replaced distributed.utils.key_split with dask.utils.key_split (7005) `Luke Conibear`_

• Revert "Revert "Limit incoming data transfers by amount of data" (6994)" (:pr:7007`) `Florian Jetter`_

• CI job running tests with queuing on (6989) `Gabe Joseph`_

• Fix distributed/tests/test_client_executor.py::test_wait (6990) `Florian Jetter`_

Released on September 2, 2022

Enhancements

• Limit incoming data transfers by amount of data (6975) `Hendrik Makait`_

• Expose transfer-related metrics in Worker.get_metrics and WorkerMetricCollector (6936) `Hendrik Makait`_

• Withhold root tasks (no co assignment) (6614) `Gabe Joseph`_

• Improve differentiation between incoming/outgoing connections and transfers (6933) `Hendrik Makait`_

• Change memory bars color on spilling/paused status (6959) `crusaderky`_

• Ensure restart clears taskgroups et al (6944) `Florian Jetter`_

• Optimise scheduler.get_comm_cost set difference (6931) `Lawrence Mitchell`_

• Expose setting multiple protocols and ports via the dask-scheduler CLI (6898) `Jacob Tomlinson`_

• Make TextProgressBar clear the line when finished (5968) `Vincenzo Eduardo Padulano`_

Bug Fixes

• Revert getaddrinfo fast path for Python 3.8 (6978) `Florian Jetter`_

• cancelled/resumed->long-running transitions (6916) `crusaderky`_

Deprecations

• Deprecate default value for Client.wait_for_workers (6942) `Florian Jetter`_

Documentation

• Document Scheduler and Worker state machine (6948) `crusaderky`_

• Insert memory_limit parameter into LocalCluster docstring (6839) `Crislana Rafael`_

Maintenance

• Revert "Limit incoming data transfers by amount of data" (6994) `Florian Jetter`_

• Cache conda environment between CI test runs (6855) `Charles Blackmon-Luca`_

• Revert "Fix co-assignment for binary operations" (6985) `Gabe Joseph`_

• Cache test_report shelves in CI (6937) `Florian Jetter`_

• Cleanup ipywidgets mocking (6918) `Thomas Grainger`_

• Improve testing of {Scheduler|Worker}MetricCollector (6945) `Hendrik Makait`_

• Clean up nanny WorkerProcess.kill (6972) `Gabe Joseph`_

• Rewrite test_reconnect to use subprocess to kill scheduler reliably (6967) `Florian Jetter`_

• Task state domain on the scheduler side (6929) `crusaderky`_

• Remove @avoid_ci from test_steal (6872) `crusaderky`_

• Use async with Worker in tests (6958) `crusaderky`_

• Ignore spurious warnings in test_quiet_close_process (6955) `crusaderky`_

• Fix tests on Windows (6954) `Hendrik Makait`_

• Prevent duplicates in HeapSet.sorted() (6952) `crusaderky`_

• Propagate worker address and improve _remove_from_processing behavior (6946) `Hendrik Makait`_

• Add HeapSet._sorted internal flag (6949) `Gabe Joseph`_

• Add HeapSet.peekn (6947) `Gabe Joseph`_

• Fix pyright error when importing from distributed (6904) `Ian Liu Rodrigues`_

• Always return ws.address from _remove_from_processing (6884) `Hendrik Makait`_

• Use async with Client: in tests (6921) `Thomas Grainger`_

• Ensure relative memory limits work as percentage of system memory (6923) `Florian Jetter`_

Released on August 19, 2022

New Features

Enhancements

• Drop comparison of versions against all clients (6861) `Hendrik Makait`_

• Log the worker name if set (6866) `Johannes Lange`_

• Skip getaddrinfo thread if host is already resolved, using socket.AI_NUMERIC* (6847) `Thomas Grainger`_

• Display unexpected state in Worker.execute validation (6856) `James Bourbeau`_

• pre-spawn-environ (6841) `crusaderky`_

• Dump has_what, missing_dep_flight (6830) `Gabe Joseph`_

Bug Fixes

• cancelled / resumed -> rescheduled transition (6913) `crusaderky`_

• Fix resource deallocation for resumed tasks (6914) `crusaderky`_

• Only close scheduler in SpecCluster if it exists (6888) `Matthew Rocklin`_

• Fix issue if exc.reason is None (6881) `Hendrik Makait`_

• Always close BatchedSend write coroutines (6865) `Gabe Joseph`_

• Harden preamble of Worker.execute against race conditions (6878) `crusaderky`_

Documentation

• Fix typo (6870) `Pieter Gijsbers`_

Maintenance

• Use retries for the test report (6926) `Florian Jetter`_

• Duplicated code: in_flight_tasks validation (6917) `crusaderky`_

• ipywidgets 8 compatibility (6912) `James Bourbeau`_

• Overhaul transitions for the resumed state (6699) `crusaderky`_

• Don't upgrade to ipywidgets 8 (6910) `crusaderky`_

• Clean up cluster process reaping (6840) `Gabe Joseph`_

• Don't use bokeh Figure in tests (6721) `Bryan Van de Ven`_

• Work around incompatibility of crick with setuptools 65 (6887) `crusaderky`_

• Add max version constraint for dask-core in nightlies (6862) `Charles Blackmon-Luca`_

• Replace port = random.randint( with port = d.utils.open_port (6883) `Thomas Grainger`_

• Fix flaky test_wall_clock (6879) `crusaderky`_

• Add descriptive error message to assert (6871) `Hendrik Makait`_

• Increase timeout in test_quiet_process (6857) `Florian Jetter`_

• Descriptive title for test report (6849) `Hendrik Makait`_

• Add flake8-bugbear as plugin to pre-commit (6809) `Hendrik Makait`_

• Remove redundant use of with clean(): (6852) `Thomas Grainger`_

• Show actual Job URL on test report (6837) `Florian Jetter`_

• Update pre-commit dependencies (6851) `Hendrik Makait`_

• Call exit callback even if AsyncProcess is reaped elsewhere (6684) `Thomas Grainger`_

• Avoid function calls in argument defaults (6812) `Hendrik Makait`_

• Ignore warning for unclose SSHCluster in tests (6827) `Florian Jetter`_

Released on August 5, 2022

New Features

• Add Jupyter Server to Dask Scheduler (6737) `Matthew Rocklin`_

Enhancements

• Human-readable formatting for disk I/O and renaming to diff net and disk (6835) `Hendrik Makait`_

• Add Cluster.get_client() method (6745) `Julia Signell`_

• Start bokeh app to activate bokeh's clean session callbacks (6728) `Martí Zamora`_

Bug Fixes

• Ensure Nanny doesn't restart workers that fail to start, and joins subprocess (6427) `Gabe Joseph`_

• Don't connect to cluster subprocesses at shutdown (6829) `Gabe Joseph`_

• Fix restart wait for workers edge case (6823) `Gabe Joseph`_

• Fix spilled size calculation in Slow (6789) `Hendrik Makait`_

Deprecations

• Deprecate passing stopped loops to LoopRunner (and therefore Client/Cluster) (6680) `Thomas Grainger`_

Documentation

• Add text to top of API docs to make sure that users are exposed to LocalCluster early (6793) `Julia Signell`_

• Change title for plugins documentation (6733) `Sarah Charlotte Johnson`_

Maintenance

• Only set 5s connect timeout in gen_cluster tests (6822) `Gabe Joseph`_

• Fix flaky test_worker_who_has_clears_after_failed_connection (6832) `Gabe Joseph`_

• Add missing skips for pyarrow (6787) `Elliott Sales de Andrade`_

• Miscellaneous flake8-bugbear issues (6814) `Hendrik Makait`_

• Assert otherwise pointless comparisons (B015) (6811) `Hendrik Makait`_

• Remove unused functions from utils_test.py (6807) `Hendrik Makait`_

• Fix Jupyter security note (6818) `Jacob Tomlinson`_

• Improve check_thread_leak output (6797) `Gabe Joseph`_

• Use contextmanager to ensure clients are closed and do not leak (6817) `Hendrik Makait`_

• Robust thread termination in test_watch and test_watch_requires_lock_to_run (6788) `Hendrik Makait`_

• Avoid unused loop control variable or name them _ (6813) `Hendrik Makait`_

• Replace assert False where an exception should always be thrown (6815) `Hendrik Makait`_

• Avoid mutable argument defaults in tests (6810) `Hendrik Makait`_

• Avoid mutable argument defaults outside of tests (6665) `Hendrik Makait`_

• Update gpuCI RAPIDS_VER to 22.10 (6798)

• Use same Python for dask worker tests (6786) `Elliott Sales de Andrade`_

Released on July 22, 2022

New Features

• Dashboard for failed tasks (6595) `Ian Rose`_

Enhancements

• Wait for workers to return in Client.restart (6714) `Gabe Joseph`_

• Remove global mutable Cluster._cluster_info (6487) `Thomas Grainger`_

Bug Fixes

• Fix: nvml no early init (6678) `Lawrence Mitchell`_

• Fix bug when restarting client (6654) `Iain Dorrington`_

• Failure to spill breaks available resources (6703) `crusaderky`_

• Fix resource allocation for tasks with dependencies (6676) `Hendrik Makait`_

Maintenance

• Revert "Set MALLOC_TRIM_THRESHOLD_ before interpreter start" (6777) `Gabe Joseph`_

• Fix mypy lint in CI (6779) `jakirkham`_

• Remove test_restart_fast_sync, test_fast_kill (6750) `Gabe Joseph`_

• Fix flaky test_async_task_group_call_later_executes_delayed_task_in_background (6744) `Hendrik Makait`_

• Drop redundant geninc (6740) `Hendrik Makait`_

• Remove unused worker_coroutines (6739) `Gabe Joseph`_

• Store ready and constrained tasks in heapsets (6711) `crusaderky`_

• Improve tests for cancelled state (6717) `crusaderky`_

• Future-proof Bokeh value import (6707) `Bryan Van de Ven`_

• Revert temporary stress test (6712) `crusaderky`_

• Validate constrained tasks (6698) `crusaderky`_

• Minor quality-of-life tweaks to cancelled state (6701) `crusaderky`_

• Pickle worker state machine exceptions (6702) `crusaderky`_

• Partial matches for worker state machine instructions (6704) `crusaderky`_

• Automatically mark all WorkerState tests (6706) `crusaderky`_

Released on July 8, 2022

Enhancements

• Use a tempdir path by default instead of cwd for the worker scratch dir (6658) `Florian Jetter`_

• Add WorkerState.all_running_tasks (6690) `Hendrik Makait`_

• Scheduler.reschedule() works only by accident (6339) `crusaderky`_

• Remove spurious UnpauseEvent at worker start (6652) `crusaderky`_

• Log if closing an executor is not possible in thread (6644) `Florian Jetter`_

• Cloudpickle register by value (6466) `Ian Rose`_

• Adding replicas to a task in fetch now sends it to flight immediately (6594) `crusaderky`_

Bug Fixes

• Fix dump output of parameter-less events (6695) `crusaderky`_

• Set MALLOC_TRIM_THRESHOLD_ before interpreter start (6681) `crusaderky`_

• Fix deadlocks around rescheduled and resumed states (6673) `crusaderky`_

• has_arg returns True for keyword-only arguments (6648) `Lawrence Mitchell`_

• Transitions caused by worker death use old 'worker-connect' stimulus_id (6657) `crusaderky`_

• A key is forgotten while acquire-replicas is running (6638) `crusaderky`_

Maintenance

• Revisit WorkerState.long_running set (6697) `crusaderky`_

• WorkerState unit tests for resumed state (6688) `crusaderky`_

• Bump version of pandas-stubs (6691) `crusaderky`_

• Add dummy factory methods for ExecuteSuccessEvent and ExecuteFailureEvent (6687) `Hendrik Makait`_

• Pin tornado<6.2 in nightly conda recipes (6675) `Peter Andreas Entschev`_

• Refactor resource restriction handling in WorkerState (6672) `Hendrik Makait`_

• test_signal and test_terminate occasionally return SIGKILL on MacOS (6671) `crusaderky`_

• Use the loop fixture in even more tests (6674) `Thomas Grainger`_

• Unconditionally import ssl (6670) `Thomas Grainger`_

• Use the loop fixture in more tests (6642) `Thomas Grainger`_

• Pin tornado to <6.2 (6668) `Florian Jetter`_

• Handle AsyncTaskGroupClosedError (6664) `Hendrik Makait`_

• Replace occurrences of large delay slowinc with locks (6656) `Florian Jetter`_

• Merge extend-ignore and ignore values for flake8 (6660) `Hendrik Makait`_

• Remove server close background task grace period (6633) `Thomas Grainger`_

• Do not use tempfile in utils_test (6651) `Florian Jetter`_

• close_worker cleanup (6650) `crusaderky`_

• Rewrite test_cancelled_resumed_after_flight_with_dependencies using WorkerState (6645) `crusaderky`_

• Log the actual duration to create a directory (6647) `Florian Jetter`_

• pandas type stubs (6635) `crusaderky`_

• Remove unused __started Event in Server (6615) `Florian Jetter`_

• Use safe temp directory in gen_cluster (6628) `Florian Jetter`_

• Print CI host info (6629) `crusaderky`_

• Deduplicate data_needed (6587) `crusaderky`_

• Remove EnsureCommunicatingAfterTransitions (6462) `crusaderky`_

• Pickle WorkerState (6623) `crusaderky`_

• Harden vs. TaskState collisions (6593) `crusaderky`_

• Do not interact with the event loop when the cluster is garbage collected (6627) `Thomas Grainger`_

Released on June 24, 2022

Highlights

This release includes the Worker State Machine refactor. The expectation should be that the worker state is its own synchronous subclass. Pulling all the state out into its own class allows us to write targeted unit tests without invoking any concurrent or asynchronous code.

See 5736 for more information.

Enhancements

• Make worker state machine methods private (6564) `crusaderky`_

• Yank state machine out of Worker class (6566) `crusaderky`_

• Track worker_state_machine.TaskState instances (6525) `Hendrik Makait`_

• Trivial tweaks to the Worker State Machine (6586) `crusaderky`_

• Replace loop.call_later and loop.add_callback with background tasks added to Server. (6603) `Thomas Grainger`_

• Support for neater WorkerState tests (6609) `crusaderky`_

• Limit TCP writes with Tornado to 2GB (6557) `hhuuggoo`_

• Enable no_implicit_optional for scheduler (6622) `Thomas Grainger`_

Bug Fixes

• Partial revert of compute-task message format (6626) `Florian Jetter`_

• Restore log message about received signals in CLI (6618) `Florian Jetter`_

• Handle empty memoryviews of bytearrays when (de)serializing (6576) `Benjamin Zaitlen`_

• Ensure steal requests from same-IP but distinct workers are rejected (6585) `Florian Jetter`_

• Fix tls_(min|max)_ version having no effect on openssl 1.1.0g or lower (6562) `Thomas Grainger`_

• Fix idle_timeout and unxfail test (6563) `Matthew Rocklin`_

• Fix crashing debug statement in _purge_state (6589) `crusaderky`_

• Abort connections on CancelledError (6574) `Thomas Grainger`_

• Fix Active Memory Manager ignores nbytes thresholds (6583) `crusaderky`_

Deprecations

• Deprecate WorkerState accessors (6579) `crusaderky`_

Maintenance

• Remove ipython hack (6599) `crusaderky`_

• Mypy enforce --no-implicit-optional (6606) `crusaderky`_

• Update versioneer: change from using SafeConfigParser to ConfigParser (6605) `Thomas A Caswell`_

• Warn unreachable for scheduler.py (6611) `Florian Jetter`_

• Refactor wait_for_state() (6581) `crusaderky`_

• Hardcode wait_for_signals signal list (6619) `Thomas Grainger`_

• Always pick an open port when running tests (6591) `Florian Jetter`_

• Log popen stdout/err when subprocess times out (6567) `Gabe Joseph`_

• Fix test_error_during_startup (6608) `Florian Jetter`_

• Make test_idle_timeout_no_workers more robust (6602) `Florian Jetter`_

• Mypy enforce --disallow-incomplete-defs (6601) `crusaderky`_

• Do not log during signal handler (6590) `Florian Jetter`_

• Don't initialize mp_context on import (6580) `Lawrence Mitchell`_

• Test retire workers deadlock (6240) `Gabe Joseph`_

• Rework some tests related to gather_dep (6472) `crusaderky`_

• Minor cosmetic review of scheduler_story and worker_story (6442) `crusaderky`_

• Force __future__.annotations with isort (6621) `Thomas Grainger`_

Released on June 10, 2022

Enhancements

• Make disk access in system monitor configurable (6537) `Matthew Rocklin`_

• Log and except errors on preload start (6553) `Matthew Rocklin`_

• Fix Scheduler.restart logic (6504) `Gabe Joseph`_

• Don't heartbeat while Worker is closing (6543) `Gabe Joseph`_

• No longer retry LocalCluster on errno.EADDRINUSE (6369) `Thomas Grainger`_

• Don't invoke log_event from state machine (6512) `crusaderky`_

• Add config option to disable profiling and disable it in many tests per default (6490) `Hendrik Makait`_

• Encapsulate Worker.batched_stream.send() (6475) `crusaderky`_

Bug Fixes

• refresh-who-has can break the worker state machine (6529) `crusaderky`_

• Restart worker if it's unrecognized by scheduler (6505) `Gabe Joseph`_

• Fix import error when distributed.rmm.pool-size is set (6482) `KoyamaSohei`_

Deprecations

• Restore signature compatibility for dask-gateway (6561) `Tom Augspurger`_

• Deprecate the io_loop and loop kwarg to Server, Worker, and Nanny (6473) `Thomas Grainger`_

• Deprecate the loop kwarg to Scheduler (6443) `Thomas Grainger`_

Documentation

• Fix typo in .nthreads() docstring example (6545) `Pavithra Eswaramoorthy`_

• Update docs theme for rebranding (6495) `Sarah Charlotte Johnson`_

Maintenance

• Refactor gather_dep (6388) `crusaderky`_

• Fix flaky test_gather_dep_one_worker_always_busy (6554) `crusaderky`_

• Remove missing-data message (6546) `crusaderky`_

• Port test_local.LocalTest to pytest tests to allow use of loop fixture (6523) `Thomas Grainger`_

• Fix test_quiet_client_close (6541) `Gabe Joseph`_

• Use default_initializer in WorkerProcess (6534) `jakirkham`_

• Avoid deadlocks in tests that use popen (6483) `Gabe Joseph`_

• Revert "Fix CLI Scheduler Tests (6502)" (6547) `Gabe Joseph`_

• Update test report URL in summary message (6532) `Gabe Joseph`_

• Update test report url (6531) `Ian Rose`_

• Assert AsyncProcess.set_exit_callback is not called with a coroutine function (6526) `Thomas Grainger`_

• Typing and docstring for Worker.close (6518) `Hendrik Makait`_

• Fix CLI Scheduler Tests (6502) `Benjamin Zaitlen`_

• Collect assertions in test_as_current_is_thread_local (6520) `Thomas Grainger`_

• Link test report from test results comment (6524) `Hendrik Makait`_

• Ignore the return value of signal.signal (6519) `Thomas Grainger`_

• Refactor all event handlers (6410) `crusaderky`_

• Fix dashboard favicon background (6514) `Jacob Tomlinson`_

• Update dashboard logo (6513) `Jacob Tomlinson`_

• Fix test_stress_scatter_death (6404) `Florian Jetter`_

• Remove CrossFilter widget (6484) `crusaderky`_

• data_needed exclusively contains tasks in fetch state (6481) `crusaderky`_

• Assert possible previous states (6488) `Florian Jetter`_

• @fail_hard can kill the whole test suite; hide errors (6474) `crusaderky`_

• Assert that a fetch->cancelled->resumed->fetch cycle is impossible (6460) `crusaderky`_

• Refactor busy workers reinsertion (6379) `crusaderky`_

• Refactor find_missing and refresh_who_has (6348) `crusaderky`_

• Rename test_collections.py to test_dask_collections.py (6486) `crusaderky`_

• update_who_has can remove workers (6342) `crusaderky`_

• Restructure test_watch_requires_lock_to_run to avoid flakes (6469) `Hendrik Makait`_

• Fix intermittent test_profile_plot failure (6456) `Matthew Rocklin`_

• Use asyncio.run to run gen_cluster, gen_test and cluster (6231) `Thomas Grainger`_

• Improve tests that watch for subprocess logs (6461) `Gabe Joseph`_

Released on May 26, 2022

Enhancements

• Add a lock to distributed.profile for better concurrency control (6421) `Hendrik Makait`_

• Send SIGKILL after SIGTERM when passing 95% memory (6419) `crusaderky`_

Bug Fixes

• Log rather than raise exceptions in preload.teardown() (6458) `Matthew Rocklin`_

• Handle failing plugin.close() calls during scheduler shutdown (6450) `Matthew Rocklin`_

• Fix slicing bug in ensure_memoryview (6449) `jakirkham`_

• Generalize UCX errors on connect() and correct pytest fixtures (6434) `Peter Andreas Entschev`_

• Run cluster widget periodic callbacks on the correct event loop (6444) `Thomas Grainger`_

Maintenance

• Disable pytest-asyncio if installed (6436) `Jacob Tomlinson`_

• Close client in sync test_actor tests (6459) `Thomas Grainger`_

• Ignore ServerSession.with_document_locked unawaited (6447) `Thomas Grainger`_

• Remove coverage pin from Python 3.10 environment (6439) `Thomas Grainger`_

• Annotate remove_worker (6441) `crusaderky`_

• Update gpuCI RAPIDS_VER to 22.08 (6428)

Released on May 24, 2022

New Features

• Add HTTP API to scheduler (6270) `Matthew Murray`_

• Shuffle Service with Scheduler Logic (6007) `Matthew Rocklin`_

Enhancements

• Follow-up on removing report and safe from Worker.close (6423) `Gabe Joseph`_

• Server close faster (6415) `Florian Jetter`_

• Disable HTTP API by default (6420) `Jacob Tomlinson`_

• Remove report and safe from Worker.close (6363) `Florian Jetter`_

• Allow deserialized plugins in register_scheduler_plugin (6401) `Matthew Rocklin`_

• WorkerState are different for different addresses (6398) `Florian Jetter`_

• Do not filter tasks before gathering data (6371) `crusaderky`_

• Remove worker reconnect (6361) `Gabe Joseph`_

• Add SchedulerPlugin.log_event handler (6381) `Matthew Rocklin`_

• Ensure occupancy tracking works as expected for long running tasks (6351) `Florian Jetter`_

• stimulus_id for all Instructions (6347) `crusaderky`_

• Refactor missing-data command (6332) `crusaderky`_

• Add idempotent to register_scheduler_plugin client (6328) `Alex Ford`_

• Add option to specify a scheduler address for workers to use (5944) `Enric Tejedor`_

Bug Fixes

• Remove stray breakpoint (6417) `Thomas Grainger`_

• Fix API JSON MIME type (6397) `Jacob Tomlinson`_

• Remove wrong assert in handle compute (6370) `Florian Jetter`_

• Ensure multiple clients can cancel their key without interference (6016) `Florian Jetter`_

• Fix Nanny shutdown assertion (6357) `Gabe Joseph`_

• Fix fail_hard for sync functions (6269) `Gabe Joseph`_

• Prevent infinite transition loops; more aggressive validate_state() (6318) `crusaderky`_

• Ensure cleanup of many GBs of spilled data on terminate (6280) `crusaderky`_

• Fix WORKER_ANY_RUNNING regression (6297) `Florian Jetter`_

• Race conditions from fetch to compute while AMM requests replica (6248) `Florian Jetter`_

• Ensure resumed tasks are not accidentally forgotten (6217) `Florian Jetter`_

• Do not allow closing workers to be awaited again (5910) `Florian Jetter`_

Deprecations

• Move wait_for_signals to private module and deprecate distributed.cli.utils (6367) `Hendrik Makait`_

Documentation

• Fix typos and whitespace in worker.py (6326) `Hendrik Makait`_

• Fix link to memory trimming documentation (6317) `Marco Wolsza`_

Maintenance

• Make gen_test show up in VSCode test discovery (6424) `Gabe Joseph`_

• WSMR / deserialize_task (6411) `crusaderky`_

• Restore signal handlers after wait for signals is done (6400) `Thomas Grainger`_

• fail_hard should reraise (6399) `crusaderky`_

• Revisit tests mocking gather_dep (6385) `crusaderky`_

• Fix flaky test_in_flight_lost_after_resumed (6372) `Florian Jetter`_

• Restore install_signal_handlers due to downstream dependencies (6366) `Hendrik Makait`_

• Improve catch_unhandled_exceptions (6358) `Gabe Joseph`_

• Remove all invocations of IOLoop.run_sync from CLI (6205) `Hendrik Makait`_

• Remove transition-counter-max from config (6349) `crusaderky`_

• Use list comprehension in pickle_loads (6343) `jakirkham`_

• Improve ensure_memoryview test coverage & make minor fixes (6333) `jakirkham`_

• Remove leaking reference to workers from gen_cluster (6337) `Hendrik Makait`_

• Partial annotations for stealing.py (6338) `crusaderky`_

• Validate and debug state machine on handle_compute_task (6327) `crusaderky`_

• Bump pyupgrade and clean up # type: ignore (6293) `crusaderky`_

• gen_cluster to write to /tmp (6335) `crusaderky`_

• Transition table as a ClassVar (6331) `crusaderky`_

• Simplify ensure_memoryview test with array (6322) `jakirkham`_

• Refactor ensure_communicating (6165) `crusaderky`_

• Review scheduler annotations, part 2 (6253) `crusaderky`_

• Use w for writeable branch in pickle_loads (6314) `jakirkham`_

• Simplify frame handling in ws (6294) `jakirkham`_

• Use ensure_bytes from dask.utils (6295) `jakirkham`_

• Use ensure_memoryview in array deserialization (6300) `jakirkham`_

• Escape < > when generating Junit report (6306) `crusaderky`_

• Use codecs.decode to deserialize errors (6274) `jakirkham`_

• Minimize copying in maybe_compress & byte_sample (6273) `jakirkham`_

• Skip test_release_evloop_while_spilling on OSX (6291) `Florian Jetter`_

• Simplify logic in get_default_compression (6260) `jakirkham`_

• Cleanup old compression workarounds (6259) `jakirkham`_

• Re-enable NVML monitoring for WSL (6119) `Charles Blackmon-Luca`_

Released on May 2, 2022

Highlights

This is a bugfix release for this issue.

Enhancements

• Handle writeable in buffer_callback (6238) `jakirkham`_

• Use .data with NumPy array allocation (6242) `jakirkham`_

Bug Fixes

• Close executor in event loop if interpreter is closing (6256) `Matthew Rocklin`_

Released on April 29, 2022

Enhancements

• Unblock event loop while waiting for ThreadpoolExecutor to shut down (6091) `Florian Jetter`_

• RetireWorker policy is done if removed (6234) `Gabe Joseph`_

• Pause to disable dependency gathering (6195) `crusaderky`_

• Add EOFError to nanny multiprocessing.queue except list (6213) `Matthew Rocklin`_

• Re-interpret error in lost worker scenario (6193) `Matthew Rocklin`_

• Add Stimulus IDs to Scheduler (6161) `Florian Jetter`_

• Set a five minute TTL for Dask workers (6200) `Matthew Rocklin`_

• Add distributed.metrics.monotonic (6181) `crusaderky`_

• Send worker validation errors to scheduler and err on test completion (6192) `Matthew Rocklin`_

• Redesign worker exponential backoff on busy-gather (6173) `crusaderky`_

• Log all invalid worker transitions to scheduler (6134) `Matthew Rocklin`_

• Make Graph dashboard plot have invisible axes (6149) `Matthew Rocklin`_

• Remove Nanny auto_restart state (6138) `Matthew Rocklin`_

Bug Fixes

• Ensure scheduler events do not hold on to TaskState objects (6226) `Florian Jetter`_

• Allow pausing and choke event loop while spilling (6189) `crusaderky`_

• Do not use UUID in stealing (6179) `Florian Jetter`_

• Handle int worker names in info page (6158) `Brett Naul`_

• Fix psutil dio counters none (6093) `ungarj`_

• Join Nanny watch thread (6146) `Matthew Rocklin`_

• Improve logging when closing workers (6129) `Matthew Rocklin`_

• Avoid stack overflow in profiling (6141) `Matthew Rocklin`_

• Clean up SSHCluster if failure to start (6130) `Matthew Rocklin`_

Deprecations

• Deprecate rpc synchronous context manager (6171) `Thomas Grainger`_

Documentation

• Update actors.rst (6167) `Scott Sievert`_

Maintenance

• Add fail_hard decorator for worker methods (6210) `Matthew Rocklin`_

• Do not require pytest_timeout (6224) `Florian Jetter`_

• Remove remaining run_sync calls from tests (6196) `Thomas Grainger`_

• Increase test timeout if debugger is running (6218) `Florian Jetter`_

• Do not list closes keyword in list of bullet points (6219) `Florian Jetter`_

• Harmonize (6161) and (6173) (6207) `crusaderky`_

• Xfail test_worker_death_timeout (6186) `Matthew Rocklin`_

• Use random port in test_dask_spec.py::test_text (6187) `Matthew Rocklin`_

• Mark all websocket tests as flaky (6188) `Matthew Rocklin`_

• Fix flaky test_dont_steal_long_running_tasks (6197) `crusaderky`_

• Cleanup names in stealing (6185) `Matthew Rocklin`_

• log_errors decorator (6184) `crusaderky`_

• Pass mypy validation on Windows (6180) `crusaderky`_

• Add locket as a dependency instead of vendoring (6166) `Michael Adkins`_

• Remove unittestmock for gather_dep and get_data_from_worker (6172) `Florian Jetter`_

• mypy tweaks (6175) `crusaderky`_

• Avoid easy deprecated calls to asyncio.get_event_loop() (6170) `Thomas Grainger`_

• Fix flaky test_cancel_fire_and_forget (6099) `crusaderky`_

• Remove deprecated code (6144) `Matthew Rocklin`_

• Update link of test badge (6154) `Florian Jetter`_

• Remove legacy state mappings (6145) `Matthew Rocklin`_

• Fix test_worker_waits_for_scheduler (6155) `Matthew Rocklin`_

• Disallow leaked threads on windows (6152) `Thomas Grainger`_

• Review annotations and docstrings in scheduler.py, part 1 (6132) `crusaderky`_

• Relax test_asyncprocess.py::test_simple (6150) `Matthew Rocklin`_

• Drop cast ing which is effectively a no-op (6101) `jakirkham`_

• Mark tests that use a specific port as flaky (6139) `Matthew Rocklin`_

• AMM Suggestion namedtuples (6108) `crusaderky`_

Released on April 15, 2022

New Features

• Add KillWorker Plugin (6126) `Matthew Rocklin`_

Enhancements

• Sort worker list in info pages (6135) `Matthew Rocklin`_

• Add back Worker.transition_fetch_missing (6112) `Matthew Rocklin`_

• Log state machine events (6092) `crusaderky`_

• Migrate ensure_executing transitions to new WorkerState event mechanism - part 1 (6003) `crusaderky`_

• Migrate ensure_executing transitions to new WorkerState event mechanism - part 2 (6062) `crusaderky`_

• Annotate worker transitions to error (6012) `crusaderky`_

Bug Fixes

• Avoid transitioning from memory/released to missing in worker (6123) `Matthew Rocklin`_

• Don't try to reconnect client on interpreter shutdown (6120) `Matthew Rocklin`_

• Wrap UCX init warnings in importable functions (6121) `Charles Blackmon-Luca`_

• Cancel asyncio tasks on worker close (6098) `crusaderky`_

• Avoid port collisions when defining port ranges (6054) `crusaderky`_

Maintenance

• Avoid intermittent failure in test_cancel_fire_and_forget (6131) `Matthew Rocklin`_

• Ignore bokeh warning in pytest (6127) `Matthew Rocklin`_

• Start uncythonization (6104) `Martin Durant`_

• Avoid redundant cleanup fixture in gen_test tests (6118) `Thomas Grainger`_

• Move comm.close to finally in test_comms (6109) `Florian Jetter`_

• Use async with Server in test_core.py (6100) `Matthew Rocklin`_

• Elevate warnings to errors in the test suite (6094) `Thomas Grainger`_

• Add urllib3 to nightly conda builds (6102) `James Bourbeau`_

• Drop Blosc (6027) `Matthew Rocklin`_

• Robust test_get_returns_early (6090) `Florian Jetter`_

• Overhaul test_priorities.py (6077) `crusaderky`_

• Remove pytest-asyncio (6063) `Thomas Grainger`_

• Clean up usage around plain rpc (6082) `Florian Jetter`_

• Drop OSX builds for Python 3.9 (6073) `Florian Jetter`_

• Bump periods in utils_test.wait_for (6081) `Florian Jetter`_

• Check for ucx-py nightlies when updating gpuCI (6006) `Charles Blackmon-Luca`_

• Type annotations for profile.py (6067) `crusaderky`_

• Fix flaky test_worker_time_to_live (6061) `crusaderky`_

• Fix flaky test_as_completed_async_for_cancel (6072) `crusaderky`_

• Fix regression in test_weakref_cache (6033) `crusaderky`_

• Trivial fix to test_nanny_worker_port_range (6070) `crusaderky`_

• Drop deprecated tornado.netutil.ExecutorResolver (6031) `Thomas Grainger`_

• Delete asyncio.py (6066) `Thomas Grainger`_

• Tweak conda environment files (6037) `crusaderky`_

• Harden test_abort_execution_to_fetch and more (6026) `crusaderky`_

• Fix test_as_completed_with_results_no_raise and name comm (6042) `Matthew Rocklin`_

• Use more robust limits in test_worker_memory (6055) `Florian Jetter`_

Released on April 1, 2022

Note

This is the first release with support for Python 3.10

New Features

• Add Python 3.10 support (5952) `Thomas Grainger`_

• New cluster dump utilities (5920) `Simon Perkins`_

• New ClusterDump SchedulerPlugin for dumping cluster state on close (5983) `Simon Perkins`_

• Track Event Loop intervals in dashboard plot (5964) `Matthew Rocklin`_

• ToPickle - Unpickle on the Scheduler (5728) `Mads R. B. Kristensen`_

Enhancements

• Retry on transient error codes in preload (5982) `Matthew Rocklin`_

• More idiomatic mypy configuration (6022) `crusaderky`_

• Name extensions and enable extension heartbeats (5957) `Matthew Rocklin`_

• Better error message on misspelled executor annotation (6009) `crusaderky`_

• Clarify that SchedulerPlugin must be subclassed (6008) `crusaderky`_

• Remove duplication from stealing (5787) `Duncan McGregor`_

• Remove cache in iscoroutinefunction to avoid holding on to refs (5985) `Florian Jetter`_

• Add title to individual plots (5967) `Matthew Rocklin`_

• Specify average in timeseries titles (5974) `Matthew Rocklin`_

Bug Fixes

• Do not catch CancelledError in CommPool (6005) `Florian Jetter`_

Deprecations

• Remove distributed._ipython_utils and dependents (6036) `Thomas Grainger`_

• Remove support for PyPy (6029) `James Bourbeau`_

• Drop runtime dependency to setuptools (6017) `crusaderky`_

• Remove heartbeats from events (5989) `Matthew Rocklin`_

Documentation

• Mention default value of Client's timeout (5933) `Eric Engestrom`_

• Update celery and other outdated 3rd party URLs (5988) `Thomas Grainger`_

Maintenance

• Improve test_hardware test (6039) `Matthew Rocklin`_

• Short variant of test_report.html (6034) `crusaderky`_

• Make test_reconnect async (6000) `Matthew Rocklin`_

• Update gpuCI RAPIDS_VER to 22.06 (5962)

• Add tiny test for ToPickle (6021) `Matthew Rocklin`_

• Remove check_python_3 (broken with click>=8.1.0) (6018) `Thomas Grainger`_

• Fix black in CI (6019) `crusaderky`_

• Add a hardware benchmark to test memory, disk, and network bandwidths (5966) `Matthew Rocklin`_

• Relax variable test_race (5993) `Matthew Rocklin`_

• Skip dask-ssh tests without paramiko (5907) `Elliott Sales de Andrade`_

• Remove test_restart_sync_no_center (5994) `Matthew Rocklin`_

• Set lower tick frequency in tests (5977) `Matthew Rocklin`_

• Catch NotADirectoryError in SafeTemporaryDirectory (5984) `Florian Jetter`_

• Fix flaky test_weakref_cache (5978) `crusaderky`_

• Fixup test_worker_doesnt_await_task_completion (5979) `Matthew Rocklin`_

• Use broader range in test_nanny_worker_port_range (5980) `Matthew Rocklin`_

• Use tempfile directory in cluster fixture (5825) `Florian Jetter`_

• Drop setuptools from distributed recipe (5963) `jakirkham`_

Released on March 18, 2022

New Features

• Support dumping cluster state to URL (5863) `Gabe Joseph`_

Enhancements

• Prevent data duplication on unspill (5936) `crusaderky`_

• Encapsulate spill buffer and memory_monitor (5904) `crusaderky`_

• Drop pkg_resources in favour of importlib.metadata (5923) `Thomas Grainger`_

• Worker State Machine refactor: redesign TaskState and scheduler messages (5922) `crusaderky`_

• Tidying of OpenSSL 1.0.2/Python 3.9 (and earlier) handling (5854) `jakirkham`_

• zict type annotations (5905) `crusaderky`_

• Add key to compute failed message (5928) `Florian Jetter`_

• Change default log format to include timestamp (5897) `Florian Jetter`_

• Improve type annotations in worker.py (5814) `crusaderky`_

Bug Fixes

• Fix progress_stream teardown (5823) `Thomas Grainger`_

• Handle concurrent or failing handshakes in InProcListener (5903) `Thomas Grainger`_

• Make log_event threadsafe (5946) `Gabe Joseph`_

Documentation

• Fixes to documentation regarding plugins (5940) `crendoncoiled`_

• Some updates to scheduling policies docs (5911) `Gabe Joseph`_

Maintenance

• Fix test_nanny_worker_port_range hangs on Windows (5956) `crusaderky`_

• (REVERTED) Unblock event loop while waiting for ThreadpoolExecutor to shut down (5883) `Florian Jetter`_

• Revert 5883 (5961) `crusaderky`_

• Invert event_name check in test-report job (5959) `jakirkham`_

• Only run gh-pages workflow on dask/distributed (5942) `jakirkham`_

• absolufy-imports - No relative imports - PEP8 (5924) `Florian Jetter`_

• Fix track_features for distributed pre-releases (5927) `Charles Blackmon-Luca`_

• Xfail test_submit_different_names (5916) `Florian Jetter`_

• Fix distributed pre-release's distributed-impl constraint (5867) `Charles Blackmon-Luca`_

• Mock process memory readings in test_worker.py (v2) (5878) `crusaderky`_

• Drop unused _round_robin global variable (5881) `jakirkham`_

• Add GitHub URL for PyPi (5886) `Andrii Oriekhov`_

• Mark xfail COMPILED tests skipif instead (5884) `Florian Jetter`_

Released on February 25, 2022

New Features

• Add the ability for Client to run preload code (5773) `Bryan W. Weber`_

Enhancements

• Optionally use NumPy to allocate buffers (5750) `jakirkham`_

• Add git hash to distributed-impl version (5865) `Charles Blackmon-Luca`_

• Immediately raise exception when trying to connect to a closed cluster (5855) `Florian Jetter`_

• Lazily get dask version information (5822) `Thomas Grainger`_

• Remove the requirements to add comm to every handler (5820) `Florian Jetter`_

• Raise on unclosed comms in check_instances (5836) `Florian Jetter`_

• Python 3.8 f-strings (5828) `crusaderky`_

• Constrained spill (5543) `Naty Clementi`_

• Measure actual spilled bytes, not output of sizeof() (5805) `crusaderky`_

• Remove redundant str() conversions (5810) `crusaderky`_

• Cluster dump now excludes run_spec by default (5803) `Florian Jetter`_

• Dump more objects with dump_cluster_state (5806) `crusaderky`_

• Do not connect to any sockets on import (5808) `Florian Jetter`_

Bug Fixes

• Avoid deadlock when two tasks are concurrently waiting for an unresolved ActorFuture (5709) `Thomas Grainger`_

Deprecations

• Drop Python 3.7 (5683) `James Bourbeau`_

• Remove support for UCX < 1.11.1 (5859) `Peter Andreas Entschev`_

Documentation

• Fix typo in memory types documentation relative links (5845) `James Bourbeau`_

• Document and test spill->target hysteresis cycle (5813) `crusaderky`_

Maintenance

• Fix flaky test_remove_replicas_while_computing (5860) `crusaderky`_

• Fix time based test_assert_worker_story_malformed_story parameterize (5856) `Thomas Grainger`_

• Remove xfail from test_no_unnecessary_imports_on_worker (5862) `crusaderky`_

• Start building pre-releases with cythonized scheduler (5831) `Charles Blackmon-Luca`_

• Do not mark tests xfail if they don't come up in time (5824) `Florian Jetter`_

• Use gen_cluster where possible in test_dask_worker.py (5842) `Florian Jetter`_

• Generate junit report when pytest-timeout kills pytest (5832) `crusaderky`_

• Decrease timeout-minutes for GHA jobs (5837) `Florian Jetter`_

• Fix some timeouts (5647) `Florian Jetter`_

• Bump pre-release version to be greater than stable releases (5816) `Charles Blackmon-Luca`_

• Do not run schedule jobs on forks (5821) `Florian Jetter`_

• Remove pillow<9 pin in CI (5775) `Thomas Grainger`_

• Show scheduled test runs in report (5812) `Ian Rose`_

• Add obvious exclusions with pragma statement (5801) `Sarah Charlotte Johnson`_

• Add coverage exclusions for cli files (5800) `Sarah Charlotte Johnson`_

• Add pragma statements (5749) `Sarah Charlotte Johnson`_

• Remove pragma: no cover from distributed.cli.dask_ssh (5809) `Thomas Grainger`_

• Add pragma - worker.py, client.py, stealing.py (5827) `Sarah Charlotte Johnson`_

• Relax distributed / dask-core dependencies for pre-releases (5802) `Charles Blackmon-Luca`_

• Remove test_ucx_config_w_env_var flaky condition (5765) `Peter Andreas Entschev`_

Released on February 11, 2022

Note

This is the last release with support for Python 3.7

Enhancements

• Update client.scheduler_info in wait_for_workers (5785) `Matthew Rocklin`_

• Increase robustness to TimeoutError during connect (5096) `Florian Jetter`_

• Respect KeyboardInterrupt in sync (5758) `Thomas Grainger`_

• Add workflow / recipe to generate Dask/distributed pre-releases (5636) `Charles Blackmon-Luca`_

• Review Scheduler / Worker display repr (5746) `crusaderky`_

• AMM: Graceful Worker Retirement (5381) `crusaderky`_

• AMM: tentatively stabilize flaky tests around worker pause (5735) `crusaderky`_

• AMM: speed up and stabilize test_memory (5737) `crusaderky`_

• Defer pandas import on worker in P2P shuffle (5695) `Gabe Joseph`_

Bug Fixes

• Fix for distributed.worker.memory.target=False and spill=0.7 (5788) `crusaderky`_

• Transition flight to missing if no who_has (5653) `Florian Jetter`_

Deprecations

• Remove deprecated ncores (5780) `crusaderky`_

• Deprecate registering plugins by class (5699) `Thomas Grainger`_

• Deprecate --nprocs option for dask-worker CLI (5641) `Bryan W. Weber`_

Documentation

• Fix imbalanced backticks (5784) `Matthias Bussonnier`_

Maintenance

• xfail test_worker_reconnects_mid_compute (5797) `crusaderky`_

• Fix linting CI build (5794) `James Bourbeau`_

• Update pre-commit versions (5782) `James Bourbeau`_

• Reactivate pytest_resourceleaks (5771) `crusaderky`_

• Set test assumption for test_client_timeout (5790) `Florian Jetter`_

• Remove client timeout from test_ucx_config_w_env_var (5792) `Florian Jetter`_

• Remove test_failed_worker_without_warning (5789) `Florian Jetter`_

• Fix longitudinal report (5783) `Ian Rose`_

• Fix flaky test_robust_to_bad_sizeof_estimates (5753) `crusaderky`_

• Revert "Pin coverage to 6.2 (5716)" (5770) `Thomas Grainger`_

• Trigger test runs periodically to increases failure statistics (5769) `Florian Jetter`_

• More fault tolerant test report (5732) `Ian Rose`_

• Pin pillow<9 to work around torch incompatibility (5755) `Thomas Grainger`_

• Overhaul check_process_leak (5739) `crusaderky`_

• Fix flaky test_exit_callback test (5713) `Jim Crist-Harif`_

• Generate tests summary (5710) `crusaderky`_

• Upload different architectured pre-releases separately (5741) `Charles Blackmon-Luca`_

• Ignore non-test directories (5720) `Gabe Joseph`_

• Bump gpuCI PYTHON_VER to 3.9 (5738) `Charles Blackmon-Luca`_

• Regression: threads noted down before they start (5796) `crusaderky`_

Released on January 28, 2022

New Features

• P2P shuffle skeleton (5520) `Gabe Joseph`_

Enhancements

• Fix <Task pending name='...' coro=<Client._handle_report()> (5721) `Thomas Grainger`_

• Add distributed.client.security-loader config (5693) `Jim Crist-Harif`_

• Avoid Client._handle_report cancelling itself on Client._close (5672) `Thomas Grainger`_

• Paused workers shouldn't steal tasks (5665) `crusaderky`_

• Add option for timestamps from output of Node.get_logs (4932) `Charles Blackmon-Luca`_

• Don't use time.time() or IOLoop.time() (5661) `crusaderky`_

Bug Fixes

• Raise plugin exceptions on Worker.start() (4298) `Peter Andreas Entschev`_

Documentation

• Fixing docstrings (5696) `Julia Signell`_

• Fix typo in Client.run docstring (5687) `Thomas Grainger`_

• Update client.py docstrings (5670) `Tim Harris`_

Maintenance

• Skip shuffle tests if pandas / dask.dataframe not installed (5730) `James Bourbeau`_

• Improve test coverage (5655) `Sarah Charlotte Johnson`_

• Test report improvements (5714) `Ian Rose`_

• P2P shuffle: ignore row order in tests (5706) `Gabe Joseph`_

• Fix flaky test_no_reconnect[--no-nanny] (5686) `Thomas Grainger`_

• Pin coverage to 6.2 (5716) `Thomas Grainger`_

• Check for new name of timeouts artifact and be more fault tolerant (5707) `Ian Rose`_

• Revisit rebalance unit tests (5697) `crusaderky`_

• Update comment in rearrange_by_column_p2p (5701) `James Bourbeau`_

• Update gpuCI RAPIDS_VER to 22.04 (5676)

• Fix groupby test after meta requirements got stricter in Dask PR#8563 (5694) `Julia Signell`_

• Fix flaky test_close_gracefully and test_lifetime (5677) `crusaderky`_

• Fix flaky test_workspace_concurrency (5690) `crusaderky`_

• Fix flaky test_shuffle_extension.py::test_get_partition (5689) `Gabe Joseph`_

• Fix flaky test_dump_cluster_unresponsive_remote_worker (5679) `crusaderky`_

• Dump cluster state on all test failures (5674) `crusaderky`_

• Update license format (5652) `James Bourbeau`_

• Fix flaky test_drop_with_paused_workers_with_running_tasks_3_4 (5673) `crusaderky`_

• Do not raise an exception if the GitHub token cannot be found (5668) `Florian Jetter`_