NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #1690 most downloaded on PyPI
Distributed scheduler for Dask
Last release 1 months ago
24 Aug 2026
Ships fairly regularly
a new release about every 5 weeks
Nearly every release is documented
notes for 59 of the last 60 stable releases
1 version withdrawn
withdrawn after publishing
11 years old
269 releases · first in 2015
Fix excess threading on missing connections (2403) Daniel Farrell_
Fix excess threading on missing connections (2403) `Daniel Farrell`_
Fix typo in doc (2457) `Loïc Estève`_
Start fewer but larger workers with LocalCluster (2452) `Matthew Rocklin`_
Check for non-zero length first in read loop (2465) `John Kirkham`_
DOC: Use of local cluster in script (2462) `Peter Killick`_
DOC/API: Signature for base class write / read (2472) `Tom Augspurger`_
Support Pytest 4 in Tests (2478) `Adam Beberg`_
Ensure async behavior in event loop with LocalCluster (2484) `Matthew Rocklin`_
Fix spurious CancelledError (2485) `Loïc Estève`_
Properly reset dask.config scheduler and shuffle when closing the client (2475) `George Sakkis`_
Make it more explicit that resources are per worker. (2470) `Loïc Estève`_
Remove references to center (2488) `Matthew Rocklin`_
Expand client clearing timeout to 10s in testing (2493) `Matthew Rocklin`_
Propagate key keyword in progressbar (2492) `Matthew Rocklin`_
Use provided cluster's IOLoop if present in Client (2494) `Matthew Rocklin`_
One column per quarter.
Clean up LocalCluster logging better in async mode (2448) Matthew Rocklin_
Clean up LocalCluster logging better in async mode (2448) `Matthew Rocklin`_
Add short error message if bokeh cannot be imported (2444) `Dirk Petersen`_
Add optional environment variables to Nanny (2431) `Matthew Rocklin`_
Make the direct keyword docstring entries uniform (2441) `Matthew Rocklin`_
Make LocalCluster.close async friendly (2437) `Matthew Rocklin`_
gather_dep: don't request dependencies we already found out we don't want (2428) `tjb900`_
Add parameters to Client.run docstring (2429) `Matthew Rocklin`_
Support coroutines and async-def functions in run/run_scheduler (2427) `Matthew Rocklin`_
Name threads in ThreadPoolExecutors (2408) `Matthew Rocklin`_
Serialize numpy.ma.masked objects properly (2384) Jim Crist_
Serialize numpy.ma.masked objects properly (2384) `Jim Crist`_
Turn off bokeh property validation in dashboard (2387) `Jim Crist`_
Fully initialize WorkerState objects (2388) `Jim Crist`_
Fix typo in scheduler docstring (2393) `Russ Bubley`_
DOC: fix typo in distributed.worker.Worker docstring (2395) `Loïc Estève`_
Remove clients and workers from event log after removal (2394) `tjb900`_
Support msgpack 0.6.0 by providing length keywords (2399) `tjb900`_
Use async-await on large messages test (2404) `Matthew Rocklin`_
Fix race condition in normalize_collection (2386) `Jim Crist`_
Fix redict collection after HighLevelGraph fix upstream (2413) `Matthew Rocklin`_
Add a blocking argument to Lock.acquire() (2412) `Stephan Hoyer`_
Fix long traceback test (2417) `Matthew Rocklin`_
Update x509 certificates to current OpenSSL standards. (2418) `Diane Trout`_
Fixed the 404 error on the Scheduler Dashboard homepage (2361) Michael Wheeler_
Fixed the 404 error on the Scheduler Dashboard homepage (2361) `Michael Wheeler`_
Consolidate two Worker classes into one (2363) `Matthew Rocklin`_
Avoid warnings in pyarrow and msgpack (2364) `Matthew Rocklin`_
Avoid race condition in Actor's Future (2374) `Matthew Rocklin`_
Support missing packages keyword in Client.get_versions (2379) `Matthew Rocklin`_
Fixup serializing masked arrays (2373) `Jim Crist`_
Avoid deprecation warnings (2355) (2357) Matthew Rocklin_
Add support for Bokeh 1.0 (2348) (2356) `Matthew Rocklin`_
Fix regression that dropped support for Tornado 4 (2353) `Roy Wedge`_
Avoid deprecation warnings (2355) (2357) `Matthew Rocklin`_
Fix typo in worker documentation (2349) `Tom Rochette`_
Use tornado's builtin AnyThreadLoopEventPolicy (2326) Matthew Rocklin_
Use tornado's builtin AnyThreadLoopEventPolicy (2326) `Matthew Rocklin`_
Adjust TLS tests for openssl 1.1 (2331) `Marius van Niekerk`_
Avoid setting event loop policy if within Jupyter notebook server (2343) `Matthew Rocklin`_
Add preload script to conf (2325) `Guillaume Eynard-Bontemps`_
Add serializer for Numpy masked arrays (2335) `Peter Killick`_
Use psutil.Process.oneshot (2339) `NotSqrt`_
Use worker SSL context when getting client from worker. (2301) Anonymous
Remove Joblib Dask Backend from codebase (2298) Matthew Rocklin_
Remove Joblib Dask Backend from codebase (2298) `Matthew Rocklin`_
Include worker tls protocol in Scheduler.restart (2295) `Matthew Rocklin`_
Adapt to new Bokeh selection for 1.0 (2292) `Matthew Rocklin`_
Add explicit retry method to Future and Client (2299) `Matthew Rocklin`_
Point to main worker page in bokeh links (2300) `Matthew Rocklin`_
Limit concurrency when gathering many times (2303) `Matthew Rocklin`_
Add tls_cluster pytest fixture (2302) `Matthew Rocklin`_
Convert ConnectionPool.open and active to properties (2304) `Matthew Rocklin`_
change export_tb to format_tb (2306) `Eric Ma`_
Redirect joblib page to dask-ml (2307) `Matthew Rocklin`_
Include unserializable object in error message (2310) `Matthew Rocklin`_
Import Mapping, Iterator, Set from collections.abc in Python 3 (2315) `Gaurav Sheni`_
Extend Client.scatter docstring (2320) `Eric Ma`_
Update for new flake8 (2321) `Matthew Rocklin`_
Err in dask serialization if not a NotImplementedError (2251) Matthew Rocklin_
Err in dask serialization if not a NotImplementedError (2251) `Matthew Rocklin`_
Protect against key missing from priority in GraphLayout (2259) `Matthew Rocklin`_
Do not pull data twice in Client.gather (2263) `Adam Klein`_
Add pytest fixture for cluster tests (2262) `Matthew Rocklin`_
Cleanup bokeh callbacks (2261) (2278) `Matthew Rocklin`_
Fix bokeh error for memory_limit=None (2255) `Brett Naul`_
Place large keywords into task graph in Client.map (2281) `Matthew Rocklin`_
Remove redundant blosc threading code from protocol.numpy (2284) `Mike Gevaert`_
Add ncores to workertable (2289) `Matthew Rocklin`_
Support upload_file on files with no extension (2290) `Matthew Rocklin`_
Discard dependent rather than remove (2250) Matthew Rocklin_
Discard dependent rather than remove (2250) `Matthew Rocklin`_
Use dask_sphinx_theme `Matthew Rocklin`_
Drop the Bokeh index page (2241) `John Kirkham`_
Revert change to keep link relative (2242) `Matthew Rocklin`_
docs: Fix broken AWS link in setup.rst file (2240) `Vladyslav Moisieienkov`_
Return cancelled futures in as_completed (2233) `Chris White`_
Raise informative error when mixing futures between clients (2227) Matthew Rocklin_
Raise informative error when mixing futures between clients (2227) `Matthew Rocklin`_
add byte_keys to unpack_remotedata call (2232) `Matthew Rocklin`_
Add documentation for gist/rawgit for get_task_stream (2236) `Matthew Rocklin`_
Quiet Client.close by waiting for scheduler stop signal (2237) `Matthew Rocklin`_
Display system graphs nicely on different screen sizes (2239) `Derek Ludwig`_
Mutate passed in workers dict in TaskStreamPlugin.rectangles (2238) `Matthew Rocklin`_
Add direct_to_workers to Client Matthew Rocklin_
Add direct_to_workers to Client `Matthew Rocklin`_
Add Scheduler.proxy to workers `Matthew Rocklin`_
Implement Actors `Matthew Rocklin`_
Fix tooltip (2168) `Loïc Estève`_
Fix scale / avoid returning coroutines (2171) `Joe Hamman`_
Clarify dask-worker --nprocs (2173) `Yu Feng`_
Concatenate all bytes of small messages in TCP comms (2172) `Matthew Rocklin`_
Add dashboard_link property (2176) `Jacob Tomlinson`_
Always offload to_frames (2170) `Matthew Rocklin`_
Warn if desired port is already in use (2191) (2199) `Matthew Rocklin`_
Add profile page for event loop thread (2144) `Matthew Rocklin`_
Use dispatch for dask serialization, also add sklearn, pytorch (2175) `Matthew Rocklin`_
Handle corner cases with busy signal (2182) `Matthew Rocklin`_
Check self.dependencies when looking at tasks in memory (2196) `Matthew Rocklin`_
Add ability to log additional custom metrics from each worker (2169) `Loïc Estève`_
Fix formatting when port is a tuple (2204) `Loïc Estève`_
Describe what ZeroMQ is (2211) `Mike DePalatis`_
Tiny typo fix (2214) `Anderson Banihirwe`_
Add Python 3.7 to travis.yml (2203) `Matthew Rocklin`_
Add plot= keyword to get_task_stream (2198) `Matthew Rocklin`_
Add support for optional versions in Client.get_versions (2216) `Matthew Rocklin`_
Add routes for solo bokeh figures in dashboard (2185) `Matthew Rocklin`_
Be resilient to missing dep after busy signal (2217) `Matthew Rocklin`_
Use CSS Grid to layout status page on the dashboard (2213) `Derek Ludwig`_ and `Luke Canavan`_
Fix deserialization of queues on main ioloop thread (2221) `Matthew Rocklin`_
Add a worker initialization function (2201) `Guillaume Eynard-Bontemps`_
Collapse navbar in dashboard (2223) `Luke Canavan`_
Fix msgpack PendingDeprecationWarning for encoding='utf-8' (2153) Olivier Grisel_
Add worker_class= keyword to Nanny to support different worker types (2147) `Martin Durant`_
Cleanup intermittent worker failures (2152) (2146) `Matthew Rocklin`_
Fix msgpack PendingDeprecationWarning for encoding='utf-8' (2153) `Olivier Grisel`_
Make bokeh coloring deterministic using hash function (2143) `Matthew Rocklin`_
Allow client to query the task stream plot (2122) `Matthew Rocklin`_
Use PID and counter in thread names (2084) (2128) `Dror Birkman`_
Test that worker restrictions are cleared after cancellation (2107) `Matthew Rocklin`_
Expand resources in graph_to_futures (2131) `Matthew Rocklin`_
Add custom serialization support for pyarrow (2115) `Dave Hirschfeld`_
Update dask-scheduler cli help text for preload (2120) `Matt Nicolls`_
Added another nested parallelism test (1710) `Tom Augspurger`_
insert newline by default after TextProgressBar (1976) `Phil Tooley`_
Retire workers from scale (2104) `Matthew Rocklin`_
Allow worker to refuse data requests with busy signal (2092) `Matthew Rocklin`_
Don't forget released keys (2098) `Matthew Rocklin`_
Update example for stopping a worker (2088) `John Kirkham`_
removed hardcoded value of memory terminate fraction from a log message (2096) `Bartosz Marcinkowski`_
Adjust worker doc after change in config file location and treatment (2094) `Aurélien Ponte`_
Prefer gathering data from same host (2090) `Matthew Rocklin`_
Handle exceptions on deserialized comm with text error (2093) `Matthew Rocklin`_
Fix typo in docstring (2087) `Loïc Estève`_
Provide communication context to serialization functions (2054) `Matthew Rocklin`_
Allow name to be explicitly passed in publish_dataset (1995) `Marius van Niekerk`_
Avoid accessing Worker.scheduler_delay around yield point (2074) `Matthew Rocklin`_
Support TB and PB in format bytes (2072) `Matthew Rocklin`_
Add test for as_completed for loops in Python 2 (2071) `Matthew Rocklin`_
Allow adaptive to exist without a cluster (2064) `Matthew Rocklin`_
Have worker data transfer wait until recipient acknowledges (2052) `Matthew Rocklin`_
Support async def functions in Client.sync (2070) `Matthew Rocklin`_
Add asynchronous parameter to docstring of LocalCluster `Matthew Rocklin`_
Normalize address before comparison (2066) `Tom Augspurger`_
Use ConnectionPool for Worker.scheduler `Matthew Rocklin`_
Avoid reference cycle in str_graph `Matthew Rocklin`_
Pull data outside of while loop in gather (2059) `Matthew Rocklin`_
Replace deprecated workers.iloc with workers.values() (2013) Grant Jenks_
Overhaul configuration (1948) `Matthew Rocklin`_
Replace get= keyword with scheduler= (1959) `Matthew Rocklin`_
Use tuples in msgpack (2000) `Matthew Rocklin`_ and `Marius van Niekerk`_
Unify handling of high-volume connections (1970) `Matthew Rocklin`_
Automatically scatter large arguments in joblib connector (2020) (2030) `Olivier Grisel`_
Turn click Python 3 locales failure into a warning (2001) `Matthew Rocklin`_
Rely on dask implementation of sizeof (2042) `Matthew Rocklin`_
Replace deprecated workers.iloc with workers.values() (2013) `Grant Jenks`_
Introduce serialization families (1912) `Matthew Rocklin`_
Add PubSub (1999) `Matthew Rocklin`_
Add Dask stylesheet to documentation `Matthew Rocklin`_
Avoid recomputation on partially-complete results (1840) `Matthew Rocklin`_
Use sys.prefix in popen for testing (1954) `Matthew Rocklin`_
Include yaml files in manifest `Matthew Rocklin`_
Use self.sync so Client.processing works in asynchronous context (1962) `Henry Doupe`_
Fix bug with bad repr on closed client (1965) `Matthew Rocklin`_
Parse --death-timeout keyword in dask-worker (1967) `Matthew Rocklin`_
Support serializers in BatchedSend (1964) `Matthew Rocklin`_
Use normal serialization mechanisms to serialize published datasets (1972) `Matthew Rocklin`_
Add security support to LocalCluster. (1855) `Marius van Niekerk`_
add ConnectionPool.remove method (1977) `Tony Lorenzo`_
Cleanly close workers when scheduler closes (1981) `Matthew Rocklin`_
Add .pyz support in upload_file (1781) `@bmaisson`_
add comm to packages (1980) `Matthew Rocklin`_
Replace dask.set_options with dask.config.set `Matthew Rocklin`_
Exclude versions of sortedcontainers which do not have .iloc. (1993) `Russ Bubley`_
Exclude gc statistics under PyPy (1997) `Marius van Niekerk`_
Manage recent config and dataframe changes in dask (2009) `Matthew Rocklin`_
Cleanup lingering clients in tests (2012) `Matthew Rocklin`_
Use timeouts during Client._ensure_connected (2011) `Martin Durant`_
Avoid reference cycle in joblib backend (2014) `Matthew Rocklin`_, also `Olivier Grisel`_
DOC: fixed test example (2017) `Tom Augspurger`_
Add worker_key parameter to Adaptive (1992) `Matthew Rocklin`_
Prioritize tasks with their true keys, before stringifying (2006) `Matthew Rocklin`_
Serialize worker exceptions through normal channels (2016) `Matthew Rocklin`_
Include exception in progress bar (2028) `Matthew Rocklin`_
Avoid logging orphaned futures in All (2008) `Matthew Rocklin`_
Don't use spill-to-disk dictionary if we're not spilling to disk `Matthew Rocklin`_
Only avoid recomputation if key exists (2036) `Matthew Rocklin`_
Use client connection and serialization arguments in progress (2035) `Matthew Rocklin`_
Rejoin worker client on closing context manager (2041) `Matthew Rocklin`_
Avoid forgetting erred tasks when losing dependencies (2047) `Matthew Rocklin`_
Avoid collisions in graph_layout (2050) `Matthew Rocklin`_
Avoid recursively calling bokeh callback in profile plot (2048) `Matthew Rocklin`_
Remove errant print statement (1957) Matthew Rocklin_
Remove errant print statement (1957) `Matthew Rocklin`_
Only add reevaluate_occupancy callback once (1953) `Tony Lorenzo`_
Newline needed for doctest rendering (1917) Loïc Estève_
Newline needed for doctest rendering (1917) `Loïc Estève`_
Support Client._repr_html_ when in async mode (1909) `Matthew Rocklin`_
Add parameters to dask-ssh command (1910) `Irene Rodriguez`_
Sanitize get_dataset trace (1888) `John Kirkham`_
Fix bug where queues would not clean up cleanly (1922) `Matthew Rocklin`_
Delete cached file safely in upload file (1921) `Matthew Rocklin`_
Accept KeyError when closing tornado IOLoop in tests (1937) `Matthew Rocklin`_
Quiet the client and scheduler when gather(..., errors='skip') (1936) `Matthew Rocklin`_
Clarify couldn't gather keys warning (1942) `Kenneth Koski`_
Support submit keywords in joblib (1947) `Matthew Rocklin`_
Avoid use of external resources in bokeh server (1934) `Matthew Rocklin`_
Drop __contains__ from Datasets (1889) `John Kirkham`_
Fix bug with queue timeouts (1950) `Matthew Rocklin`_
Replace msgpack-python by msgpack (1927) `Loïc Estève`_
Fix numeric environment variable configuration (1885) Joseph Atkins-Kurkish_
Fix numeric environment variable configuration (1885) `Joseph Atkins-Kurkish`_
support bytearrays in older lz4 library (1886) `Matthew Rocklin`_
Remove started timeout in nanny (1852) `Matthew Rocklin`_
Don't log errors in sync (1894) `Matthew Rocklin`_
downgrade stale lock warning to info logging level (1890) `Matthew Rocklin`_
Fix UnboundLocalError for key (1900) `John Kirkham`_
Resolve deployment issues in Python 2 (1905) `Matthew Rocklin`_
Support retries and priority in Client.get method (1902) `Matthew Rocklin`_
Add additional attributes to task page if applicable (1901) `Matthew Rocklin`_
Add count method to as_completed (1897) `Matthew Rocklin`_
Extend default timeout to 10s (1904) `Matthew Rocklin`_
Increase default allowable tick time to 3s (1854) Matthew Rocklin_
Increase default allowable tick time to 3s (1854) `Matthew Rocklin`_
Handle errant workers when another worker has data (1853) `Matthew Rocklin`_
Close multiprocessing queue in Nanny to reduce open file descriptors (1862) `Matthew Rocklin`_
Extend nanny started timeout to 30s, make configurable (1865) `Matthew Rocklin`_
Comment out the default config file (1871) `Matthew Rocklin`_
Update to fix bokeh 0.12.15 update errors (1872) `Matthew Rocklin`_
Downgrade Event Loop unresponsive warning to INFO level (1870) `Matthew Rocklin`_
Add fifo timeout to control priority generation (1828) `Matthew Rocklin`_
Add retire_workers API to Client (1876) `Matthew Rocklin`_
Catch NoSuchProcess error in Nanny.memory_monitor (1877) `Matthew Rocklin`_
Add uid to nanny queue communications (1880) `Matthew Rocklin`_
Avoid passing bytearrays to snappy decompression (1831) Matthew Rocklin_
Avoid passing bytearrays to snappy decompression (1831) `Matthew Rocklin`_
Specify IOLoop in Adaptive (1841) `Matthew Rocklin`_
Use connect-timeout config value throughout client (1839) `Matthew Rocklin`_
Support direct= keyword argument in Client.get (1845) `Matthew Rocklin`_
Add cluster superclass and improve adaptivity (1813) Matthew Rocklin_
Add cluster superclass and improve adaptivity (1813) `Matthew Rocklin`_
Fixup tests and support Python 2 for Tornado 5.0 (1818) `Matthew Rocklin`_
Fix bug in recreate_error when dependencies are dropped (1815) `Matthew Rocklin`_
Add worker time to live in Scheduler (1811) `Matthew Rocklin`_
Scale adaptive based on total_occupancy (1807) `Matthew Rocklin`_
Support calling compute within worker_client (1814) `Matthew Rocklin`_
Add percentage to profile plot (1817) `Brett Naul`_
Overwrite option for remote python in dask-ssh (1812) `Sven Kreiss`_
Fix bug where we didn't check idle/saturated when stealing (1801) Matthew Rocklin_
Fix bug where we didn't check idle/saturated when stealing (1801) `Matthew Rocklin`_
Fix bug where client was noisy when scheduler closed unexpectedly (1806) `Matthew Rocklin`_
Use string-based timedeltas (like '500 ms') everywhere (1804) `Matthew Rocklin`_
Keep logs in scheduler and worker even if silenced (1803) `Matthew Rocklin`_
Support minimum, maximum, wait_count keywords in Adaptive (1797) `Jacob Tomlinson`_ and `Matthew Rocklin`_
Support async protocols for LocalCluster, replace start= with asynchronous= (1798) `Matthew Rocklin`_
Avoid restarting workers when nanny waits on scheduler (1793) `Matthew Rocklin`_
Use IOStream.read_into() when available (1477) `Antoine Pitrou`_
Reduce LocalCluster logging threshold from CRITICAL to WARN (1785) `Andy Jones`_
Add futures_of to API docs (1783) `John Kirkham`_
Make diagnostics link in client configurable (1810) `Matthew Rocklin`_
Fixed an uncaught exception in distributed.joblib with a LocalCluster using only threads (1775) Tom Augspurger_
Fixed an uncaught exception in distributed.joblib with a LocalCluster using only threads (1775) `Tom Augspurger`_
Format bytes in info worker page (1752) `Matthew Rocklin`_
Add pass-through arguments for scheduler/worker --preload modules. (1634) `Alex Ford`_
Use new LZ4 API (1757) `Thrasibule`_
Replace dask.optimize with dask.optimization (1754) `Matthew Rocklin`_
Add graph layout engine and bokeh plot (1756) `Matthew Rocklin`_
Only expand name with --nprocs if name exists (1776) `Matthew Rocklin`_
specify IOLoop for stealing PeriodicCallback (1777) `Matthew Rocklin`_
Fixed distributed.joblib with no processes `Tom Augspurger`_
Use set.discard to avoid KeyErrors in stealing (1766) `Matthew Rocklin`_
Avoid KeyError when task has been released during steal (1765) `Matthew Rocklin`_
Add versions routes to avoid the use of run in Client.get_versions (1773) `Matthew Rocklin`_
Add write_scheduler_file to Client (1778) `Joe Hamman`_
Default host to tls:// if tls information provided (1780) `Matthew Rocklin`_
Deprecate Client.shutdown (1699) Matthew Rocklin_
Refactor scheduler to use TaskState objects rather than dictionaries (1594) `Antoine Pitrou`_
Plot CPU fraction of total in workers page (1624) `Matthew Rocklin`_
Use thread CPU time in Throttled GC (1625) `Antoine Pitrou`_
Fix bug with memory_limit=None (1639) `Matthew Rocklin`_
Add futures_of to top level api (1646) `Matthew Rocklin`_
Warn on serializing large data in Client (1636) `Matthew Rocklin`_
Fix intermittent windows failure when removing lock file (1652) `Antoine Pitrou`_
Add diagnosis and logging of poor GC Behavior (1635) `Antoine Pitrou`_
Add client-scheduler heartbeats (1657) `Matthew Rocklin`_
Return dictionary of worker info in retire_workers (1659) `Matthew Rocklin`_
Ensure dumps_function works with unhashable functions (1662) `Matthew Rocklin`_
Collect client name ids rom client-name config variable (1664) `Matthew Rocklin`_
Allow simultaneous use of --name and --nprocs in dask-worker (1665) `Matthew Rocklin`_
Add support for grouped adaptive scaling and adaptive behavior overrides (1632) `Alex Ford`_
Share scheduler RPC between worker and client (1673) `Matthew Rocklin`_
Allow retries= in ClientExecutor (1672) `@rqx`_
Improve documentation for get_client and dask.compute examples (1638) `Scott Sievert`_
Support DASK_SCHEDULER_ADDRESS environment variable in worker (1680) `Matthew Rocklin`_
Support tuple-keys in retries (1681) `Matthew Rocklin`_
Use relative links in bokeh dashboard (1682) `Matthew Rocklin`_
Make message log length configurable, default to zero (1691) `Matthew Rocklin`_
Deprecate Client.shutdown (1699) `Matthew Rocklin`_
Add warning in configuration docs to install pyyaml (1701) `Cornelius Riemenschneider`_
Handle nested parallelism in distributed.joblib (1705) `Tom Augspurger`_
Don't wait for Worker.executor to shutdown cleanly when restarting process (1708) `Matthew Rocklin`_
Add support for user defined priorities (1651) `Matthew Rocklin`_
Catch and log OSErrors around worker lock files (1714) `Matthew Rocklin`_
Remove worker prioritization. Coincides with changes to dask.order (1730) `Matthew Rocklin`_
Use process-measured memory rather than nbytes in Bokeh dashboard (1737) `Matthew Rocklin`_
Enable serialization of Locks (1738) `Matthew Rocklin`_
Support Tornado 5 beta (1735) `Matthew Rocklin`_
Cleanup remote_magic client cache after tests (1743) `Min RK`_
Allow service ports to be specified as (host, port) (1744) `Bruce Merry`_
Fix deprecation warning from logger.warn (1616) Brett Naul_
Clear deque handlers after each test (1586) `Antoine Pitrou`_
Handle deserialization in FutureState.set_error (1592) `Matthew Rocklin`_
Add process leak checker to tests (1596) `Antoine Pitrou`_
Customize process title for subprocess (1590) `Antoine Pitrou`_
Make linting a separate CI job (1599) `Antoine Pitrou`_
Fix error from get_client() with no global client (1595) `Daniel Li`_
Remove Worker.host_health, correct WorkerTable metrics (1600) `Matthew Rocklin`_
Don't mark tasks as suspicious when retire_workers called. Addresses (1607) `Russ Bubley`_
Do not include processing workers in workers_to_close (1609) `Russ Bubley`_
Disallow simultaneous scale up and down in Adaptive (1608) `Russ Bubley`_
Parse bytestrings in --memory-limit (1615) `Matthew Rocklin`_
Use environment variable for scheduler address if present (1610) `Matthew Rocklin`_
Fix deprecation warning from logger.warn (1616) `Brett Naul`_
Wrap import ssl statements with try-except block for ssl-crippled environments, (1570) Xander Johnson_
Wrap import ssl statements with try-except block for ssl-crippled environments, (1570) `Xander Johnson`_
Support zero memory-limit in Nanny (1571) `Matthew Rocklin`_
Avoid PeriodicCallback double starts (1573) `Matthew Rocklin`_
Add disposable workspace facility (1543) `Antoine Pitrou`_
Use format_time in task_stream plots (1575) `Matthew Rocklin`_
Avoid delayed finalize calls in compute (1577) `Matthew Rocklin`_
Doc fix about secede (1583) `Scott Sievert`_
Add tracemalloc option when tracking test leaks (1585) `Antoine Pitrou`_
Add JSON routes to Bokeh server (1584) `Matthew Rocklin`_
Handle exceptions cleanly in Variables and Queues (1580) `Matthew Rocklin`_
Drop use of pandas.msgpack (1473) Matthew Rocklin_
Drop use of pandas.msgpack (1473) `Matthew Rocklin`_
Add methods to get/set scheduler metadata `Matthew Rocklin`_
Add distributed lock `Matthew Rocklin`_
Add reschedule exception for worker tasks `Matthew Rocklin`_
Fix nbytes() for bytearrays `Matthew Rocklin`_
Capture scheduler and worker logs `Matthew Rocklin`_
Garbage collect after data eviction on high worker memory usage (1488) `Olivier Grisel`_
Add scheduler HTML routes to bokeh server (1478) (1514) `Matthew Rocklin`_
Add pytest plugin to test for resource leaks (1499) `Antoine Pitrou`_
Improve documentation for scheduler states (1498) `Antoine Pitrou`_
Correct warn_if_longer timeout in ThrottledGC (1496) `Fabian Keller`_
Catch race condition in as_completed on cancelled futures (1507) `Matthew Rocklin`_
Transactional work stealing (1489) (1528) `Matthew Rocklin`_
Avoid forkserver in PyPy (1509) `Matthew Rocklin`_
Add dict access to get/set datasets (1508) `Mike DePalatis`_
Support Tornado 5 (1509) (1512) (1518) (1534) `Antoine Pitrou`_
Move thread_state in Dask (1523) `Jim Crist`_
Use new Dask collections interface (1513) `Matthew Rocklin`_
Add nanny flag to dask-mpi `Matthew Rocklin`_
Remove JSON-based HTTP servers `Matthew Rocklin`_
Avoid doing I/O in repr/str (1536) `Matthew Rocklin`_
Fix URL for MPI4Py project (1546) `Ian Hopkinson`_
Allow automatic retries of a failed task (1524) `Antoine Pitrou`_
Clean and accelerate tests (1548) (1549) (1552) (1553) (1560) (1564) `Antoine Pitrou`_
Move HDFS functionality to the hdfs3 library (1561) `Jim Crist`_
Fix bug when using events page with no events (1562) `@rbubley`_
Improve diagnostic naming of tasks within tuples (1566) `Kelvyn Yang`_
Handle None case in profile.identity (1456)
Handle None case in profile.identity (1456)
Asyncio rewrite (1458)
Add rejoin function partner to secede (1462)
Nested compute (1465)
Use LooseVersion when comparing Bokeh versions (1470)
as_completed doesn't block on cancelled futures (1436)
as_completed doesn't block on cancelled futures (1436)
Notify waiting threads/coroutines on cancellation (1438)
Set Future(inform=True) as default (1437)
Rename Scheduler.transition_story to story (1445)
Future uses default client by default (1449)
Add keys= keyword to Client.call_stack (1446)
Add get_current_task to worker (1444)
Ensure that Client remains asynchronous before ioloop starts (1452)
Remove "click for worker page" in bokeh plot (1453)
Add Client.current() (1450)
Clean handling of restart timeouts (1442)
Fix tool issues with TaskStream plot (1425)
Fix tool issues with TaskStream plot (1425)
Move profile module to top level (1423)
Avoid storing messages in message log (1361)
Avoid storing messages in message log (1361)
fileConfig does not disable existing loggers (1380)
Offload upload_file disk I/O to separate thread (1383)
Add missing SSLContext (1385)
Collect worker thread information from sys._curent_frames (1387)
Add nanny timeout (1395)
Restart worker if memory use goes above 95% (1397)
Track workers memory use with psutil (1398)
Track scheduler delay times in workers (1400)
Add time slider to profile plot (1403)
Change memory-limit keyword to refer to maximum number of bytes (1405)
Add cancel(force=) keyword (1408)
Nothing published for this version
Silently pass on cancelled futures in as_completed (1366)
Silently pass on cancelled futures in as_completed (1366)
Fix unicode keys error in Python 2 (1370)
Support numeric worker names
Add dask-mpi executable (1367)
Clean up forgotten keys in fire-and-forget workloads (1250)
Clean up forgotten keys in fire-and-forget workloads (1250)
Handle missing extensions (1263)
Allow recreate_exception on persisted collections (1253)
Add asynchronous= keyword to blocking client methods (1272)
Restrict to horizontal panning in bokeh plots (1274)
Rename client.shutdown to client.close (1275)
Avoid blocking on event loop (1270)
Avoid cloudpickle errors for Client.get_versions (1279)
Yield on Tornado IOStream.write futures (1289)
Assume async behavior if inside a sync statement (1284)
Avoid error messages on closing (1297), (1296) (1318) (1319)
Add timeout= keyword to get_client (1290)
Respect timeouts when restarting (1304)
Clean file descriptor and memory leaks in tests (1317)
Deprecate Executor (1302)
Add timeout to ThreadPoolExecutor.shutdown (1330)
Clean up AsyncProcess handling (1324)
Allow unicode keys in Python 2 scheduler (1328)
Avoid leaking stolen data (1326)
Improve error handling on failed nanny starts (1337), (1331)
Make Adaptive more flexible
Support --contact-address and --listen-address in worker (1278)
Remove old dworker, dscheduler executables (1355)
Exit workers if nanny process fails (1345)
Auto pep8 and flake (1353)
Multi-threading safety (1191), (1228), (1229)
Multi-threading safety (1191), (1228), (1229)
Improve handling of byte counting (1198) (1224)
Add get_client, secede functions, refactor worker-client relationship (1201)
Allow logging configuration using logging.dictConfig() (1206) (1211)
Offload serialization and deserialization to separate thread (1218)
Support fire-and-forget tasks (1221)
Support bytestrings as keys (for Julia) (1234)
Resolve testing corner-cases (1236), (1237), (1240), (1241), (1242), (1244)
Automatic use of scatter/gather(direct=True) in more cases (1239)
Remove Python 3.4 testing from travis-ci (1157)
Remove Python 3.4 testing from travis-ci (1157)
Remove ZMQ Support (1160)
Fix memoryview nbytes issue in Python 2.7 (1165)
Re-enable counters (1168)
Improve scheduler.restart (1175)
Reevaluate worker occupancy periodically during scheduler downtime (1038) (1101)
Reevaluate worker occupancy periodically during scheduler downtime (1038) (1101)
Add AioClient asyncio-compatible client API (1029) (1092) (1099)
Update Keras serializer (1067)
Support TLS/SSL connections for security (866) (1034)
Always create new worker directory when passed --local-directory (1079)
Support pre-scattering data when using joblib frontend (1022)
Make workers more robust to failure of sizeof function (1108) and writing to disk (1096)
Add is_empty and update methods to as_completed (1113)
Remove _get coroutine and replace with get(..., sync=False) (1109)
Improve API compatibility with async/await syntax (1115) (1124)
Add distributed Queues (1117) and shared Variables (1128) to enable inter-client coordination
Support direct client-to-worker scattering and gathering (1130) as well as performance enhancements when scattering data
Style improvements for bokeh web dashboards (1126) (1141) as well as a removal of the external bokeh process
HTML reprs for Future and Client objects (1136)
Support nested collections in client.compute (1144)
Use normal client API in asynchronous mode (1152)
Remove old distributed.collections submodule (1153)
Add bokeh template files to MANIFEST (1063)
Add bokeh template files to MANIFEST (1063)
Don't set worker_client.get as default get (1061)
Clean up logging on Client().shutdown() (1055)
Support async with Client syntax (1053)
Support async with Client syntax (1053)
Use internal bokeh server for default diagnostics server (1047)
Improve styling of bokeh plots when empty (1046) (1037)
Support efficient serialization for sparse arrays (1040)
Prioritize newly arrived work in worker (1035)
Prescatter data with joblib backend (1022)
Make client.restart more robust to worker failure (1018)
Support preloading a module or script in dask-worker or dask-scheduler processes (1016)
Specify network interface in command line interface (1007)
Client.scatter supports a single element (1003)
Use blosc compression on all memoryviews passing through comms (998)
Add concurrent.futures-compatible Executor (997)
Add as_completed.batches method and return results (994) (971)
Allow worker_clients to optionally stay within the thread pool (993)
Add bytes-stored and tasks-processing diagnostic histograms (990)
Run supports non-msgpack-serializable results (965)
Use inproc transport in LocalCluster (919)
Use inproc transport in LocalCluster (919)
Add structured and queryable cluster event logs (922)
Use connection pool for inter-worker communication (935)
Robustly shut down spawned worker processes at shutdown (928)
Worker death timeout (940)
More visual reporting of exceptions in progressbar (941)
Render disk and serialization events to task stream visual (943)
Support async for / await protocol (952)
Ensure random generators are re-seeded in worker processes (953)
Upload sourcecode as zip module (886)
Replay remote exceptions in local process (894)
First come first served priorities on client submissions (840)
First come first served priorities on client submissions (840)
Can specify Bokeh internal ports (850)
Allow stolen tasks to return from either worker (853), (875)
Add worker resource constraints during execution (857)
Send small data through Channels (858)
Better estimates for SciPy sparse matrix memory costs (863)
Avoid stealing long running tasks (873)
Maintain fortran ordering of NumPy arrays (876)
Add --scheduler-file keyword to dask-scheduler (877)
Add serializer for Keras models (878)
Support uploading modules from zip files (886)
Improve titles of Bokeh dashboards (895)
Fix a bug where arrays with large dtypes or shapes were being improperly compressed (830 832 833)
Fix a bug where arrays with large dtypes or shapes were being improperly compressed (830 832 833)
Extend as_completed to accept new futures during iteration (829)
Add --nohost keyword to dask-ssh startup utility (827)
Support scheduler shutdown of remote workers, useful for adaptive clusters (:pr: 811 816 821)
Add Client.run_on_scheduler method for running debug functions on the scheduler (808)
Make compatible with Bokeh 0.12.4 (803)
Make compatible with Bokeh 0.12.4 (803)
Avoid compressing arrays if not helpful (777)
Optimize inter-worker data transfer (770) (790)
Add --local-directory keyword to worker (788)
Enable workers to arrive to the cluster with their own data. Useful if a worker leaves and comes back (785)
Resolve thread safety bug when using local_client (802)
Resolve scheduling issues in worker (804)
Major Scheduler refactor (717) (722) (724) (742) (743
Major Worker refactor (704)
Major Scheduler refactor (717) (722) (724) (742) (743
Add check (default is False) option to Client.get_versions to raise if the versions don't match on client, scheduler & workers (664)
Future.add_done_callback executes in separate thread (656)
Clean up numpy serialization (670)
Support serialization of Tornado v4.5 coroutines (673)
Use CPickle instead of Pickle in Python 2 (684)
Use Forkserver rather than Fork on Unix in Python 3 (687)
Support abstract resources for per-task constraints (694) (720) (737)
Add TCP timeouts (697)
Add embedded Bokeh server to workers (709) (713) (738)
Add embedded Bokeh server to scheduler (724) (736) (738)
Add more precise timers for Windows (713)
Add Versioneer (715)
Support inter-client channels (729) (749)
Scheduler Performance improvements (740) (760)
Improve load balancing and work stealing (747) (754) (757)
Run Tornado coroutines on workers
Avoid slow sizeof call on Pandas dataframes (758)
Nothing published for this version
Remove custom Bokeh export tool that implicitly relied on nodejs (655)
Remove custom Bokeh export tool that implicitly relied on nodejs (655)
Clean up scheduler logging (657)
Nothing published for this version
Add Client.get_versions() function to return software and package information from the scheduler, workers, and client (595)
Add Client.get_versions() function to return software and package information from the scheduler, workers, and client (595)
Improved windows support (577) (590) (583) (597)
Clean up rpc objects explicitly (584)
Normalize collections against known futures (587)
Add key= keyword to map to specify keynames (589)
Custom data serialization (606)
Refactor the web interface (608) (615) (621)
Allow user-supplied Executor in Worker (609)
Pass Worker kwargs through LocalCluster
Schedulers can retire workers cleanly
Schedulers can retire workers cleanly
Add Future.add_done_callback for concurrent.futures compatibility
Update web interface to be consistent with Bokeh 0.12.3
Close streams explicitly, avoiding race conditions and supporting more robust restarts on Windows.
Improved shuffled performance for dask.dataframe
Add adaptive allocation cluster manager
Reduce administrative overhead when dealing with many workers
dask-ssh --log-directory . no longer errors
Microperformance tuning for the scheduler
Revert dask_worker to use fork rather than subprocess by default
Revert dask_worker to use fork rather than subprocess by default
Scatter retains type information
Bokeh always uses subprocess rather than spawn
Fix critical Windows error with dask_worker executable
Fix critical Windows error with dask_worker executable
Rename Executor to Client (492)
Rename Executor to Client (492)
Add --memory-limit option to dask-worker, enabling spill-to-disk behavior when running out of memory (485)
Add --pid-file option to dask-worker and --dask-scheduler (496)
Add upload_environment function to distribute conda environments. This is experimental, undocumented, and may change without notice. (494)
Add workers= keyword argument to Client.compute and Client.persist, supporting location-restricted workloads with Dask collections (484)
Add upload_environment function to distribute conda environments. This is experimental, undocumented, and may change without notice. (494)
Add optional dask_worker= keyword to client.run functions that gets provided the worker or nanny object
Add nanny=False keyword to Client.run, allowing for the execution of arbitrary functions on the nannies as well as normal workers
This release adds some new features and removes dead code
This release adds some new features and removes dead code
Publish and share datasets on the scheduler between many clients (453). See publish.
Launch tasks from other tasks (experimental) (471). See task-launch.
Remove unused code, notably the Center object and older client functions (478)
Executor() and LocalCluster() is now robust to Bokeh's absence (481)
Removed s3fs and boto3 from requirements. These have moved to Dask.
This release is largely a bugfix release, recovering from the previous large refactor.
This release is largely a bugfix release, recovering from the previous large refactor.
Ensure idempotence across clients
Stress test losing scattered data permanently
Add start_ipython_scheduler method to Executor
Add %remote magic for workers
Clean up code and tests
Pool connects to maintain reuse and reduce number of open file handles
Re-implement work stealing algorithm
Support cancellation of tuple keys, such as occur in dask.arrays
Start synchronizing against worker data that may be superfluous
Add memory plot tracking number of bytes
Make the progress bars more compact and align colors
Add workers/ page with workers table, stacks/processing plot, and memory
Add this release notes document
This release was largely a refactoring release. Internals were changed significantly without many new features.
This release was largely a refactoring release. Internals were changed significantly without many new features.
Major refactor of the scheduler to use transitions system
Tweak protocol to traverse down complex messages in search of large bytestrings
Add dask-submit and dask-remote
Refactor HDFS writing to align with changes in the dask library
Executor reconnects to scheduler on broken connection or failed scheduler
Support sklearn.external.joblib as well as normal joblib
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Nothing published for this version
Your coding agent can read these notes before it upgrades. Set up the MCP server →