joblib

• The Memory object won't overwrite an already existing .gitignore file in its cache directory anymore. https://github.com/joblib/joblib/pull/1742

• Harden the safety checks in eval_expr(pre_dispatch) to prevent excessive memory allocation and potential crashes by limiting the allowed length of the expression and the maximum numeric value of sub-expressions and not evaluating expressions with non-numeric literals. https://github.com/joblib/joblib/pull/1744

• Vendor cloudpickle 3.1.2 to fix a pickling problem with interactively defined abstract base classes and type annotations in Python 3.14+.

• Vendor loky3.5.6 fixing the resource tracker for python 3.13.7+ https://github.com/joblib/joblib/pull/1740

Memory:

• Ensure that temporary files managed by the Memory object do not collide when using the same cache directory when the cache directory is accessed concurrently from different nodes on a cluster with a shared filesystem. https://github.com/joblib/joblib/pull/1656

• Fix backend hints causing errors when no multiprocessing is present https://github.com/joblib/joblib/issues/1721

• Vendor loky3.5.5 fixing the resource_tracker clean up with earlier Python versions. https://github.com/joblib/joblib/issues/1724

Memory:

• Enforce age_limit is a positive timedelta for Memory.reduce_size, to avoid silently ignoring it. https://github.com/joblib/joblib/pull/1613

• Remove deprecated bytes_limit argument for Memory, which should be passed directly to Memory.reduce_size. https://github.com/joblib/joblib/pull/1569

• Extend functionality of the check_call_in_cache method to now also check against cache validity. Before, it would only check for a given call if it is in cache memory. https://github.com/joblib/joblib/pull/1584

• The Memory object now automatically creates a .gitignore file in its cache directory, instructing git to ignore the entire folder. https://github.com/joblib/joblib/pull/1674

Parallel:

• Fixed a bug that caused the timeout parameter in joblib.Parallel to be ineffective when used along with return_as='generator_unordered'. https://github.com/joblib/joblib/issues/1586

• Pretty printing of Parallel execution progress when the number of tasks is known. https://github.com/joblib/joblib/pull/1608

• Make it possible to pass extra arguments to the LokyBackend and MultiprocessingBackend, enabling the use of initializer. https://github.com/joblib/joblib/pull/1525

• Refactor and document the custom parallel backend API. https://github.com/joblib/joblib/pull/1667

Maintenance:

• Drop support for Python 3.8. https://github.com/joblib/joblib/pull/1669

• Support for Python 3.13 free-threaded has been added. https://github.com/joblib/joblib/pull/1589

• Drop support for PyPy. https://github.com/joblib/joblib/pull/1670

• Fixed an issue affecting joblib.load calls with non-null mmap_mode parameter when loading compressed python objects. It wrongly attempted to load with np.memmap anyway, resulting in python exceptions or corrupted data. The result now properly use in-memory np.array arrays, in accordance with the warnings that are emitted in this case. https://github.com/joblib/joblib/pull/1681

• Fix a regression in 1.3 and 1.4 that caused large big endian arrays to trigger a serialization error. https://github.com/joblib/joblib/issues/1545

• Added a ensure_native_byte_order parameter to joblib.load. When True and mmap_mode is None, loaded arrays are automatically coerced to a byte order that matches the endianness of the host system. This behavior has been the default since joblib==1.3, and can now be disabled if the parameter is set to False instead. Note that setting it to True will raise an error if mmap_mode is not null. The default value 'auto' is equivalent to always setting True if mmap_mode is None, else always False. https://github.com/joblib/joblib/pull/1561

• Fix support for python 3.14 in hashing, with the addition of an extra argument in Pickler._batch_setitems. https://github.com/joblib/joblib/pull/1688

• Fix tests on platforms with only one CPU core. https://github.com/joblib/joblib/pull/1682

• Bump vendored cloudpickle to 3.1.1 to support Python 3.14 (dev) and various other fixes.

• Bump vendored loky to 3.5.3 to support recent Python versions without raising the warning on calls to os.fork and fix various sources of crashes and deadlocks.

• Use pickle protocol 5 for pickling numpy arrays with object type. https://github.com/joblib/joblib/pull/1682

What's Changed

• TST add a test that ensures conservation of byte order during IPC by @fcharras in #1562
• DOC fix typos in CHANGES.rst by @cclauss in #1571
• DOC typo in docs by @jmerkow in #1574
• FIX revert MemorizedFunc.call API change by @tomMoral in #1576

New Contributors

• @jmerkow made their first contribution in #1574

Full Changelog: 1.4.0...1.4.2

What's Changed

• FIX raise iterator exception in user's thread by @tomMoral in #1491
• MAINT: Update byte_bounds import by @mtsokol in #1501
• FEA Implement generator unordered parameter by @fcharras in #1463
• MAINT: Cleanup expired ndarray methods by @mtsokol in #1506
• Fix README cloning url by @andreaso in #1508
• Fix README archive download link by @andreaso in #1509
• Prefer https:// links in the documentation by @andreaso in #1510
• Stricter Dask tests cleanup by @ogrisel in #1514
• CLN simplify code for easier read by @tomMoral in #1517
• Vendor cloudpickle 3.0.0 and drop support for Python 3.7. by @ogrisel in #1515
• MNT Fix Python 3.12 deprecation warning by @lesteve in #1518
• FIX _get_items_to_delete raising error when items list empty by @Dr-Blank in #1503
• Update dask backend for compatibility with return_as=generator by @fcharras in #1520
• DOC: Best randomness with Parallel generators by @paquiteau in #1490
• MTN update ci pipelines to use python3.8 + test more pypy versions by @fcharras in #1487
• MNT Restart readthedoc pipeline by @fcharras in #1524
• FIX Close cleanly distributed Client at the end of unit tests by @fcharras in #1526
• Add pytest 8.x compatability by @mr-c in #1553
• n_jobs parameter in instantiation of Parallel should be an integer by @androids-electric-sheep in #1549
• DOC : updated n_jobs docs in Parallel class by @Schefflera-Arboricola in #1537
• CI Fix url for scikit-learn nightly builds by @jeremiedbb in #1567
• ENH allow caching coroutine functions by @gsakkis in #894
• DOC consistent default formating in doc by @tomMoral in #1565
• DOC add gotcha on non-reproducible pickling by @tomMoral in #1566
• RELEASE 1.4.0 by @tomMoral in #1568

New Contributors

• @mtsokol made their first contribution in #1501
• @andreaso made their first contribution in #1508
• @Dr-Blank made their first contribution in #1503
• @paquiteau made their first contribution in #1490
• @mr-c made their first contribution in #1553
• @androids-electric-sheep made their first contribution in #1549
• @Schefflera-Arboricola made their first contribution in #1537
• @gsakkis made their first contribution in #894

Full Changelog: 1.3.2...1.4.0

Bug fix release

What's Changed

• FIX treat n_jobs=None as if left to its default value by @jeremiedbb in https://github.com/joblib/joblib/pull/1475

• FIX Init logger parent class in Parallel by @fcharras in https://github.com/joblib/joblib/pull/1494

• MNT remove unnecessary .bck file by @lesteve in https://github.com/joblib/joblib/pull/1480

• MTN adjust test regex for Python 3.12 improved error message by @hroncok in https://github.com/joblib/joblib/pull/1476

• DOC add public documentation for parallel_backend by @glemaitre in https://github.com/joblib/joblib/pull/1481

• FIX flake8 new E721: type comparison by @tomMoral in https://github.com/joblib/joblib/pull/1492

New Contributors

• @hroncok made their first contribution in https://github.com/joblib/joblib/pull/1476

Full Changelog: https://github.com/joblib/joblib/compare/1.3.1...1.3.2

What's Changed

• RELEASE joblib 1.3.1 - bugfix release - vendoring loky 3.4.1 for compatibility by @tomMoral in https://github.com/joblib/joblib/pull/1472

Full Changelog: https://github.com/joblib/joblib/compare/1.3.0...1.3.1

What's Changed

• DOC: Document cache hit function by @larsoner in #1343
• DOC: Document call better by @larsoner in #1348
• Ensure native byte order for memmap. by @cmarmo in #1354
• Drop runtime dependency on distutils by @PeterJCLaw in #1361
• Add environment variable to change default parallel backend by @lesteve in #1356
• Fix memmapping_reducer when 'os' has no attribute 'statvfs' by @rth in #1366
• Move the metadata into pyproject.toml by @KOLANICH in #1382
• MAINT pin flake8<6 on the CI by @ogrisel in #1383
• TST Close client in test_pickle_in_socket by @lesteve in #1387
• Do not swallow PicklingError by @Nielius in #1359
• FIX Avoid collisions when caching nested functions by @cache-missing in #1374
• CI/TST provide more informative traceback in case of a deadlock in tests by @fcharras in #1379
• FIX heisenfailure in doc/memory.rst by @ogrisel in #1396
• MAINT Explicit support for Python 3.11 by @jjerphan in #1389
• MNT Use faulthandler rather than custom autokill logic by @lesteve in #1398
• BENCH add benchmark script for n_jobs=1 by @tomMoral in #1400
• TST Fix test_nested_parallel_warnings_parent_backend for Python nogil by @lesteve in #1394
• TST Fix test_memmapping for Python nogil by @lesteve in #1395
• MAINT Clean deprecations by @jeremiedbb in #1397
• ENH make temp resource cleanup safer by @tomMoral in #1391
• MAINT Simplify warning in _persist_input by @jjerphan in #1390
• MNT Use full flake8 rather than flake8_diff.sh by @lesteve in #1415
• Update Dask backend by @scharlottej13 in #1411
• FIX upload to codecov by @tomMoral in #1421
• MTN vendor loky 3.4.0 by @tomMoral in #1422
• MTN skip thread_bomb mitigation test on PyPy for now by @tomMoral in #1423
• High verbosity mode that prints arguments, hash and store location. by @japhba in #1416
• DBG try to debug the thread_bomb_mitigation test by @tomMoral in #1424
• FEA return generator, #588 stripped of unrelated changes for minimal diff review by @fcharras in #1393
• BENCH scaling of a GridSearch with n_jobs by @tomMoral in #1427
• CLN make generator exit thread safe by @tomMoral in #1428
• FEA parallel_config context manager to allow more fine-grained control by @jeremiedbb in #1392
• MAINT: Be nitpicky about docs by @larsoner in #1437
• CLN tidy logger by @tomMoral in #1439
• Give Memory.reduce_size() items_limit and age_limit options by @jwodder in #1200
• CLN deprecate bytes_limit from Memory by @tomMoral in #1447
• FIX doc building failing download by @tomMoral in #1456
• Document parallel_config by @GaelVaroquaux in #1451
• ENH add cache_validation_callback in Memory by @tomMoral in #1149
• DOC example data DL from GH by @tomMoral in #1461
• merge all configs in pyproject.toml by @Borda in #1433
• fix double repeated word typos by @ivanov in #1464
• DOC reference parallel_config instead of parallel_backend by @tomMoral in #1457
• return_generator={True,False} -> return_as={'list','generator'} by @fcharras in #1458
• RELEASE 1.3.0 by @tomMoral in #1466

New Contributors

• @cmarmo made their first contribution in #1354
• @PeterJCLaw made their first contribution in #1361
• @KOLANICH made their first contribution in #1382
• @Nielius made their first contribution in #1359
• @cache-missing made their first contribution in #1374
• @scharlottej13 made their first contribution in #1411
• @japhba made their first contribution in #1416
• @Borda made their first contribution in #1433
• @ivanov made their first contribution in #1464

Full Changelog: 1.2.0...1.3.0

• Fix a security issue where eval(pre_dispatch) could potentially run arbitrary code. Now only basic numerics are supported. https://github.com/joblib/joblib/pull/1327

• Make sure that joblib works even when multiprocessing is not available, for instance with Pyodide https://github.com/joblib/joblib/pull/1256

• Avoid unnecessary warnings when workers and main process delete the temporary memmap folder contents concurrently. https://github.com/joblib/joblib/pull/1263

• Fix memory alignment bug for pickles containing numpy arrays. This is especially important when loading the pickle with mmap_mode != None as the resulting numpy.memmap object would not be able to correct the misalignment without performing a memory copy. This bug would cause invalid computation and segmentation faults with native code that would directly access the underlying data buffer of a numpy array, for instance C/C++/Cython code compiled with older GCC versions or some old OpenBLAS written in platform specific assembly. https://github.com/joblib/joblib/pull/1254

• Vendor cloudpickle 2.2.0 which adds support for PyPy 3.8+.

• Vendor loky 3.3.0 which fixes several bugs including:

  • robustly forcibly terminating worker processes in case of a crash (https://github.com/joblib/joblib/pull/1269);

  • avoiding leaking worker processes in case of nested loky parallel calls;

  • reliability spawn the correct number of reusable workers.

• Fix a security issue where eval(pre_dispatch) could potentially run arbitrary code. Now only basic numerics are supported. https://github.com/joblib/joblib/pull/1327

• Fix byte order inconsistency issue during deserialization using joblib.load in cross-endian environment: the numpy arrays are now always loaded to use the system byte order, independently of the byte order of the system that serialized the pickle. https://github.com/joblib/joblib/pull/1181

• Fix joblib.Memory bug with the ignore parameter when the cached function is a decorated function. https://github.com/joblib/joblib/pull/1165

• Fix joblib.Memory to properly handle caching for functions defined interactively in a IPython session or in Jupyter notebook cell. https://github.com/joblib/joblib/pull/1214

• Update vendored loky (from version 2.9 to 3.0) and cloudpickle (from version 1.6 to 2.0) https://github.com/joblib/joblib/pull/1218

• Add check_call_in_cache method to check cache without calling function. https://github.com/joblib/joblib/pull/820

• dask: avoid redundant scattering of large arguments to make a more efficient use of the network resources and avoid crashing dask with "OSError: [Errno 55] No buffer space available" or "ConnectionResetError: [Errno 104] connection reset by peer". https://github.com/joblib/joblib/pull/1133

• Make joblib.hash and joblib.Memory caching system compatible with numpy >= 1.20.0. Also make it explicit in the documentation that users should now expect to have their joblib.Memory cache invalidated when either joblib or a third party library involved in the cached values definition is upgraded. In particular, users updating joblib to a release that includes this fix will see their previous cache invalidated if they contained reference to numpy objects. https://github.com/joblib/joblib/pull/1136

• Remove deprecated check_pickle argument in delayed. https://github.com/joblib/joblib/pull/903

• Fix a spurious invalidation of Memory.cache'd functions called with Parallel under Jupyter or IPython. https://github.com/joblib/joblib/pull/1093

• Bump vendored loky to 2.9.0 and cloudpickle to 1.6.0. In particular this fixes a problem to add compat for Python 3.9.

• Fix a problem in the constructors of Parallel backends classes that inherit from the AutoBatchingMixin that prevented the dask backend to properly batch short tasks. https://github.com/joblib/joblib/pull/1062

• Fix a problem in the way the joblib dask backend batches calls that would badly interact with the dask callable pickling cache and lead to wrong results or errors. https://github.com/joblib/joblib/pull/1055

• Prevent a dask.distributed bug from surfacing in joblib's dask backend during nested Parallel calls (due to joblib's auto-scattering feature) https://github.com/joblib/joblib/pull/1061

• Workaround for a race condition after Parallel calls with the dask backend that would cause low level warnings from asyncio coroutines: https://github.com/joblib/joblib/pull/1078

• Make joblib work on Python 3 installation that do not ship with the lzma package in their standard library.

• Drop support for Python 2 and Python 3.5. All objects in joblib.my_exceptions and joblib.format_stack are now deprecated and will be removed in joblib 0.16. Note that no deprecation warning will be raised for these objects Python < 3.7. https://github.com/joblib/joblib/pull/1018

• Fix many bugs related to the temporary files and folder generated when automatically memory mapping large numpy arrays for efficient inter-process communication. In particular, this would cause PermissionError exceptions to be raised under Windows and large leaked files in /dev/shm under Linux in case of crash. https://github.com/joblib/joblib/pull/966

• Make the dask backend collect results as soon as they complete leading to a performance improvement: https://github.com/joblib/joblib/pull/1025

• Fix the number of jobs reported by effective_n_jobs when n_jobs=None called in a parallel backend context. https://github.com/joblib/joblib/pull/985

• Upgraded vendored cloupickle to 1.4.1 and loky to 2.8.0. This allows for Parallel calls of dynamically defined functions with type annotations in particular.

• Configure the loky workers' environment to mitigate oversubsription with nested multi-threaded code in the following case:

  • allow for a suitable number of threads for numba (NUMBA_NUM_THREADS);

  • enable Interprocess Communication for scheduler coordination when the nested code uses Threading Building Blocks (TBB) (ENABLE_IPC=1)

  https://github.com/joblib/joblib/pull/951

• Fix a regression where the loky backend was not reusing previously spawned workers. https://github.com/joblib/joblib/pull/968

• Revert https://github.com/joblib/joblib/pull/847 to avoid using pkg_resources that introduced a performance regression under Windows: https://github.com/joblib/joblib/issues/965

• Improved the load balancing between workers to avoid stranglers caused by an excessively large batch size when the task duration is varying significantly (because of the combined use of joblib.Parallel and joblib.Memory with a partially warmed cache for instance). https://github.com/joblib/joblib/pull/899

• Add official support for Python 3.8: fixed protocol number in Hasher and updated tests.

• Fix a deadlock when using the dask backend (when scattering large numpy arrays). https://github.com/joblib/joblib/pull/914

• Warn users that they should never use joblib.load with files from untrusted sources. Fix security related API change introduced in numpy 1.6.3 that would prevent using joblib with recent numpy versions. https://github.com/joblib/joblib/pull/879

• Upgrade to cloudpickle 1.1.1 that add supports for the upcoming Python 3.8 release among other things. https://github.com/joblib/joblib/pull/878

• Fix semaphore availability checker to avoid spawning resource trackers on module import. https://github.com/joblib/joblib/pull/893

• Fix the oversubscription protection to only protect against nested Parallel calls. This allows joblib to be run in background threads. https://github.com/joblib/joblib/pull/934

• Fix ValueError (negative dimensions) when pickling large numpy arrays on Windows. https://github.com/joblib/joblib/pull/920

• Upgrade to loky 2.6.0 that add supports for the setting environment variables in child before loading any module. https://github.com/joblib/joblib/pull/940

• Fix the oversubscription protection for native libraries using threadpools (OpenBLAS, MKL, Blis and OpenMP runtimes). The maximal number of threads is can now be set in children using the inner_max_num_threads in parallel_backend. It defaults to cpu_count() // n_jobs. https://github.com/joblib/joblib/pull/940

Pierre Glaser

Upgrade to cloudpickle 0.8.0

Add a non-regression test related to joblib issues #836 and #833, reporting that cloudpickle versions between 0.5.4 and 0.7 introduced a bug where global variables changes in a parent process between two calls to joblib.Parallel would not be propagated into the workers

Pierre Glaser

Memory now accepts pathlib.Path objects as location parameter. Also, a warning is raised if the returned backend is None while location is not None.

Olivier Grisel

Make Parallel raise an informative RuntimeError when the active parallel backend has zero worker.

Make the DaskDistributedBackend wait for workers before trying to schedule work. This is useful in particular when the workers are provisionned dynamically but provisionning is not immediate (for instance using Kubernetes, Yarn or an HPC job queue).

Thomas Moreau

Include loky 2.4.2 with default serialization with cloudpickle. This can be tweaked with the environment variable LOKY_PICKLER.

Thomas Moreau

Fix nested backend in SequentialBackend to avoid changing the default backend to Sequential. (#792)

Thomas Moreau, Olivier Grisel

Fix nested_backend behavior to avoid setting the default number of workers to -1 when the backend is not dask. (#784)

Thomas Moreau, Olivier Grisel

Include loky 2.3.1 with better error reporting when a worker is abruptly terminated. Also fixes spurious debug output.

Pierre Glaser

Include cloudpickle 0.5.6. Fix a bug with the handling of global variables by locally defined functions.

Thomas Moreau, Pierre Glaser, Olivier Grisel

Include loky 2.3.0 with many bugfixes, notably w.r.t. when setting non-default multiprocessing contexts. Also include improvement on memory management of long running worker processes and fixed issues when using the loky backend under PyPy.

Maxime Weyl

Raises a more explicit exception when a corrupted MemorizedResult is loaded.

Maxime Weyl

Loading a corrupted cached file with mmap mode enabled would recompute the results and return them without memory mapping.

Thomas Moreau

Fix joblib import setting the global start_method for multiprocessing.

Alexandre Abadie

Fix MemorizedResult not picklable (#747).

Loïc Estève

Fix Memory, MemorizedFunc and MemorizedResult round-trip pickling + unpickling (#746).

James Collins

Fixed a regression in Memory when positional arguments are called as kwargs several times with different values (#751).

Thomas Moreau and Olivier Grisel

Integration of loky 2.2.2 that fixes issues with the selection of the default start method and improve the reporting when calling functions with arguments that raise an exception when unpickling.

Maxime Weyl

Prevent MemorizedFunc.call_and_shelve from loading cached results to RAM when not necessary. Results in big performance improvements

Olivier Grisel

Integrate loky 2.2.0 to fix regression with unpicklable arguments and functions reported by users (#723, #643).

Loky 2.2.0 also provides a protection against memory leaks long running applications when psutil is installed (reported as #721).

Joblib now includes the code for the dask backend which has been updated to properly handle nested parallelism and data scattering at the same time (#722).

Alexandre Abadie and Olivier Grisel

Restored some private API attribute and arguments (MemorizedResult.argument_hash and BatchedCalls.__init__'s pickle_cache) for backward compat. (#716, #732).

Joris Van den Bossche

Fix a deprecation warning message (for Memory's cachedir) (#720).

Thomas Moreau

Make sure that any exception triggered when serializing jobs in the queue will be wrapped as a PicklingError as in past versions of joblib.

Noam Hershtig

Fix kwonlydefaults key error in filter_args (#715)

Alexandre Abadie

Remove support for python 2.6

Alexandre Abadie

Remove deprecated format_signature, format_call and load_output functions from Memory API.

Loïc Estève

Add initial implementation of LRU cache cleaning. You can specify the size limit of a Memory object via the bytes_limit parameter and then need to clean explicitly the cache via the Memory.reduce_size method.

Olivier Grisel

Make the multiprocessing backend work even when the name of the main thread is not the Python default. Thanks to Roman Yurchak for the suggestion.

Karan Desai

pytest is used to run the tests instead of nosetests. python setup.py test or python setup.py nosetests do not work anymore, run pytest joblib instead.

Loïc Estève

An instance of joblib.ParallelBackendBase can be passed into the parallel argument in joblib.Parallel.

Loïc Estève

Fix handling of memmap objects with offsets greater than mmap.ALLOCATIONGRANULARITY in joblib.Parallel. See https://github.com/joblib/joblib/issues/451 for more details.

Loïc Estève

Fix performance regression in joblib.Parallel with n_jobs=1. See https://github.com/joblib/joblib/issues/483 for more details.

Loïc Estève

Fix race condition when a function cached with joblib.Memory.cache was used inside a joblib.Parallel. See https://github.com/joblib/joblib/issues/490 for more details.

Loïc Estève

Fix tests when multiprocessing is disabled via the JOBLIB_MULTIPROCESSING environment variable.

harishmk

Remove warnings in nested Parallel objects when the inner Parallel has n_jobs=1. See https://github.com/joblib/joblib/pull/406 for more details.

Loïc Estève

FIX a bug in stack formatting when the error happens in a compiled extension. See https://github.com/joblib/joblib/pull/382 for more details.

Vincent Latrouite

FIX a bug in the constructor of BinaryZlibFile that would throw an exception when passing unicode filename (Python 2 only). See https://github.com/joblib/joblib/pull/384 for more details.

Olivier Grisel

Expose joblib.parallel.ParallelBackendBase and joblib.parallel.AutoBatchingMixin in the public API to make them officially reusable by backend implementers.

Alexandre Abadie

ENH: joblib.dump/load now accept file-like objects besides filenames. https://github.com/joblib/joblib/pull/351 for more details.

Niels Zeilemaker and Olivier Grisel

Refactored joblib.Parallel to enable the registration of custom computational backends. https://github.com/joblib/joblib/pull/306 Note the API to register custom backends is considered experimental and subject to change without deprecation.

Alexandre Abadie

Joblib pickle format change: joblib.dump always create a single pickle file and joblib.dump/joblib.save never do any memory copy when writing/reading pickle files. Reading pickle files generated with joblib versions prior to 0.10 will be supported for a limited amount of time, we advise to regenerate them from scratch when convenient. joblib.dump and joblib.load also support pickle files compressed using various strategies: zlib, gzip, bz2, lzma and xz. Note that lzma and xz are only available with python >= 3.3. https://github.com/joblib/joblib/pull/260 for more details.

Antony Lee

ENH: joblib.dump/load now accept pathlib.Path objects as filenames. https://github.com/joblib/joblib/pull/316 for more details.

Olivier Grisel

Workaround for "WindowsError: [Error 5] Access is denied" when trying to terminate a multiprocessing pool under Windows: https://github.com/joblib/joblib/issues/354

Olivier Grisel

FIX a race condition that could cause a joblib.Parallel to hang when collecting the result of a job that triggers an exception. https://github.com/joblib/joblib/pull/296

Olivier Grisel

FIX a bug that caused joblib.Parallel to wrongly reuse previously memmapped arrays instead of creating new temporary files. https://github.com/joblib/joblib/pull/294 for more details.

Loïc Estève

FIX for raising non inheritable exceptions in a Parallel call. See https://github.com/joblib/joblib/issues/269 for more details.

Alexandre Abadie

FIX joblib.hash error with mixed types sets and dicts containing mixed types keys when using Python 3. see https://github.com/joblib/joblib/issues/254

Loïc Estève

FIX joblib.dump/load for big numpy arrays with dtype=object. See https://github.com/joblib/joblib/issues/220 for more details.

Loïc Estève

FIX joblib.Parallel hanging when used with an exhausted iterator. See https://github.com/joblib/joblib/issues/292 for more details.

Olivier Grisel

Revert back to the fork start method (instead of forkserver) as the latter was found to cause crashes in interactive Python sessions.

Loïc Estève

Joblib hashing now uses the default pickle protocol (2 for Python 2 and 3 for Python 3). This makes it very unlikely to get the same hash for a given object under Python 2 and Python 3.

In particular, for Python 3 users, this means that the output of joblib.hash changes when switching from joblib 0.8.4 to 0.9.2 . We strive to ensure that the output of joblib.hash does not change needlessly in future versions of joblib but this is not officially guaranteed.

Loïc Estève

Joblib pickles generated with Python 2 can not be loaded with Python 3 and the same applies for joblib pickles generated with Python 3 and loaded with Python 2.

During the beta period 0.9.0b2 to 0.9.0b4, we experimented with a joblib serialization that aimed to make pickles serialized with Python 3 loadable under Python 2. Unfortunately this serialization strategy proved to be too fragile as far as the long-term maintenance was concerned (For example see https://github.com/joblib/joblib/pull/243). That means that joblib pickles generated with joblib 0.9.0bN can not be loaded under joblib 0.9.2. Joblib beta testers, who are the only ones likely to be affected by this, are advised to delete their joblib cache when they upgrade from 0.9.0bN to 0.9.2.

Arthur Mensch

Fixed a bug with joblib.hash that used to return unstable values for strings and numpy.dtype instances depending on interning states.

Olivier Grisel

Make joblib use the 'forkserver' start method by default under Python 3.4+ to avoid causing crash with 3rd party libraries (such as Apple vecLib / Accelerate or the GCC OpenMP runtime) that use an internal thread pool that is not reinitialized when a fork system call happens.

Olivier Grisel

New context manager based API (with block) to reuse the same pool of workers across consecutive parallel calls.

Vlad Niculae and Olivier Grisel

Automated batching of fast tasks into longer running jobs to hide multiprocessing dispatching overhead when possible.

Olivier Grisel

FIX make it possible to call joblib.load(filename, mmap_mode='r') on pickled objects that include a mix of arrays of both memory memmapable dtypes and object dtype.

2014-11-20 Olivier Grisel

OPTIM use the C-optimized pickler under Python 3

This makes it possible to efficiently process parallel jobs that deal with numerous Python objects such as large dictionaries.

2014-08-19 Olivier Grisel

FIX disable memmapping for object arrays

2014-08-07 Lars Buitinck

MAINT NumPy 1.10-safe version comparisons

2014-07-11 Olivier Grisel

FIX #146: Heisen test failure caused by thread-unsafe Python lists

This fix uses a queue.Queue datastructure in the failing test. This datastructure is thread-safe thanks to an internal Lock. This Lock instance not picklable hence cause the picklability check of delayed to check fail.

When using the threading backend, picklability is no longer required, hence this PRs give the user the ability to disable it on a case by case basis.

2014-06-30 Olivier Grisel

BUG: use mmap_mode='r' by default in Parallel and MemmappingPool

The former default of mmap_mode='c' (copy-on-write) caused problematic use of the paging file under Windows.

2014-06-27 Olivier Grisel

BUG: fix usage of the /dev/shm folder under Linux

2014-05-29 Gael Varoquaux

BUG: fix crash with high verbosity

2014-05-14 Olivier Grisel

Fix a bug in exception reporting under Python 3

2014-05-10 Olivier Grisel

Fixed a potential segfault when passing non-contiguous memmap instances.

2014-04-22 Gael Varoquaux

ENH: Make memory robust to modification of source files while the interpreter is running. Should lead to less spurious cache flushes and recomputations.

2014-02-24 Philippe Gervais

New Memory.call_and_shelve API to handle memoized results by reference instead of by value.

2014-01-10 Olivier Grisel & Gael Varoquaux

FIX #105: Race condition in task iterable consumption when pre_dispatch != 'all' that could cause crash with error messages "Pools seems closed" and "ValueError: generator already executing".

2014-01-12 Olivier Grisel

FIX #72: joblib cannot persist "output_dir" keyword argument.

2013-12-23 Olivier Grisel

ENH: set default value of Parallel's max_nbytes to 100MB

Motivation: avoid introducing disk latency on medium sized parallel workload where memory usage is not an issue.

FIX: properly handle the JOBLIB_MULTIPROCESSING env variable

FIX: timeout test failures under windows

2013-12-19 Olivier Grisel

FIX: support the new Python 3.4 multiprocessing API

2013-12-05 Olivier Grisel

ENH: make Memory respect mmap_mode at first call too

ENH: add a threading based backend to Parallel

This is low overhead alternative backend to the default multiprocessing backend that is suitable when calling compiled extensions that release the GIL.

Author: Dan Stahlke <dan@stahlke.org> Date: 2013-11-08

FIX: use safe_repr to print arg vals in trace

This fixes a problem in which extremely long (and slow) stack traces would be produced when function parameters are large numpy arrays.

2013-09-10 Olivier Grisel

ENH: limit memory copy with Parallel by leveraging numpy.memmap when possible

2013-07-25 Gael Varoquaux

MISC: capture meaningless argument (n_jobs=0) in Parallel

2013-07-09 Lars Buitinck

ENH Handles tuples, sets and Python 3's dict_keys type the same as lists. in pre_dispatch

2013-05-23 Martin Luessi

ENH: fix function caching for IPython

2012-09-15 Yannick Schwartz

BUG: make sure that sets and dictionaries give reproducible hashes

2012-07-18 Marek Rudnicki

BUG: make sure that object-dtype numpy array hash correctly

2012-07-12 GaelVaroquaux

BUG: Bad default n_jobs for Parallel

2012-05-07 Vlad Niculae

ENH: controlled randomness in tests and doctest fix

2012-02-21 GaelVaroquaux

ENH: add verbosity in memory

2012-02-21 GaelVaroquaux

BUG: non-reproducible hashing: order of kwargs

The ordering of a dictionary is random. As a result the function hashing was not reproducible. Pretty hard to test

2012-02-14 GaelVaroquaux

BUG: fix joblib Memory pickling

2012-02-11 GaelVaroquaux

BUG: fix hasher with Python 3

2012-02-09 GaelVaroquaux

API: filter_args: *args, **kwargs -> args, kwargs

2012-02-06 Gael Varoquaux

BUG: make sure Memory pickles even if cachedir=None