mpire

(2024-02-19)

• Added support for macOS (#27, #79, #91)

  • Fixes memory leaks on macOS

  • Reduced the amount of semaphores used

  • Issues a warning when cpu_ids is used on macOS

• Added mpire.dashboard.set_stacklevel to set the stack level in the dashboard. This influences what line to display in the 'Invoked on line' section. (#118)

• Use function details from the __call__ method on the dashboard in case the callable being executed is a class instance (#117)

• Use (global) average rate for the estimate on the dashboard when smoothing=0 (#117)

• Make it possible to reuse the same progress_bar_options without raising warnings (#117)

• Removed deprecated progress_bar_position parameter from the map functions. Use progress_bar_options['position'] instead (added since v2.6.0)

(2024-01-08)

• Added support for the rich progress bar style (#96)

• Added the option to only show progress on the dashboard. (#107)

• Progress bars are now supported on Windows when using threading as start method.

• Insights now also work when using the forkserver and spawn start methods. (#104)

• When using insights on Windows the arguments of the top 5 longest tasks are now available as well.

• Fixed deprecated escape import from flask by importing directly from markupsafe. (#106)

• Fixed mpire.dashboard.start_dashboard freeze when there are no two ports available. (#112)

• Added mpire.dashboard.shutdown_dashboard to shutdown the dashboard.

• Added py.typed file to prompt mypy for type checking. (#108)

(2023-11-08)

• Excluded the tests folder from MPIRE distributions (#89)

• Added a workaround for semaphore leakage on macOS and fixed a bug when working in a fork context while the system default is spawn (#92)

• Fix progressbar percentage on dashboard (#101)

• Fixed a bug where starting multiple apply_async tasks with a task timeout didn't interrupt all tasks when the timeout was reached (#98)

• Add testing python 3.12 to workflow and drop 3.6 and 3.7 (#102)

(2023-08-16)

• Added support for Python 3.11 (#67)

(2023-04-14)

• Transfered ownership of the project from Slimmer AI to sybrenjansen

(2023-03-17)

• Added the mpire.WorkerPool.apply and mpire.WorkerPool.apply_async functions (#63)

• When inside a Jupyter notebook, the progress bar will not automatically switch to a widget anymore. tqdm cannot always determine with certainty that someone is in a notebook or, e.g., a Jupyter console. Another reason is to avoid the many errors people get when having widgets or javascript disabled. See progress_bar_style for changing the progress bar to a widget (#71)

• The mpire.dashboard.connect_to_dashboard function now raises a ConnectionRefused error when the dashboard isn't running, instead of silently failing and deadlocking the next map call with a progress bar (#68)

• Added support for a progress bar without knowing the size of the iterable. It used to disable the progress bar when the size was unknown

• Changed how max_tasks_active is handled. It now applies to the number of tasks that are currently being processed, instead of the number of chunks of tasks, as you would expect from the name. Previously, when the chunk size was set to anything other than 1, the number of active tasks could be higher than max_tasks_active

• Updated some exception messages and docs (#69)

• Changed how worker results, restarts, timeouts, unexpected deaths, and exceptions are handled. They are now handled by individual threads such that the main thread is more responsive. The API is the same, so no user changes are needed

• Mixing multiple map calls now raises an error (see mixing-multiple-map-calls)

• Fixed a bug where calling a map function with a progress bar multiple times in a row didn't display the progress bar correctly

• Fixed a bug where the dashboard didn't show an error when an exit function raised an exception

(2022-08-29)

• Added Python 3.10 support

• The tqdm progress bar can now be customized using the progress_bar_options parameter in the map functions (#57)

• Using progress_bar_position from a map function is now deprecated and will be removed in MPIRE v2.10.0. Use progress_bar_options['position'] instead

• Deprecated enable_insights from a map function, use enable_insights in the WorkerPool constructor instead

• Fixed a bug where a worker could exit before an exception was entirely sent over the queue, causing a deadlock (#56)

• Fixed a bug where exceptions with init arguments weren't handled correctly (#58)

• Fixed a rare and weird bug in Windows that could cause a deadlock (probably fixes #55)

(2022-07-25)

• Added the option to fix the order of tasks given to the workers (#46)

• Fixed a bug where updated WorkerPool parameters aren't used in subsequent map calls when keep_alive is enabled

(2022-05-25)

• A timeout for the target, worker_init, and worker_exit functions can be specified after which a worker is stopped (#36)

• A WorkerPool can now be started within a thread which isn't the main thread (#44)

(2022-04-25)

• MPIRE now handles defunct child processes properly, instead of deadlocking (#34)

• Added benchmark highlights to README (#38)

(2022-03-29)

• Platform specific dependencies are now handled using environment markers as defined in PEP-508 (#30)

• Fixes hanging WorkerPool when using worker_lifespan and returning results that exceed the pipe capacity (#32)

• Fixes insights unit tests that could sometime fail because it was too fast

(2021-11-29)

• Changed progress bar handler process to thread, making it more stable (especially in notebooks)

• Changed progress bar tasks completed queue to array, to make it more responsive and faster

• Disabled the tqdm monitor thread which, in combination with MPIRE's own tqdm lock, could result in deadlocks

(2021-11-19)

• Included license file in source distribution (#25)

(2021-11-16)

• Made connecting to the tqdm manager more robust (#23)

(2021-10-15)

• Fixed progress bar in a particular setting with iPython and django installed (#13)

• keep_alive now works even when the function to be called or any other parameter passed to the map function is changed (#15)

• Moved enable_insights to the WorkerPool constructor. Using enable_insights from a map function is now deprecated and will be removed in MPIRE v2.6.0.

• Restructured docs and updated several sections for Windows users.

(2021-08-31)

• Fixed compatibility with newer tqdm versions (>= 4.62.2) (#11)

(2021-08-30)

• Added support for Windows (#6, #7). Support has a few caveats:

  • When using worker insights the arguments of the top 5 longest tasks are not available

  • Progress bar is not supported when using threading as start method

  • When using dill and an exception occurs, or when the exception occurs in an exit function, it can print additional OSError messages in the terminal, but these can be safely ignored.

(2021-08-26)

• Fixed a bug with newer versions of tqdm. The progress bar would throw an AttributeError when connected to a dashboard.

• README and documentation updated

(2021-08-06)

• Workers now have their own task queue, which speeds up tasks with bigger payloads

• Fixed progress bar showing error information when completed without error

• Fixed progress bar and worker insights not displaying properly when using threading

• Progress bar handling improved accross several scenarios

• Dashboard can now handle progress bars when using spawn or forkserver as start method

• Added closing of multiprocessing.JoinableQueue objects, to clean up intermediate junk

• Removed numpy dependency

• Made dill optional again. In many cases it slows processing down

(2021-07-07)

• Worker insights added, providing users insight in multiprocessing efficiency

• worker_init and worker_exit parameters added to each map function

• max_active_tasks is now set to n_jobs * 2 when max_active_tasks=None, to speed up most jobs

• n_splits is now set to n_jobs * 64 when both chunk_size and n_splits are None

• Dashboard ports can now be configured

• Renamed func_pointer to func in each map function

• Fixed a bug with the threading backend not terminating correctly

• Fixed a bug with the progress bar not showing correctly in notebooks

• Using multiprocess is now the default

• Added some debug logging

• Refactored a lot of code

• Minor bug fixes, which should make things more stable.

• Removed Python 3.5 support

• Removed add_task, get_result, insert_poison_pill, stop_workers, and join functions from mpire.WorkerPool. Made start_workers private. There wasn't any reason to use these functions.

(2021-04-23)

• Updated documentation CSS which fixes bullet lists not showing properly

(2021-04-22)

• Updated some unittests and fixed some linting issues

• Minor improvements in documentation

(2021-04-22)

• Workers can be kept alive in between consecutive map calls

• Setting CPU affinity is no longer restricted to Linux platforms

• README updated to use RST format for better compatibility with PyPI

• Added classifiers to the setup file

(2020-09-03)

• First public release on Github and PyPi