NewYour coding agent can read the release notes before it upgrades.Set up the MCP server →
PyPI · #1065 most downloaded on PyPI
A hyperparameter optimization framework
Last release 11 days ago
07 Sep 2026
Ships on a steady schedule
a new release about every 2 months
Nearly every release is documented
notes for 60 of the last 60 stable releases
Nothing withdrawn
no release was ever pulled
8 years old
79 releases · first in 2018
This is the release note of v3.6.1.
This is the release note of v3.6.1.
average_is_best implementation in WilcoxonPruner (#5373)This release was made possible by the authors and the people who participated in the reviews and discussions.
@HideakiImamura, @eukaryo, @nabenabe0928
Remove the deprecated decorator of KerasPruningCallback
This is the release note of v3.6.0.
Optuna 3.6 newly supports the following new features. See our release blog for more detailed information.
optuna.terminator using optuna._gp (#5241)These migration-related PRs do not break the backward compatibility as long as optuna-integration v3.6.0 or later is installed in your environment.
optuna-integration (#5161, thanks @dheemantha-bhat!)sklearn integration (#5225)SkoptSampler (#5234)cma integration (#5236)wandb integration (#5237)sklearn integration (https://github.com/optuna/optuna-integration/pull/66)SkoptSampler (https://github.com/optuna/optuna-integration/pull/74)pycma integration (https://github.com/optuna/optuna-integration/pull/77)MLflow integration (https://github.com/optuna/optuna-integration/pull/84)GPSampler (#5185)formats.sh based on optuna/master (https://github.com/optuna/optuna-integration/pull/75)TypeError if params is not a dict in enqueue_trial (#5164, thanks @adjeiv!)FrozenTrial._validate() (#5211)optuna._gp (#5224)GPSampler (#5274)GPSampler performance other than introducing local search (#5279)README.md (https://github.com/optuna/optuna-integration/pull/88)LightGBMTuner test (https://github.com/optuna/optuna-integration/pull/89)JSONDecodeError in JournalStorage (#5195)gp.fit_kernel_params more robust (#5247)study.tell (#5269, thanks @ryota717!)_split_trials of TPESampler for constrained optimization with constant liar (#5298)study optimize from CLI tutorial page (#5152)GridSampler with ask-and-tell interface (#5153)faq.rst (#5170)plotly.graph_objs with plotly.graph_objects (#5223)optuna.terminator module (#5243, thanks @HarshitNagpal29!)lightgbm dependency in visualization tutorial (#5257)Specify Hyperparameters Manually tutorial page (#5258)n_trials>10000 (#5310)PedAnovaImportanceEvaluator (#5312)WilcoxonPruner (#5313)WilcoxonPruner (#5315)-pre option in the rl integration (https://github.com/optuna/optuna-examples/pull/243)dask and tensorflow (https://github.com/optuna/optuna-examples/pull/245)_create_frozen_trial() under testing module (#5157)__init__.py and fix its documentation generation (https://github.com/optuna/optuna-integration/pull/71)optuna.integration with optuna_integration in the doc and the issue template (https://github.com/optuna/optuna-integration/pull/73)__init__.py (https://github.com/optuna/optuna-integration/pull/86)KerasPruningCallback (https://github.com/optuna/optuna-integration/pull/93)UserWarning by tests/test_keras.py (https://github.com/optuna/optuna-integration/pull/94)TPESampler for more clarity before c-TPE integration (#5117)Checks(integration) failure (#5167)_ParzenEstimatorParameters to more modern style (#5193)optuna/study/_optimize.py (#5261, thanks @shahpratham!)plot_timeline test (#5281)black 24.* (https://github.com/optuna/optuna-integration/pull/64)botorch<0.10. for CI failures (https://github.com/optuna/optuna-integration/pull/96)Checks (Integration) CI (#5217)test_reproducible_in_other_process for GPSampler with Python 3.12 (#5251)fakeredis (#5307)labeler.yml to disable the triage action (#5240)This release was made possible by the authors and the people who participated in the reviews and discussions.
@Alnusjaponica, @DanielAvdar, @HarshitNagpal29, @HideakiImamura, @SimonPop, @adjeiv, @buruzaemon, @c-bata, @contramundum53, @dheemantha-bhat, @eukaryo, @gen740, @hrntsm, @knshnb, @nabenabe0928, @not522, @nzw0301, @porink0424, @ryota717, @shahpratham, @toshihikoyanase, @y0z
One column per quarter.
This is the release note of v3.5.1.
This is the release note of v3.5.1.
load_study function.Add a note about the deprecation of MOTPESampler to the doc
This is the release note of v3.5.0.
This is a maintenance release with various bug fixes and improvements to the documentation and more.
n_objectives condition to be greater than 4 in candidates functions (#5121, thanks @adjeiv!)constant_liar in multi-objective TPESampler (#5021)optuna study-names cli (#5029)ExpectedHypervolumeImprovement candidates function for BotorchSampler (#5065, thanks @adjeiv!)botorch.py (#5094, thanks @sousu4!)OptunaSearchCV (#5098, thanks @adjeiv!)constant_liar in multi-objective TPESampler (#5021)plot_contour (#5107)NSGAIIChildGenerationStrategy (#5003)trials for above in MO split when n_below=0 (#5079)logpdf for scaled truncnorm (#5110)LightGBM tuner and separate train() from __init__.py (#5010)HyperbandPruner (#5075, thanks @felix-cw!)MOTPESampler from index.rst file (#5084, thanks @Ashhar-24!)MOTPESampler to the doc (#5086)README.md to fix the installation and integration (#5126)Recommended budgets include n_startup_trials (#5137)jax and jaxlib (https://github.com/optuna/optuna-examples/pull/223)optuna/optuna-dashboard (https://github.com/optuna/optuna-examples/pull/224)OptunaSearchCV with terminator (https://github.com/optuna/optuna-examples/pull/225)tests/study_tests/test_study.py (#5070, thanks @sousu4!)PyTorchLightning (#5028)Any with float in _TreeNode.children (#5040, thanks @aanghelidi!)typing.py (#5054, thanks @jot-s-bindra!)tests/storages_tests/test_heartbeat.py (#5066, thanks @sousu4!)frozen.py (#5080, thanks @Vaibhav101203!)dataframe.py (#5081, thanks @Vaibhav101203!)test_tensorflow in Python 3.11 (https://github.com/optuna/optuna-integration/pull/46)type: ignore (#5047)tests-mpi to the oldest and latest Python versions (#5067)tests-mpi (#5100)should-skip to test-trigger-type for more clarity (#5134)Pin the version of PyQt6-Qt6 (#5140)README.md (#5108)!examples from .dockerignore (#5129)This release was made possible by the authors and the people who participated in the reviews and discussions.
@Alnusjaponica, @Ashhar-24, @Guillaume227, @HideakiImamura, @JustinGoheen, @Vaibhav101203, @aanghelidi, @adjeiv, @c-bata, @contramundum53, @eukaryo, @felix-cw, @gen740, @jot-s-bindra, @keisuke-umezawa, @knshnb, @nabenabe0928, @not522, @nzw0301, @p1kit, @sousu4, @toshihikoyanase, @y-kamiya
This is the release note of v3.4.1.
This is the release note of v3.4.1.
load_study function.Remove deprecated arguments with regard to LightGBM>=4.0
This is the release note of v3.4.0.
Optuna 3.4 newly supports the following new features. See our release blog for more detailed information.
LightGBM>=4.0 (#4844)SkoptSampler (#4913)get_all_study_names() (#4898)plot_rank (#4899, thanks @ryota717!)TPESampler (#4926)metric_names getter to study (#4930)GCSArtifactStore (#4967, thanks @semiexp!)BestValueStagnationEvaluator (#4974, thanks @smygw72!)_parallel_coordinate.py when log scale (#4911)fail_stale_trials with race condition (#4886)RandomSampler (#4970, thanks @shu65!)min_child_samples (#5007)BruteForceSampler in parallel optimization (#5022)_filesystem.py (#4909)optuna-fast-fanova in documents (#4943)Boto3ArtifactStore's docstring (#4957)JournalStorage (#4980, thanks @semiexp!)ArtifactNotFound (#4982, thanks @smygw72!)n_trials in test_combination_of_different_distributions_objective (#4950)pytest-xdist (#4999)isinstance instead of if type() is ... (#4896)cmaes dependency optional (#4901)before_trial (#4914)_grid.py (#4918)checks-integration errors on LightGBMTuner (#4923)botorch method to remove warning (#4940)_split_trials instead of _get_observation_pairs and _split_observation_pairs (#4947)__future__.annotations in optuna/visualization/_optimization_history.py (#4964, thanks @YuigaWada!)optuna/visualization/_hypervolume_history.py (#4965, thanks @RuTiO2le!)optuna/_convert_positional_args.py (#4966, thanks @hamster-86!)SQLAlchemy (#4968)collections.abc in optuna/visualization/_edf.py (#4969, thanks @g-tamaki!)collections.abc in plot pareto front (#4971)experimental_func from metric_names property (#4983, thanks @semiexp!)__future__.annotations to progress_bar.py (#4992)optuna/optuna/visualization/matplotlib/_optimization_history.py (#5015, thanks @sousu4!)asv 0.6.0 (#4882)tests-mpi (#4998)README.md (https://github.com/optuna/optuna-integration/pull/39)FUNDING.yml (#4912)optional-dependencies and document deselecting integration tests in CONTRIBUTING.md (#4962)This release was made possible by the authors and the people who participated in the reviews and discussions.
@Alnusjaponica, @HideakiImamura, @RuTiO2le, @YuigaWada, @adjeiv, @c-bata, @ciffelia, @contramundum53, @cross32768, @eukaryo, @g-tamaki, @g-votte, @gen740, @hamster-86, @hrntsm, @hvy, @keisuke-umezawa, @knshnb, @lucasmrdt, @louis-she, @moririn2528, @nabenabe0928, @not522, @nzw0301, @ryota717, @semiexp, @shu65, @smygw72, @sousu4, @torotoki, @toshihikoyanase, @xadrianzetx
Handle deprecated argument early_stopping_rounds
This is the release note of v3.3.0.
A new variant of CMA-ES has been added. By setting the lr_adapt argument to True in CmaEsSampler, you can utilize it. For multimodal and/or noisy problems, adapting the learning rate can help avoid getting trapped in local optima. For more details, please refer to #4817. We want to thank @nomuramasahir0, one of the authors of LRA-CMA-ES, for his great work and the development of cmaes library.
<img width="513" alt="256118903-6796d0c4-3278-4d99-bdb2-00b6fe0fa13b" src="https://github.com/optuna/optuna/assets/5564044/50ed3200-2e02-4b10-8ad1-1f237cb3f3ea">
In multiobjective optimization, the history of hypervolume is commonly used as an indicator of performance. Optuna now supports this feature in the visualization module. Thanks to @y0z for your great work!
| Plotly | matplotlib |
|---|---|
| <img width="1056" alt="254270811-e85c3c5e-44e5-4a04-ba8a-f6ea2c53611f (1)" src="https://github.com/optuna/optuna/assets/5564044/c043c79b-a6ad-46bc-92f5-fd54ee61f995"> |
Some samplers support constrained optimization, however, many other features cannot handle it. We are continuously enhancing support for constraints. In this release, plot_optimization_history starts to consider constraint violations. Thanks to @hrntsm for your great work!
import optuna
def objective(trial):
x = trial.suggest_float("x", -15, 30)
y = trial.suggest_float("y", -15, 30)
v0 = 4 * x**2 + 4 * y**2
trial.set_user_attr("constraint", [1000 - v0])
return v0
def constraints_func(trial):
return trial.user_attrs["constraint"]
sampler = optuna.samplers.TPESampler(constraints_func=constraints_func)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=100)
fig = optuna.visualization.plot_optimization_history(study)
fig.show()
<img width="1127" alt="streamlit_integration" src="https://github.com/optuna/optuna/assets/5564044/e8ea5d13-c834-4ed3-8c7b-24ab07c37105">
Optuna Dashboard v0.11.0 provides the tight integration with Streamlit framework. By using this feature, you can create your own application for human-in-the-loop optimization. Please check out the documentation and the example for details.
ordered_dict argument from IntersectionSearchSpace (#4846)logei_candidate_func and make it default when available (#4667)JournalFileStorage and JournalRedisStorage on CLI (#4696)cv_results_ to OptunaSearchCV (#4751, thanks @jckkvs!)optuna.integration.botorch.qnei_candidates_func (#4753, thanks @kstoneriv3!)plotly backend (#4757, thanks @y0z!)FileSystemArtifactStore (#4763)_optimization_history_plot (#4793, thanks @hrntsm!)LightGBM version to v4.0.0 (#4810)matplotlib._optimization_history_plot (#4816, thanks @hrntsm!)upload_artifact api (#4823)before_trial (#4825)Boto3ArtifactStore (#4840)logpdf in _truncnorm.py (#4712)erf (#4713)get_all_trials in InMemoryStorage (#4716)BruteForceSampler consider failed trials (#4747)_get_latest_trial (#4774)plot_hypervolume_history (#4776)BruteForceSampler for pruned trials (#4720)plot_slice bug when some of the choices are numeric (#4724)LightGBMTuner reproducible (#4795)jquery-extension (#4691)plot_rank and plot_timeline plots to visualization tutorial (#4735)integration/sklearn.py (#4745)study.n_objectives from document (#4796)sphinx_rtd_theme (#4853)LICENSE file (https://github.com/optuna/optuna-examples/pull/200)pytestmark (https://github.com/optuna/optuna-integration/pull/29)GridSampler test for failed trials (#4721)OptunaSearchCV behavior (#4758)test_log_gass_mass with SciPy 1.11.0 (#4766)benchmarks (#4703, thanks @caprest!)TPESampler (#4717)_get_observation_pairs (#4742)early_stopping_rounds (#4752)_fast_non_dominated_sort() (#4759)after_trial strategy (#4760)TPESampler (#4769)pkg_resources (#4770)_calculate_weights_below_for_multi_objective (#4773)_study_id parameter from Trial class (#4811, thanks @adjeiv!)OrderedDict (#4838, thanks @taniokay!)samplers._search_space.IntersectionSearchSpace (#4857)tests-integration (#4784)type:ignores (#4787)This release was made possible by the authors and the people who participated in the reviews and discussions.
@Alnusjaponica, @HideakiImamura, @adjeiv, @c-bata, @caprest, @contramundum53, @cross32768, @eukaryo, @gen740, @hrntsm, @jckkvs, @knshnb, @kstoneriv3, @nomuramasahir0, @not522, @nzw0301, @rishabsinghh, @taniokay, @toshihikoyanase, @wouterzwerink, @xadrianzetx, @y0z
Improve deprecated messages in the old suggest functions
This is the release note of v3.2.0.
With the latest release, we have incorporated support for human-in-the-loop optimization. It enables an interactive optimization process between users and the optimization algorithm. As a result, it opens up new opportunities for the application of Optuna in tuning Generative AI. For further details, please check out our human-in-the-loop optimization tutorial.
<img width="826" alt="human-in-the-loop-optimization" src="https://github.com/optuna/optuna/assets/3255979/cb03dd4d-2521-499c-bbe6-06dd7144fb4b">
Overview of human-in-the-loop optimization. Generated images and sounds are displayed on Optuna Dashboard, and users can directly evaluate them there.
Optuna Terminator is a new feature that quantitatively estimates room for optimization and automatically stops the optimization process. It is designed to alleviate the burden of figuring out an appropriate value for the number of trials (n_trials), or unnecessarily consuming computational resources by indefinitely running the optimization loop. See #4398 and optuna-examples#190.
Transition of estimated room for improvement. It steadily decreases towards the level of cross-validation errors.
We've introduced the NSGAIIISampler as a new multi-objective optimization sampler. It implements NSGA-III, which is an extended variant of NSGA-II, designed to efficiently optimize even when the dimensionality of the objective values is large (especially when it's four or more). NSGA-II had an issue where the search would become biased towards specific regions when the dimensionality of the objective values exceeded four. In NSGA-III, the algorithm is designed to distribute the points more uniformly. This feature was introduced by #4436.
Objective value space for multi-objective optimization (minimization problem). Red points represent Pareto solutions found by NSGA-II. Blue points represent those found by NSGA-III. NSGA-II shows a tendency for points to concentrate towards each axis (corresponding to the ends of the Pareto Front). On the other hand, NSGA-III displays a wider distribution across the Pareto Front.
Continuing from v3.1, significant improvements have been made to the CMA-ES Sampler. As a new feature, we've added the BI-population CMA-ES algorithm, a kind of restart strategy that mitigates the problem of falling into local optima. Whether the IPOP CMA-ES, which we've been providing so far, or the new BI-population CMA-ES is better depends on the problems. If you're struggling with local optima, please try BI-population CMA-ES as well. For more details, please see #4464.
The timeline plot visualizes the progress (status, start and end times) of each trial. In this plot, the horizontal axis represents time, and trials are plotted in the vertical direction. Each trial is represented as a horizontal bar, drawn from the start to the end of the trial. With this plot, you can quickly get an understanding of the overall progress of the optimization experiment, such as whether parallel optimization is progressing properly or if there are any trials taking an unusually long time.
Similar to other plot functions, all you need to do is pass the study object to plot_timeline. For more details, please refer to #4470 and #4538.
A new visualization feature, plot_rank, has been introduced. This plot provides valuable insights into landscapes of objective functions, i.e., relationship between parameters and objective values. In this plot, the vertical and horizontal axes represent the parameter values, and each point represents a single trial. The points are colored according to their ranks.
Similar to other plot functions, all you need to do is pass the study object to plot_rank. For more details, please refer to #4427 and #4541.
We have separated Optuna's integration module into a different package called optuna-integration. Maintaining many integrations within the Optuna package was becoming costly. By separating the integration module, we aim to improve the development speed of both Optuna itself and its integration module. As of the release of v3.2, we have migrated six integration modules: allennlp, catalyst, chainer, keras, skorch, and tensorflow (excepting for the TensorBoard integration). To use integration module, pip install optuna-integration will be necessary. See #4484.
chainermn integration (https://github.com/optuna/optuna-integration/pull/1)integration/keras.py (https://github.com/optuna/optuna-integration/pull/5)integration/allennlp (https://github.com/optuna/optuna-integration/pull/8)tf.keras integration (https://github.com/optuna/optuna-integration/pull/21)skorch (https://github.com/optuna/optuna-integration/pull/22)tensorflow integration (https://github.com/optuna/optuna-integration/pull/23)sklearn.model_selection.GridSearchCV's arguments (#4336)optuna.integration.ChainerPruningExtension for migrating to optuna-integration package (#4370)optuna.integration.ChainerMNStudy for migrating to optuna-integration package (#4497)optuna.integration.KerasPruningCallback for migration to optuna-integration (#4558)AllenNLP integration for migration to optuna-integration (#4579)tf.keras integration (#4662)skorch integration for migration to optuna-integration (#4663)tensorflow integration (#4666)We have started supporting Optuna on Mac and Windows. While many features already worked in previous versions, we have fixed issues that arose in certain modules, such as Storage. See #4457 and #4458.
system_attrs and set_system_attr (https://github.com/optuna/optuna-integration/pull/4)system_attrs and set_system_attr (#4550)PyTorch-Lightning (#4384)CmaEsSampler (#4464)optuna.samplers._search_space.intersection.py to optuna.search_space.intersection.py (#4505)plot_terminator_improvement as visualization of optuna.terminator (#4609)optuna.terminator to optuna/terminator/__init__.py (#4669)plot_terminator_improvement (#4701)cmaes package lazily (#4394)BruteForceSampler stateless (#4408)optuna.terminator.improvement.gp.botorch (#4483)Yvar in _BoTorchGaussianProcess (#4488)_BoTorchGaussianProcess to suppress warning messages (#4510)intersection_search_space from study to trials (#4514)distributed>=2023.3.2 (#4589, thanks @jrbourbeau!)plot_rank marker lines (#4602)study.ask and study.get_trials (#4631)botorch dependency (#4368)colorlog compatibility problem (#4406)add_trial (#4416)RDBStorage.get_best_trial when there are infs (#4422)RDBStorage or JournalStorage (#4434)param_mask for multivariate TPE with constant_liar (#4462)QMCSampler samplers reproducible with seed=0 (#4480)metric_names on _log_completed_trial() function (#4594)ImportError for botorch<=0.4.0 (#4626)n_retries += 1 in RDBStorage (#4658)CachedStorage (#4670)ValueError: Rank 0 node expects an optuna.trial.Trial instance as the trial argument (#4698, thanks @keisukefukuda!)plot_terminator_improvement and fix some bugs (#4702)pyproject.toml for packaging (#4164)sphinxcontrib.jquery explicitly (https://github.com/optuna/optuna-integration/pull/18)Terminator class (#4596)intersphinx_mapping in conf.py (#4290)MeanDecreaseImpurityImportanceEvaluator (#4385)sphinxcontrib.jquery extension to conf.py (#4615)SkoptSampler (#4625)rank_plot function and its matplotlib version (#4660)optuna.termintor (#4675)plot_terminator_improvement (#4677)versionadded directives (#4681)DaskStorage (#4694)min_n_trials (#4709)black . with black 23.1.0 (https://github.com/optuna/optuna-examples/pull/168)pytorch_distributed_spawn.py (https://github.com/optuna/optuna-examples/pull/175)optuna-integration in chainer CI (https://github.com/optuna/optuna-examples/pull/176)FutureWarning about Trial.set_system_attr in storage tests (#4323)test_nsgaii.py (#4387)test_with_server.py (#4402)Chainer (#4410)optuna.terminator.improvement._preprocessing.py (#4506)PyTorch Lightning (#4520)_imports.py from optuna (https://github.com/optuna/optuna-integration/pull/16)AllenNLP in Checks (integration) (#4277)tests/hypervolume_tests/test_hssp.py (#4329)CmaEsSampler (#4395)PyTorch Distributed (#4413)numpy.polynomial in _erf.py (#4415)_ParzenEstimator (#4433)RegretBoundEvaluator (#4442)Checks(integration) about terminator/.../botorch.py (#4461)RegretBoundEvaluator (#4469)optuna.samplers._search_space.group_decomposed.py to optuna.search_space.group_decomposed.py (#4491)optuna.visualization (#4525, thanks @harupy!)tests.visualization_tests (#4526, thanks @harupy!)_BoTorchGaussianProcess (#4530)optuna.visualization.plot_timeline (#4540)SingleTaskGP for Optuna terminator (#4542)optuna.samplers.IntersectionSearchSpace and optuna.samplers.intersection_search_space (#4549)IntersectionSearchSpace in optuna.terminator module (#4595)BaseErrorEvaluator and classes that inherit from it (#4607)import Rectangle in visualization/matplotlib (#4620)visualize/_rank.py and visualization_tests/ (#4628)_distribution_is_log to optuna.distributionsP from optuna/terminator/init.py` (#4668)_fast_non_dominated_sort() from the samplers (#4671)get_all_trials of _CachedStorage is called (#4672)actions/setup-python in mac-tests (follow-up for #4307) (#4343)ProcessGroup import from torch.distributed (#4347)pypigh-action-pypi-publish (#4359)checks (#4364)NO_COLOR env or not tty (#4376)ubuntu-latest in PyPI publish CI (#4400)PyYAML==5.1 on tests-with-minimum-dependencies (#4435)Checks(integration) (#4482)Distributed version (#4545)codecov (#4606)test in checks-integration CI (#4612)Output dependency tree by pipdeptree to Actions (#4624)fakeredis (#4637)mlflow with Python 3.11 (#4647)cached-path from setup.py (#4357)hacking with flake8 (#4556)lightning_logs to .gitignore (#4565)black and isort in formats.sh (#4610)benchmark, optional, and test in dev Docker image (#4611)optuna-integration (#4636)This release was made possible by the authors and the people who participated in the reviews and discussions.
@Alnusjaponica, @HideakiImamura, @Ilevk, @Jendker, @Kaushik-Iyer, @amylase, @c-bata, @contramundum53, @cross32768, @eukaryo, @g-votte, @gen740, @gituser789, @harupy, @himkt, @hvy, @jrbourbeau, @keisuke-umezawa, @keisukefukuda, @knshnb, @kstoneriv3, @li-li-github, @nomuramasahir0, @not522, @nzw0301, @toshihikoyanase, @tungbq
This is the release note of v3.1.1.
This is the release note of v3.1.1.
cmaes package lazily (#4573)RDBStorage or JournalStorage (#4572)infs (#4574)types-tqdm for lint (#4566)This release was made possible by the authors and the people who participated in the reviews and discussions.
@HideakiImamura, @contramundum53, @not522
Deprecate set_system_attr in Study and Trial
This is the release note of v3.1.0.
This is not something you have to read from top to bottom to learn about the summary of Optuna v3.1. The recommended way is reading the release blog.
CMA-ES CMA-ES with Margin “The animation is referred from https://github.com/EvoConJP/CMA-ES_with_Margin, which is distributed under the MIT license.”
CMA-ES achieves strong performance for continuous optimization, but there is still room for improvement in mixed-integer search spaces. To address this, we have added support for the "CMA-ES with Margin" algorithm to our CmaEsSampler, which makes it more efficient in these cases. You can see the benchmark results here. For more detailed information about CMA-ES with Margin, please refer to the paper “CMA-ES with Margin: Lower-Bounding Marginal Probability for Mixed-Integer Black-Box Optimization - arXiv”, which has been accepted for presentation at GECCO 2022.
import optuna
from optuna.samplers import CmaEsSampler
def objective(trial):
x = trial.suggest_float("y", -10, 10, step=0.1)
y = trial.suggest_int("x", -100, 100)
return x**2 + y
study = optuna.create_study(sampler=CmaEsSampler(with_margin=True))
study.optimize(objective)
JournalFileStorage, a file storage backend based on JournalStorage, supports NFS (Network File System) environments. It is the easiest option for users who wish to execute distributed optimization in environments where it is difficult to set up database servers such as MySQL, PostgreSQL or Redis (e.g. #815, #1330, #1457 and #2216).
import optuna
from optuna.storages import JournalStorage, JournalFileStorage
def objective(trial):
x = trial.suggest_float("x", -100, 100)
y = trial.suggest_float("y", -100, 100)
return x**2 + y
storage = JournalStorage(JournalFileStorage("./journal.log"))
study = optuna.create_study(storage=storage)
study.optimize(objective)
For more information on JournalFileStorage, see the blog post “Distributed Optimization via NFS Using Optuna’s New Operation-Based Logging Storage” written by @wattlebirdaz.
We have replaced the Redis storage backend with a JournalStorage-based one. The experimental RedisStorage class has been removed in v3.1. The following example shows how to use the new JournalRedisStorage class.
import optuna
from optuna.storages import JournalStorage, JournalRedisStorage
def objective(trial):
…
storage = JournalStorage(JournalRedisStorage("redis://localhost:6379"))
study = optuna.create_study(storage=storage)
study.optimize(objective)
DaskStorage, a new storage backend based on Dask.distributed, is supported. It allows you to leverage distributed capabilities in similar APIs with concurrent.futures. DaskStorage can be used with InMemoryStorage, so you don't need to set up a database server. Here's a code example showing how to use DaskStorage:
import optuna
from optuna.storages import InMemoryStorage
from optuna.integration import DaskStorage
from distributed import Client, wait
def objective(trial):
...
with Client("192.168.1.8:8686") as client:
study = optuna.create_study(storage=DaskStorage(InMemoryStorage()))
futures = [
client.submit(study.optimize, objective, n_trials=10, pure=False)
for i in range(10)
]
wait(futures)
print(f"Best params: {study.best_params}")
Setting up a Dask cluster is easy: install dask and distributed, then run the dask scheduler and dask worker commands, as detailed in the Quick Start Guide in the Dask.distributed documentation.
$ pip install optuna dask distributed
$ dark scheduler
INFO - Scheduler at: tcp://192.168.1.8:8686
INFO - Dashboard at: :8687
…
$ dask worker tcp://192.168.1.8:8686
$ dask worker tcp://192.168.1.8:8686
$ dask worker tcp://192.168.1.8:8686
See the documentation for more information.
BruteForceSampler, a new sampler for brute-force search, tries all combinations of parameters. In contrast to GridSampler, it does not require passing the search space as an argument and works even with branches. This sampler constructs the search space with the define-by-run style, so it works by just adding sampler=optuna.samplers.BruteForceSampler().
import optuna
def objective(trial):
c = trial.suggest_categorical("c", ["float", "int"])
if c == "float":
return trial.suggest_float("x", 1, 3, step=0.5)
elif c == "int":
a = trial.suggest_int("a", 1, 3)
b = trial.suggest_int("b", a, 3)
return a + b
study = optuna.create_study(sampler=optuna.samplers.BruteForceSampler())
study.optimize(objective)
constant_liar OptionThe constant_liar option of TPESampler is an option for the distributed optimization or batch optimization. It has been introduced in v2.8.0, but suffers from performance degradation in specific situations. In this release, we have detected the cause of the problem, and resolve it with fruitful performance verification. See #4073 for more details.
50% time of import optuna is consumed by SciPy-related modules. Also, it consumes 110MB of storage space, which is really problematic in environments with limited resources such as serverless computing.
We decided to implement scientific functions on our own to make the SciPy dependency optional. Thanks to contributors' effort on performance optimization, our implementation is as fast as the code with SciPy although ours is written in pure Python. See #4105 for more information.
Note that QMCSampler still depends on SciPy. If you use QMCSampler, please explicitly specify SciPy as your dependency.
We are developing a new UI for Optuna Dashboard that is available as an opt-in feature from the beta release - simply launch the dashboard as usual and click the link to the new UI. Please try it out and share your thoughts with us.
$ pip install "optuna-dashboard>=0.9.0b2"
Feedback Survey: The New UI for Optuna Dashboard
We have changed the supported Python versions. Specifically, Python 3.6 has been removed from the supported versions and Python 3.11 has been added. See #3021 and #3964 for more details.
study.optimize() in multiple threads (#4068)TPESampler even when multivariate=True (#4079)RedisStorage (#4156)set_system_attr in Study and Trial (#4188)directions arg to storage.create_new_study (#4189)system_attrs in Study class (#4250)Trial.system_attrs property method (#4264)device argument of TorchDistributedTrial (#4266)CmaEsSampler (#4016)BoTorchSampler (#4101)JournalStorage of Redis backend to resume from a snapshot (#4102)TorchDistributedTrial uses group as parameter instead of device (#4106, thanks @reyoung!)user_attrs to print by Optuna studies in cli.py (#4129, thanks @gonzaload!)BruteForceSampler (#4132, thanks @semiexp!)__getstate__ and __setstate__ to RedisStorage (#4135, thanks @shu65!)qNoisyExpectedHypervolumeImprovement acquisition function from Botorch (Issue#4014) (#4186)get_trial_id_from_study_id_trial_number() method to BaseStorage (#3910)search_space values of GridSampler explicitly (#4062)optimize (#4098)TPESampler (#4105)enqueue_trial (#4126)tests/samplers_tests/test_nsgaii.py::test_fast_non_dominated_sort_with_constraints (#4128, thanks @mist714!)getstate and setstate to journal storage (#4130, thanks @shu65!)None in slice plot (#4133, thanks @belldandyxtq!)plot_intermediate_value (#4134, thanks @belldandyxtq!)suggest_categorical (#4143, thanks @ConnorBaker!)study.directions to reduce the number of get_study_directions() calls (#4146)Trial class (#4240)CMAwM class even when there is no discrete params (#4289)OPTUNA_STORAGE environment variable in Optuna CLI (#4299, thanks @Hakuyume!)@overload to ChainerMNTrial and TorchDistributedTrial (Follow-up of #4143) (#4300)OPTUNA_STORAGE environment variable experimental (#4316)TPESampler (#3953, thanks @gasin!)GridSampler (#3957)sqlalchemy.orm.declarative_base (#3967)intermediate_value_type and value_type columns if exists (#4015)SkoptSampler (#4023)datetime.isoformat strings (#4025)JournalStorage set_trial_state_values (#4033)TPESampler reproducible (#4056)constant_liar option (#4073)JournalFileStorage.append_logs (#4076)MLflowCallback (#4097)OptunaSearchCV (#4120)_get_bracket_id in HyperbandPruner (#4131, thanks @zaburo-ch!)to_internal_repr of FloatDistribution and IntDistribution (#4137)PartialFixedSampler to handle None correctly (#4147, thanks @halucinor!)JournalFileStorage on Windows (#4151)TPESampler's constant_liar (#4325)ProcessGroup from torch.distributed (#4344)thop with fvcore (#3906)importlib-metadata (#4036)matplotlib (#4044)thop with fvcore (#3906)FrozenTrial (#3943)BaseStorage (#3948)log_loss instead of deprecated log since sklearn 1.1 (#3993)benchmarks/README.md (#4021)ConvergenceWarning in the ask-and-tell tutorial (#4032)NSGAIISampler (#4045)BruteForceSampler in the samplers' list (#4152)multi_objective module (#4167)QMCSampler (#4179)RedisStorage from docstring (#4232)BruteForceSampler example to the document (#4244)BruteForceSampler (#4245)BruteForceSampler (#4267)XGBoostPruningCallback (#4270)CMAEvolutionStrategy link in integration.PyCmaSampler document (#4284, thanks @hrntsm!)sphinx with nitpicky option and fix typos (#4287)JournalStorage (#4308, thanks @hrntsm!)optuna/integration/dask.py (#4333)suggest_float in BruteForceSampler (#4334)verbose_eval argument from lightgbm callback in tutorial pages (#4335)sphinx_rtd_theme supports Sphinx 6 (#4341)thop with fvcore (https://github.com/optuna/optuna-examples/pull/136)Optuna-distributed to external projects (https://github.com/optuna/optuna-examples/pull/137)CONTRIBUTING.md (https://github.com/optuna/optuna-examples/pull/139)scikit-learn instead of sklearn (https://github.com/optuna/optuna-examples/pull/141)tensorflow to <2.11.0 (https://github.com/optuna/optuna-examples/pull/146)botorch version (https://github.com/optuna/optuna-examples/pull/151)numpy version to 1.23.x for mxnet examples (https://github.com/optuna/optuna-examples/pull/154)tensorflow 2.11 syntax to fix CI error (https://github.com/optuna/optuna-examples/pull/156)Monitor to resolve stable_baselines3's warning (https://github.com/optuna/optuna-examples/pull/162)tests/test_distributions.py (#3912)tests/trial_tests (#3914)tests/study_tests/ (#3915)tests/integration_tests/test_sklearn.py (#3922)MLflowCallback and WeightsAndBiasesCallback (#3923)RuntimeWarning when nanmin and nanmax take an array only containing nan values from pruners_tests (#3924)pytorch_distributed and chainermn modules (#3927)tests/integration_tests/test_lightgbm.py (#3944)tests/visualization_tests/test_contour.py (#3954)tests/visualization_tests/test_slice.py (#3970, thanks @jmsykes83!)tests/visualization_tests/test_optimization_history.py (#4024)PYTHONHASHSEED for the hash-depedenet test (#4031)study.tell from another process (#4039, thanks @Abelarm!)get_cmap warning from tests/visualization_tests/test_param_importances.py (#4095)n_trials for CI time reduction (#4117)test_pop_waiting_trial_thread_safe on RedisStorage (#4119)BruteForceSampler for infinite search space (#4153)parametrize_sampler (#4154)dask.distributed integration (#4170)DaskStorage to existing storage tests (#4176, thanks @jrbourbeau!)test_catboost.py (#4190)test/integration_tests/test_sampler.py (#4204)PyTorch Lightning in Checks (integration) (#4279)OPTUNA_STORAGE environment variable to check missing storage errors (#4306)Trial not FrozenTrial in a test of WeightsAndBiasesCallback (#4309)_set_alembic_revision (#4319)error_score is stored (#4337)_tell.py (#3841)None parameter in TPESampler (#3886)cliff to argparse (#4100)--no-implicit-reexport option (#4110)find_any_distribution (#4127)mlflow 2.0.1 syntax (#4173)_preprocess_argv in CLI (#4187)_solve_hssp to _hypervolume/utils.py (#4227, thanks @jpbianchi!)CmaEsSampler (#4233)CmaEsSampler (#4239)JournalRedisStorage (#4246)TorchDistributedTrial (#4271)Chainer in Checks (integration) (#4276)BoTorch in Checks (integration) (#4278)dask.py in Checks (integration) (#4280)botorch module by adding the version constraint of gpytorch (#3950)# type: ignore for mypy 0.981 (#4019)Tests and Tests (Storage with server) (#4118)document (#4160)workflow_dispatch trigger to the integration tests (#4166)mlflow==2.0.1 (#4171)fakeredis in benchmark dependencies (#4177)asv speed benchmark (#4185)botorch to avoid CI failure (#4228)pytest dependency for asv (#4243)pytorch_distributed.py in Checks (integration) (#4281)test_pytorch_distributed.py again (#4301)cmaes (#4321)stale (#4071)tox.ini (#4078)days-before-issue-stale 300 days (#4091)optuna.TYPE_CHECKING (#4238)examples/README (#4283)This release was made possible by the authors and the people who participated in the reviews and discussions.
@Abelarm, @Alnusjaponica, @ConnorBaker, @Hakuyume, @HideakiImamura, @Jasha10, @amylase, @belldandyxtq, @c-bata, @contramundum53, @cross32768, @erentknn, @eukaryo, @g-votte, @gasin, @gen740, @gonzaload, @halucinor, @himkt, @hrntsm, @hvy, @jmsykes83, @jpbianchi, @jrbourbeau, @keisuke-umezawa, @knshnb, @mist714, @ncclementi, @not522, @nzw0301, @rene-rex, @reyoung, @semiexp, @shu65, @sile, @toshihikoyanase, @wattlebirdaz, @xadrianzetx, @zaburo-ch
Deprecate set_system_attr in Study and Trial
This is the release note of v3.1.0-b0.
CMA-ES CMA-ES with Margin “The animation is referred from https://github.com/EvoConJP/CMA-ES_with_Margin, which is distributed under the MIT license.”
CMA-ES achieves strong performance for continuous optimization, but there is still room for improvement in mixed-integer search spaces. To address this, we have added support for the "CMA-ES with Margin" algorithm to our CmaEsSampler, which makes it more efficient in these cases. You can see the benchmark results here. For more detailed information about CMA-ES with Margin, please refer to the paper “CMA-ES with Margin: Lower-Bounding Marginal Probability for Mixed-Integer Black-Box Optimization - arXiv”, which has been accepted for presentation at GECCO 2022.
import optuna
from optuna.samplers import CmaEsSampler
def objective(trial):
x = trial.suggest_float("y", -10, 10, step=0.1)
y = trial.suggest_int("x", -100, 100)
return x**2 + y
study = optuna.create_study(sampler=CmaEsSampler(with_margin=True))
study.optimize(objective)
JournalFileStorage, a file storage backend based on JournalStorage, supports NFS (Network File System) environments. It is the easiest option for users who wish to execute distributed optimization in environments where it is difficult to set up database servers such as MySQL, PostgreSQL or Redis (e.g. #815, #1330, #1457 and #2216).
import optuna
from optuna.storages import JournalStorage, JournalFileStorage
def objective(trial):
x = trial.suggest_float("x", -100, 100)
y = trial.suggest_float("y", -100, 100)
return x**2 + y
storage = JournalStorage(JournalFileStorage("./journal.log"))
study = optuna.create_study(storage=storage)
study.optimize(objective)
For more information on JournalFileStorage, see the blog post “Distributed Optimization via NFS Using Optuna’s New Operation-Based Logging Storage” written by @wattlebirdaz.
DaskStorage, a new storage backend based on Dask.distributed, is supported. It enables distributed computing in similar APIs with concurrent.futures. An example code is like the following (The full example code is available in the optuna-examples repository).
import optuna
from optuna.storages import InMemoryStorage
from optuna.integration import DaskStorage
from distributed import Client, wait
def objective(trial):
...
with Client("192.168.1.8:8686") as client:
study = optuna.create_study(storage=DaskStorage(InMemoryStorage()))
futures = [
client.submit(study.optimize, objective, n_trials=10, pure=False)
for i in range(10)
]
wait(futures)
print(f"Best params: {study.best_params}")
One of the interesting aspects is the availability of InMemoryStorage. You don’t need to set up database servers for distributed optimization. Although you still need to set up the Dask.distributed cluster, it’s quite easy like the following. See Quickstart of the Dask.distributed documentation for more details.
$ pip install optuna dask distributed
$ dark-scheduler
INFO - Scheduler at: tcp://192.168.1.8:8686
INFO - Dashboard at: :8687
…
$ dask-worker tcp://192.168.1.8:8686
$ dask-worker tcp://192.168.1.8:8686
$ dask-worker tcp://192.168.1.8:8686
$ python dask_simple.py
We have replaced the Redis storage backend with a JournalStorage-based one. The experimental RedisStorage class has been removed in v3.1. The following example shows how to use the new JournalRedisStorage class.
import optuna
from optuna.storages import JournalStorage, JournalRedisStorage
def objective(trial):
…
storage = JournalStorage(JournalRedisStorage("redis://localhost:6379"))
study = optuna.create_study(storage=storage)
study.optimize(objective)
BruteForceSampler, a new sampler for brute-force search, tries all combinations of parameters. In contrast to GridSampler, it does not require passing the search space as an argument and works even with branches. This sampler constructs the search space with the define-by-run style, so it works by just adding sampler=optuna.samplers.BruteForceSampler().
import optuna
def objective(trial):
c = trial.suggest_categorical("c", ["float", "int"])
if c == "float":
return trial.suggest_float("x", 1, 3, step=0.5)
elif c == "int":
a = trial.suggest_int("a", 1, 3)
b = trial.suggest_int("b", a, 3)
return a + b
study = optuna.create_study(sampler=optuna.samplers.BruteForceSampler())
study.optimize(objective)
study.optimize() in multiple threads (#4068)TPESampler even when multivariate=True (#4079)RedisStorage (#4156)set_system_attr in Study and Trial (#4188)system_attrs in Study class (#4250)CmaEsSampler (#4016)BoTorchSampler (#4101)JournalStorage of Redis backend to resume from a snapshot (#4102)user_attrs to print by optuna studies in cli.py (#4129, thanks @gonzaload!)BruteForceSampler (#4132, thanks @semiexp!)__getstate__ and __setstate__ to RedisStorage (#4135, thanks @shu65!)JournalRedisStorage (#4139, thanks @shu65!)qNoisyExpectedHypervolumeImprovement acquisition function from BoTorch (Issue#4014) (#4186)get_trial_id_from_study_id_trial_number() method to BaseStorage (#3910)search_space values of GridSampler explicitly (#4062)enqueue_trial (#4126)tests/samplers_tests/test_nsgaii.py::test_fast_non_dominated_sort_with_constraints (#4128, thanks @mist714!)None in slice plot (#4133, thanks @belldandyxtq!)plot_intermediate_value (#4134, thanks @belldandyxtq!)study.directions to reduce the number of get_study_directions() calls (#4146)Trial class (#4240)TPESampler (#3953, thanks @gasin!)GridSampler (#3957)sqlalchemy.orm.declarative_base (#3967)intermediate_value_type and value_type columns if exists (#4015)SkoptSampler (#4023)datetime.isoformat strings (#4025)set_trial_state_values (#4033)TPESampler reproducible (#4056)constant_liar option (#4073)JournalFileStorage.append_logs (#4076)MLflowCallback (#4097)OptunaSearchCV (#4120)_get_bracket_id in HyperbandPruner (#4131, thanks @zaburo-ch!)to_internal_repr of FloatDistribution and IntDistribution (#4137)PartialFixedSampler to handle None correctly (#4147, thanks @halucinor!)thop with fvcore (#3906)importlib-metadata (#4036)matplotlib (#4044)thop with fvcore (#3906)FrozenTrial (#3943)BaseStorage (#3948)log_loss instead of deprecated log since sklearn 1.1 (#3993)ConvergenceWarning in the ask-and-tell tutorial (#4032)NSGAIISampler (#4045)BruteForceSampler in the samplers' list (#4152)multi_objective module (#4167)QMCSampler (#4179)RedisStorage from docstring (#4232)BruteForceSampler example to the document (#4244)BruteForceSampler (#4245)thop with fvcore (https://github.com/optuna/optuna-examples/pull/136)Optuna-distributed to external projects (https://github.com/optuna/optuna-examples/pull/137)CONTRIBUTING.md (https://github.com/optuna/optuna-examples/pull/139)scikit-learn instead of sklearn (https://github.com/optuna/optuna-examples/pull/141)tensorflow to <2.11.0 (https://github.com/optuna/optuna-examples/pull/146)1.23.x for mxnet examples (https://github.com/optuna/optuna-examples/pull/154)tests/test_distributions.py (#3912)tests/trial_tests (#3914)tests/study_tests/ (#3915)tests/integration_tests/test_sklearn.py (#3922)MLflowCallback and WeightsAndBiasesCallback (#3923)RuntimeWarning when nanmin and nanmax take an array only containing nan values from pruners_tests (#3924)pytorch_distributed and chainermn modules (#3927)tests/integration_tests/test_lightgbm.py (#3944)tests/visualization_tests/test_contour.py (#3954)tests/visualization_tests/test_slice.py (#3970, thanks @jmsykes83!)tests/visualization_tests/test_optimization_history.py (#4024)PYTHONHASHSEED for the hash-depedenet test (#4031)study.tell from another process (#4039, thanks @Abelarm!)get_cmap warning from tests/visualization_tests/test_param_importances.py (#4095)n_trials for CI time reduction (#4117)test_pop_waiting_trial_thread_safe on RedisStorage (#4119)BruteForceSampler for infinite search space (#4153)parametrize_sampler (#4154)dask.distributed integration (#4170)DaskStorage to existing storage tests (#4176, thanks @jrbourbeau!)test_catboost.py (#4190)test/integration_tests/test_sampler.py (#4204)_tell.py (#3841)TPESampler (#3886)cliff to argparse (#4100)--no-implicit-reexport option (#4110)find_any_distribution (#4127)_preprocess_argv in CLI (#4187)_solve_hssp to _hypervolume/utils.py (#4227, thanks @jpbianchi!)JournalRedisStorage (#4246)botorch module by adding the version constraint of gpytorch (#3950)# type: ignore for mypy 0.981 (#4019)Tests and Tests (Storage with server) (#4118)document (#4160)workflow_dispatch trigger to the integration tests (#4166)mlflow==2.0.1 (#4171)fakeredis in benchmark deps (#4177)asv speed benchmark (#4185)botorch to avoid CI failure (#4228)pytest dependency for asv (#4243)stale (#4071)tox.ini (#4078)days-before-issue-stale 300 days (#4091)optuna.TYPE_CHECKING (#4238)This release was made possible by the authors and the people who participated in the reviews and discussions.
@Abelarm, @Alnusjaponica, @HideakiImamura, @amylase, @belldandyxtq, @c-bata, @contramundum53, @cross32768, @erentknn, @eukaryo, @g-votte, @gasin, @gen740, @gonzaload, @halucinor, @himkt, @hvy, @jmsykes83, @jpbianchi, @jrbourbeau, @keisuke-umezawa, @knshnb, @mist714, @ncclementi, @not522, @nzw0301, @rene-rex, @semiexp, @shu65, @sile, @toshihikoyanase, @wattlebirdaz, @xadrianzetx, @zaburo-ch
This is the release note of v3.0.6.
This is the release note of v3.0.6.
This release was made possible by the authors and the people who participated in the reviews and discussions.
@c-bata @HideakiImamura
This is the release note of v3.0.5.
This is the release note of v3.0.5.
constant_liar option (#4257)This release was made possible by the authors and the people who participated in the reviews and discussions.
@HideakiImamura, @eukaryo, @toshihikoyanase
This is the release note of v3.0.4.
This is the release note of v3.0.4.
This release was made possible by the authors and the people who participated in the reviews and discussions.
@HideakiImamura, @contramundum53
This is the release note of v3.0.3.
This is the release note of v3.0.3.
intermediate_value_type and value_type columns if exists (#4052)release-v3.0.3 branch (#4043)This release was made possible by the authors and the people who participated in the reviews and discussions.
@c-bata, @contramundum53
This is the release note of v3.0.2.
This is the release note of v3.0.2.
In v3.0.0 or v3.0.1, DB migration fails with SQLAlchemy v1.3. We fixed this issue in v3.0.2.
In v3.0.0, typing-extensions was used for fine-grained type checking. However, that resulted in import failures when using older versions of typing-extensions. We made the dependency optional in v3.0.2.
@contramundum53, @c-bata
This release was made possible by the authors and the people who participated in the reviews and discussions.
This is the release note of v3.0.1.
This is the release note of v3.0.1.
GridSampler with RDBIn v3.0.0, GridSampler with RDB raises an error. This patch fixes this combination.
@HideakiImamura, @contramundum53, @not522
This release was made possible by the authors and the people who participated in the reviews and discussions.
Add targets and deprecate axis_order in optuna.visualization.matplotlib.plot_pareto_front (#3341, thanks @shu65!)
This is the release note of v3.0.0.
This is not something you have to read from top to bottom to learn about the summary of Optuna v3. The recommended way is reading the release blog.
If you want to update your existing projects from Optuna v2.x to Optuna v3, please see the migration guide and try out Optuna v3.
New crossover options are added to NSGA-II sampler, the default multi-objective algorithm of Optuna. The performance for floating point parameters are improved. Please visit #2903, #3221, and the document for more information.
Quasi-Monte Carlo sampler is now supported. It can be used in place of RandomSampler, and can improve performance especially for high dimensional problems. See #2423, #2964, and the document for more information.
TPESampler now supports constraint-aware optimization. For more information on this feature, please visit #3506 and the document.
| Without constraints | With constraints |
|---|---|
Pareto-front plot now shows which trials satisfy the constraints and which do not. For more information, please see the following PRs (#3128, #3497, and #3389) and the document.
ShapleyImportanceEvaluatorWe introduced a new importance evaluator, optuna.integration.ShapleyImportanceEvaluator, which uses SHAP. See #3507 and the document for more information.
Optimization history plot can now compare multiple studies or display the mean and variance of multiple studies optimized with the same settings. For more information, please see the following multiple PRs (#2807, #3062, #3122, and #3736) and the document.
Optuna has a number of core APIs. One being the suggest API and the optuna.Study class. The visualization module is also frequently used to analyze results. Many of these have been simplified, stabilized, and refactored in v3.0.
The suggest API has been aggregated into 3 APIs: suggest_float for floating point parameters, suggest_int for integer parameters, and suggest_catagorical for categorical parameters. For more information, see #2939, #2941, and PRs submitted for those issues.
We have developed and published a test policy in v3.0 that defines how tests for Optuna should be written. Based on the published test policy, we have improved many unit tests. For more information, see https://github.com/optuna/optuna/issues/2974 and PRs with test label.
Optuna's visualization module had a deep history and various debts. We have worked throughout v3.0 to eliminate this debt with the help of many contributors. See #2893, #2913, #2959 and PRs submitted for those issues.
Through the development of v3.0, we have decided to provide many experimental features as stable features by going through their behavior, fixing bugs, and analyzing use cases. The following is a list of features that have been stabilized in v3.0.
Optuna has many algorithms implemented, but many of their behaviors and characteristics are unknown to the user. We have developed the following table to inform users of empirically known behaviors and characteristics. See #3571 and #3593 for more details.
To quantitatively assess the performance of our algorithms, we have developed a benchmarking environment. We also evaluated the performance of the algorithms by conducting actual benchmarking experiments using this environment. See here, #2964, and #2906 for more details.
Changes to the RDB schema:
RDBStorage that was created in the previous versions of Optuna, please execute optuna storage upgrade to migrate your database (#3113, #3559, #3603, #3668).Features deprecated in 3.0:
suggest_uniform(), suggest_loguniform(), and suggest_discrete_uniform()
UniformDistribution, LogUniformDistribution, DiscreteUniformDistribution, IntUniformDistribution, and IntLogUniformDistribution (#3246, #3420)create_study(), load_study(), delete_study(), and create_study() (#3270)axis_order argument of plot_pareto_front() (#3341)Features removed in 3.0:
optuna dashboard command (#3058)optuna.structs module (#3057)best_booster property of LightGBMTuner (#3057)type_checking module (#3235)Minor breaking changes:
_run_trial to study.tell (#3144)get_study_id_from_trial_id, the method of BaseStorage (#3538)plot_optimization_history (#2807)MLflowCallback interface (#2912, thanks @xadrianzetx!)trial.user_attrs logging optional in MLflowCallback (#3043, thanks @xadrianzetx!)IntDistribution & FloatDistribution (#3063, thanks @nyanhi!)trial.user_attrs to pareto_front hover text (#3082, thanks @kasparthommen!)optuna tell with --skip-if-finished (#3131)constraints_func in plot_pareto_front (#3128, thanks @semiexp!)skip_if_finished flag to Study.tell (#3150, thanks @xadrianzetx!)user_attrs argument to Study.enqueue_trial (#3185, thanks @knshnb!)RetryFailedTrialCallback (#3269, thanks @knshnb!)DiscreteUniformDistribution.q (#3283)create_trial (#3196)CatBoostPruningCallback (#2734, thanks @tohmae!)Trial.system_attrs (#3223, thanks @belltailjp!)study.py (#3309)targets and deprecate axis_order in optuna.visualization.matplotlib.plot_pareto_front (#3341, thanks @shu65!)targets argument to plot_pareto_plont of plotly backend (#3495, thanks @TakuyaInoue-github!)constraints_func in plot_pareto_front in matplotlib visualization (#3497, thanks @fukatani!)GridSampler reproducible (#3527, thanks @gasin!)ValueError with warning in GridSearchSampler (#3545)callbacks argument of OptunaSearchCV (#3577)RDBStorage (#3581)TPESampler (#3506)skip_if_exists argument to enqueue_trial (#3629)plot_pareto_front (#3643)popsize argument to CmaEsSampler (#3649)seed argument for BoTorchSampler (#3756)seed argument for SkoptSampler (#3791)get_trial_id_from_study_id_trial_number (#3909)BoTorchSampler (#2928)import optuna (#3000)_contains of IntLogUniformDistribution (#3005)matplotlib.plot_param_importances (#3012, thanks @xadrianzetx!)verbose_eval NoneN for LightGBMTuner/LightGBMTunerCV` to avoid conflict (#3014, thanks @chezou!)plot_contour (#3017)FixedTrial and FrozenTrial allowing not-contained parameters during suggest_* (#3018)optuna ask CLI receives --sampler-kwargs without --sampler (#3029)_get_removed_version_from_deprecated_version function (#3065, thanks @nuka137!)plotly.plot_param_importances (#3073, thanks @xadrianzetx!)plot_contour using SciPy's spsolve (#3092)directions, user_attrs and system_attrs of study summaries (#3108)FloatDistribution across codebase (#3111, thanks @xadrianzetx!)json.loads to decode pruner configuration loaded from environment variables (#3114)timeout (#3115, thanks @xadrianzetx!)IntDistribution across codebase (#3126, thanks @nyanhi!)RedisStorage in CachedStorage (#3204, thanks @masap!)functools.wraps in track_in_mlflow decorator (#3216)RedisStorage fast when running multiple trials (#3262, thanks @masap!)Study.ask() (#3274, thanks @masap!)study.tell() (#3265, thanks @masap!)optuna.study.get_all_study_summaries() of RedisStorage fast (#3278, thanks @masap!)plotly visualization methods (#3376)filter_nonfinite (#3438)precision of sqlalchemy.Float in RDBStorage table definition (#3327)nan in trial.report (#3348, thanks @belldandyxtq!)set_trial_param() of RedisStorage faster (#3391, thanks @masap!)_set_best_trial() of RedisStorage faster (#3392, thanks @masap!)set_study_directions() of RedisStorage faster (#3393, thanks @masap!)plot_slice, plot_pareto_front, and plot_optimization_history (#3449, thanks @dubey-anshuman!)BaseStorage (#3475)torch.distributed calls from TorchDistributedTrial properties (#3490, thanks @nlgranger!)trial_values table in RDB (#3559)intermediate_value_type column to represent inf/-inf on RDBStorage (#3564)is_heartbeat_enabled from storage to heartbeat (#3596)ImportanceEvaluators (#3597)bayesmark benchmark report (#3693)inf values for crowding distance (#3743)sample_relative and fix type of return values of SkoptSampler and PyCmaSampler (#2897)GridSampler with RetryFailedTrialCallback or enqueue_trial (#2946)trial.values in MLflow integration (#2991)ValueError for invalid q in DiscreteUniformDistribution (#3001)trial.report during sanity check (#3002)matplotlib.plot_contour bug (#3046, thanks @IEP!)single distributions in fANOVA evaluator (#3085, thanks @xadrianzetx!)TPESampler when group=True (#3187, thanks @xuzijian629!)matplotlib.contour_plot (#3213, thanks @xadrianzetx!)matplotlib.contour_plot (#3218, thanks @xadrianzetx!)-inf and inf values in RDBStorage (#3238, thanks @xadrianzetx!)nan (#3286)matplotlib contour plot (#3249, thanks @harupy!)fail_state_trials show warning when heartbeat is enabled (#3301)user_attrs and system_attrs in study summaries (#3352)matplotlib.plot_parallel_coordinate with log distributions (#3371)inf values from visualizations (#3395)best_index_ (#3410)px.colors.sequential.Blues that introduces pandas dependency (#3422)_is_reverse_scale (#3424)COLOR_SCALE inside import util context (#3492)-v option of optuna study set-user-attr command (#3499, thanks @nyanhi!)optuna.visualization.plot_param_importances and optuna.visualization.matplotlib.plot_param_importance (#3500, thanks @takoika!)--verbose and --quiet options in CLI (#3532, thanks @nyanhi!)ValueError with RuntimeError in get_best_trial (#3541)CategoricalDistribution by GridSampler (#3544)CategoricalDistribution with NaN (#3567)IntersectionSearchSpace (#3666)trial_values records whose values are None (#3668)_constrained_dominates (#3738)inf-related issue on implementation of _calculate_nondomination_rank (#3739)_calculate_weights such that it throws ValueError on invalid weights (#3742)axis_order of plot_pareto_front (#3802)ValueError when waiting trial is told (#3814)Study.tell with invalid values (#3819)tensorflow and tensorflow-estimator versions to <2.7.0 (#3059)keras version to <2.7.0 (#3078)tensorflow (#3084)torch related packages (#3156)pytorch-lightning>=1.5.0 (#3157)mlflow integration (#3170)nltk version (#3201)setuptools (#3207)setuptools (#3231)fastai job on Python 3.6 (#3412)click==8.1.0 that removed a deprecated feature (#3413)click==8.1.0 that removed a deprecated feature" (#3430)setup.py (#3517)document section (#3613)fakeredis (#3905)typing_extensions to use ParamSpec (#3926)trial.report (#2980)pytorch-lightning (#2984)OptunaSearchCV about direction (#3007)n_trials in the docs (#3016, thanks @Rohan138!)optuna tell (#3052)logging.set_verbosity (#3061, thanks @drumehiron!)002_configurations.py in the Trial API page (#3067, thanks @makkimaki!)003_efficient_optimization_algorithms.py in the Trial API page (#3068, thanks @makkimaki!)set_user_attrs in Study to the user_attrs entry in Tutorial (#3069, thanks @MasahitoKumada!)suggest_float docstring (#3091, thanks @xadrianzetx!)ValueError and TypeErorr to Raises section of Trial.report (#3124, thanks @MasahitoKumada!)logging_callback only works in single process situation (#3143)FrozenTrial's docstring (#3161)README.md (#3167)Study.optimize from API page (#3171, thanks @xuzijian629!)study.enqueue_trial (#3172, thanks @knshnb!)distributions documentation (#3222, thanks @xadrianzetx!)Raises section of FloatDistribution docstring (#3248, thanks @xadrianzetx!){Float,Int}Distribution to docs (#3252)AllenNLPExecutor (#3253)DiscreteUniformDistribution.q (#3279)step behavior to new distributions (#3276)show_progress_bar (#3287)copy_study: it creates a copy regardless of its state (#3295)Raises doc section (#3315)QMCSampler in tutorial (#3320)plot_param_importances (#3332, thanks @ll7!)optuna study optimize in FAQ (#3364)best_trial (#3396, thanks @divyanshugit!)Study.best_trials in multi-objective optimization tutorial (#3443)classifier to regressor in the code snippet of README.md (#3481)CONTRIBUTING.md (#3482)benchmarks/README.md for the bayesmark section (#3496)Study.stop as a criteria to stop creating trials in document (#3498, thanks @takoika!)study.optimize (#3505)README.md (#3508)FronzenTrial's docstring (#3514)GridSampler's seed option (#3568)youtube.com with youtube-nocookie.com (#3590)language from docs configuration (#3594)optuna-fast-fanova (#3647)Study.optimize (#3720, thanks @29Takuya!)CONTRIBUTING.md (#3726)plot_pareto_front's axis_order (#3803)prepare_study_with_trials (#3809)ShapleyImportanceEvaluator (#3810)trail with trial (#3861).. seealso:: in Study.get_trials and Study.trials (#3862, thanks @jmsykes83!)TrialState.is_finished (#3869)FrozenTrial (#3872, thanks @wattlebirdaz!)NSGAIISampler docs informative (#3880)constant_liar with multi-objective function (#3881)copybutton_prompt_text not to copy the bash prompt (#3882)HyperbandPruner (#3894)HyperBandPruner (#3901)CatBoostPruningCallback (#3903)RetryFailedTrialCallback in pytorch_checkpoint example (https://github.com/optuna/optuna-examples/pull/59, thanks @xadrianzetx!)suggest_uniform with suggest_float (https://github.com/optuna/optuna-examples/pull/63)lightgbm (https://github.com/optuna/optuna-examples/pull/64)tensorflow and tensorflow-estimator versions to <2.7.0 (https://github.com/optuna/optuna-examples/pull/66)pytorch-lightning (https://github.com/optuna/optuna-examples/pull/67)README.md (https://github.com/optuna/optuna-examples/pull/68, thanks @solegalli!)nltk version (https://github.com/optuna/optuna-examples/pull/75)setuptools (https://github.com/optuna/optuna-examples/pull/76)setuptools (https://github.com/optuna/optuna-examples/pull/79)haiku's CI (https://github.com/optuna/optuna-examples/pull/83)black 22.1.0 & run checks daily (https://github.com/optuna/optuna-examples/pull/84)hiplot example (https://github.com/optuna/optuna-examples/pull/86)CatBoostPruningCallback (https://github.com/optuna/optuna-examples/pull/92)fastai job on Python 3.6 (https://github.com/optuna/optuna-examples/pull/93)str in workflow files (https://github.com/optuna/optuna-examples/pull/95)SimulatedAnnealingSampler to support FloatDistribution (https://github.com/optuna/optuna-examples/pull/97)sklearn example (https://github.com/optuna/optuna-examples/pull/102, thanks @MasahitoKumada!)allennlp.yml (https://github.com/optuna/optuna-examples/pull/104)visualization.yml (https://github.com/optuna/optuna-examples/pull/109)protobuf in PyTorch Lightning example (https://github.com/optuna/optuna-examples/pull/116)thop (https://github.com/optuna/optuna-examples/pull/123)allennlp dependency (https://github.com/optuna/optuna-examples/pull/124)pytorch_simple.py (https://github.com/optuna/optuna-examples/pull/125)OMPI_MCA_rmaps_base_oversubscribe=yes before mpirun (https://github.com/optuna/optuna-examples/pull/126)python-version (https://github.com/optuna/optuna-examples/pull/127)sklearn (https://github.com/optuna/optuna-examples/pull/128)catboost integration line to integration section from pruning section (https://github.com/optuna/optuna-examples/pull/129)skimage example (https://github.com/optuna/optuna-examples/pull/130)cached-path (https://github.com/optuna/optuna-examples/pull/133)pytorch-lightning (#2983)np.asarray in lightgbm test (#2997)suggest APIs across codebase (#3027, thanks @xadrianzetx!)return_cvbooster of LightGBMTuner consistent to the original value (#3070, thanks @abatomunkuev!)parametrize_sampler (#3080)tests/integration_tests/lightgbm_tuner_tests/test_optimize.py (#3086, thanks @nyanhi!)matplotlib.plot_slice (#3121)visualization_tests/matplotlib_tests/test_slice.py (#3175, thanks @keisukefukuda!)matplotlib.plot_edf (#3178, thanks @makinzm!)plot_edf (#3188, thanks @makinzm!)matplotlib.contour_plot test (#3232, thanks @xadrianzetx!)plot_parallel_coordinate (#3266, thanks @MasahitoKumada!)matplotlib_tests/test_param_importances (#3180, thanks @belldandyxtq!)plot_optimization_history methods consistent (#3234)RedisStorage (#3258, thanks @masap!)catalyst integration test (#3308)MLflowCallback tests (#3378)matplotlib parallel coordinate test (#3368)matplotlib tests (#3414, thanks @divyanshugit!)inf test to intermediate values test (#3466)test_storages.py (#3480)optuna.visualization.plot_pareto_front (#3546)test_storages.py to another file (#3553)seed method of np.random.RandomState for reseeding and fix test_reseed_rng (#3569)test_get_observation_pairs (#3574)inf/nan objectives for ShapleyImportanceEvaluator (#3576)TypeError (#3667)_constrained_dominates (#3683)inf and NaN tests for test_constraints_func (#3690)test_frozen.py (#3696)plot_contours (#3701)NSGAIISampler._crowding_distance_sort (#3706)test_calculate_weights_below (#3741)test_intermediate_plot.py (#3745)_dominates function (#3764)create_trial (#3794)with_c_d option from prepare_study_with_trials (#3799)DeterministicRelativeSampler in test_trial.py (#3807)_fast_non_dominated_sort (#3686)plot_parallel_coordinates (#3800)multi_objective module (#3911)warnings: UserWarning from tests/visualization_tests/test_utils.py (#3919, thanks @jmsykes83!)BaseStudy (#2986, thanks @twsl!)optuna.load_study in optuna ask CLI to omit direction/directions option (#2989)Trial warning message (#3008, thanks @xadrianzetx!)suggest APIs (#3054, thanks @xadrianzetx!)remove_version to the missing @deprecated argument (#3064, thanks @nuka137!)optuna.logging.get_verbosity (#3066, thanks @MasahitoKumada!){Float|Int}Distribution in NSGA-II crossover operators (#3139, thanks @xadrianzetx!)FloatDistribution (#3166, thanks @xadrianzetx!)deprecated decorator of the feature of n_jobs (#3173, thanks @MasahitoKumada!)MaxTrialsCallback (#3261, thanks @knshnb!)_check_trial_id (#3264, thanks @masap!){Int,Float}Distribution (#3244)IntDistribution (#3181, thanks @nyanhi!)UniformDistribution, LogUniformDistribution and DiscreteUniformDistribution code paths (#3275)set_trial_state() and set_trial_values() into one function (#3323, thanks @masap!){Float, Int}Distributions (#3337, thanks @nyanhi!)get_trial_xxx abstract functions to base (#3338, thanks @belldandyxtq!)states (#3359, thanks @BasLaa!)RedisStorage (#3394, thanks @masap!)tests/integration_tests (#3408)filter_nonfinite (#3436)RetryFailedTrialCallback to optuna.storages.* (#3441)fail_stale_trials in each storage implementation (#3442, thanks @knshnb!)matplotlib.plot_parallel_coordinate (#3415)_log_completed_trial (#3551)copy.deepcopy() calls in importance module (#3554)check_trial_is_updatable (#3557)optuna.testing.integration.create_running_trial with study.ask (#3562)test_get_observation_pairs (#3574)constraints_func is specified (#3587)no-implicit-optional for mypy (#3599, thanks @harupy!)warn_redundant_casts for mypy (#3602, thanks @harupy!)TrialIntermediateValueModel (#3603)mypy checks of Alembic's get_current_head() method (#3608)_optimize.py to _heartbeat.py (#3609)except clauses (#3632, thanks @harupy!)optuna.testing.integration with optuna.testing.pruner (#3638)strict_equality for mypy #3579 (#3648, thanks @wattlebirdaz!)optuna module (#3657)read_trials_from_remote_storage in the all storages apart from CachedStorage (#3659)mypy bug (#3679)plot_contours (#3682)storage.get_all_study_summaries(include_best_trial: bool) (#3697, thanks @wattlebirdaz!)plot_param_importances functions (#3700)disallow_untyped_calls for mypy (#3704, thanks @29Takuya!)get_trials with states argument to filter trials depending on trial state (#3708)bayesmark benchmark report rendering (#3725)plot_parallel_coordinates (#3734)plot_optimization_history between plotly and matplotlib (#3736)fail_objective and pruned_objective for tests (#3737)visualization/_pareto_front.py (#3752)_ParetoInfoType (#3753)_ContourInfo to plot in plot_contour (#3755)no_trials option of prepare_study_with_trials (#3766)plot_contour files (#3767)ValueError for invalid returned type of target in _filter_nonfinite (#3768)plot_contour (#3769)Sampler.after_trial (#3775)stop_objective (#3786)plot_contour test (#3787)less_than_two and more_than_three options from prepare_study_with_trials (#3789)_get_node_value (#3818)type: ignore (#3832)QMCSampler (#3837)plot_param_importances (#3760)test_pareto_front (#3798)CategoricalChoiceType from optuna.distributions (#3846)BaseStorage.get_trial_id_from_study_id_trial_number (#3870, thanks @wattlebirdaz!)BaseStorage.get_best_trial (#3871, thanks @wattlebirdaz!)IntersectionSearchSpace.calculate (#3887)q with step in private function and warning message (#3913)tests/sampler_tests (#3921)botorch to CI jobs on mac (#2988)mac-tests CI at a scheduled time (#3028)mypy version to 0.910 (#3123)kurobako (#3155)bayesmark (#3354)setuptools (#3427)coverage directly (#3347, thanks @higucheese!)reviewdog (#3357)bayesmark benchmark results comparable to kurobako (#3584)virtualenv for benchmark extras (#3585)protobuf<4.0.0 to resolve Sphinx CI error (#3591)protobuf (#3598, thanks @harupy!)warn_unused_ignores for mypy (#3627, thanks @harupy!)onnx and version constrained protobuf to document dependencies (#3658)mo-kurobako benchmark to CI (#3691)libomp for mac tests (#3728)bayesmark CI jobs (#3750)OMPI_MCA_rmaps_base_oversubscribe=yes before mpirun (#3758)budget option to benchmarks (#3774)n_concurrency option to benchmarks (#3776)n-runs instead of repeat to represent the number of studies in the bayesmark benchmark (#3780)mypy 0.971 (#3797)trial.report in PyTorchLightningPruningCallback (#3842)README.md (#2999)tox.ini (#3025)v3.0.0b0.dev (#3289)question-and-help-support (#3305)v3.0.0a2 (#3314)v3.0.0-b0 (#3458)kurobako benchmark code to run it locally (#3468)fakeredis 1.7.4 release (#3549)fakeredis (#3561)fakeredis (#3607)tox.ini consistent with checking (#3654)This release was made possible by the authors and the people who participated in the reviews and discussions.
@29Takuya, @BasLaa, @CorentinNeovision, @Crissman, @HideakiImamura, @Hiroyuki-01, @IEP, @MasahitoKumada, @Rohan138, @TakuyaInoue-github, @abatomunkuev, @akawashiro, @andriyor, @avats-dev, @belldandyxtq, @belltailjp, @c-bata, @captain-pool, @cfkazu, @chezou, @contramundum53, @divyanshugit, @drumehiron, @dubey-anshuman, @fukatani, @g-votte, @gasin, @harupy, @higucheese, @himkt, @hppRC, @hvy, @jmsykes83, @kasparthommen, @kei-mo, @keisuke-umezawa, @keisukefukuda, @knshnb, @kstoneriv3, @liaison, @ll7, @makinzm, @makkimaki, @masaaldosey, @masap, @nlgranger, @not522, @nuka137, @nyanhi, @nzw0301, @semiexp, @shu65, @sidshrivastav, @sile, @solegalli, @takoika, @tohmae, @toshihikoyanase, @tsukudamayo, @tupui, @twsl, @wattlebirdaz, @xadrianzetx, @xuzijian629, @y0z, @yoshinobc, @ytsmiling
Add note for deprecation of plot_pareto_front's axis_order
This is the release note of v3.0.0-rc0. This is a release candidate of Optuna V3. We plan to release the major version within a few weeks. Please try this version and report bugs!
TPESampler, the default sampler of Optuna, now supports constrained optimization. It takes a function constraints_func as an argument, and examines whether trials are feasible or not. Feasible trials are prioritized over infeasible ones similarly to NSGAIISampler. See #3506 for more details.
def objective(trial):
# Binh and Korn function with constraints.
x = trial.suggest_float("x", -15, 30)
y = trial.suggest_float("y", -15, 30)
# Store the constraints as user attributes so that they can be restored after optimization.
c0 = (x - 5) ** 2 + y ** 2 - 25
c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7
trial.set_user_attr("constraints", (c0, c1))
v0 = 4 * x ** 2 + 4 * y ** 2
v1 = (x - 5) ** 2 + (y - 5) ** 2
return v0, v1
def constraints(trial):
return trial.user_attrs["constraints"]
if __name__ == "__main__":
sampler = optuna.samplers.TPESampler(
constraints_func=constraints,
)
study = optuna.create_study(
directions=["minimize", "minimize"],
sampler=sampler,
)
study.optimize(objective, n_trials=1000)
optuna.visualization.plot_pareto_front(study, constraints_func=constraints).show()
| MOTPE without constraints | MOTPE with constraints |
|---|---|
We have undertaken major refactoring of the visualization features as one of the major tasks of Optuna V3. The current situation is as follows.
plotly and matplotlibHistorically, the implementations of Optuna's visualization features were split between two different backends, plotly and matplotlib. Many of these implementations were duplicated and unmaintainable, and many were implemented as a single large function, resulting in poor testability and, as a result, becoming the cause of many bugs. We clarified the specifications that each visualization function in Optuna must meet and defined the backend-independent information needed to perform the visualization. By using this information commonly across different backends, we achieved a highly maintainable and testable implementation, and improved the stability of the visualization functions dramatically. We are currently rewriting the unit tests, and the resulting tests will be simple yet powerful.
It is very important to detect hidden bugs in the implementation through PR reviews. However, visualizations are likely to contain bugs that are difficult to find just by reading the code, and many of these bugs are only revealed when the visualization is actually performed. Therefore, we introduced the Visual Regression Test to improve the review process. In the PR for visualization features, you can jump to the Visual Regression Test link by clicking on the link generated from within the PR. Reviewers can verify that the PR implementation is performing the visualization properly.
<img width="1715" alt="173838319-24433136-bd59-47d5-afdb-2694aafe354d (1)" src="https://user-images.githubusercontent.com/38826298/183350545-bbe74c09-4721-461b-b3b5-cdba5092403f.png">
In the latter development cycle of Optuna v3, we put emphasis on improving the overall code quality of the library. We fixed several bugs and possible corruption of internal data structures on e.g. handling Inf/NaN values (#3567, #3592, #3738, #3739, #3740) and invalid inputs (#3668, #3808, #3814, #3819). For example, there had been bugs before v3 when NaN values were used in a CategoricalDistribution or GridSampler. In several other functions, NaN values were unacceptable but the library failed silently without any warning or error. Such bugs are fixed in this release.
TPESampler (#3506)skip_if_exists argument to enqueue_trial (#3629)plot_pareto_front (#3643)popsize argument to CmaEsSampler (#3649)seed argument for BoTorchSampler (#3756)seed argument for SkoptSampler (#3791)is_heartbeat_enabled from storage to heartbeat (#3596)ImportanceEvaluators (#3597)bayesmark benchmark report (#3693)inf values for crowding distance (#3743)CategoricalDistribution with NaN (#3567)IntersectionSearchSpace (#3666)trial_values records whose values are None (#3668)_constrained_dominates (#3738)inf-related issue on implementation of _calculate_nondomination_rank (#3739)_calculate_weights such that it throws ValueError on invalid weights (#3742)axis_order of plot_pareto_front (#3802)ValueError when waiting trial is told (#3814)Study.tell with invalid values (#3819)optuna-fast-fanova (#3647)Study.optimize (#3720, thanks @29Takuya!)CONTRIBUTING.md (#3726)plot_pareto_front's axis_order (#3803)prepare_study_with_trials (#3809)ShapleyImportanceEvaluator (#3810)thop (https://github.com/optuna/optuna-examples/pull/123)allennlp dependency (https://github.com/optuna/optuna-examples/pull/124)pytorch_simple.py (https://github.com/optuna/optuna-examples/pull/125)OMPI_MCA_rmaps_base_oversubscribe=yes before mpirun (https://github.com/optuna/optuna-examples/pull/126)TypeError (#3667)_constrained_dominates (#3683)inf and NaN tests for test_constraints_func (#3690)test_frozen.py (#3696)plot_contours (#3701)NSGAIISampler._crowding_distance_sort (#3706)test_calculate_weights_below (#3741)test_intermediate_plot.py (#3745)_dominates function (#3764)create_trial (#3794)with_c_d option from prepare_study_with_trials (#3799)DeterministicRelativeSampler in test_trial.py (#3807)_optimize.py to _heartbeat.py (#3609)except clauses (#3632, thanks @harupy!)optuna.testing.integration with optuna.testing.pruner (#3638)strict_equality for mypy #3579 (#3648, thanks @wattlebirdaz!)optuna module (#3657)read_trials_from_remote_storage in the all storages apart from CachedStorage (#3659)mypy bug (#3679)plot_contours (#3682)storage.get_all_study_summaries(include_best_trial: bool) (#3697, thanks @wattlebirdaz!)plot_param_importances functions (#3700)disallow_untyped_calls for mypy (#3704, thanks @29Takuya!)get_trials with states argument to filter trials depending on trial state (#3708)bayesmark benchmark report rendering (#3725)plot_parallel_coordinates (#3734)plot_optimization_history between plotly and matplotlib (#3736)fail_objective and pruned_objective for tests (#3737)visualization/_pareto_front.py (#3752)_ParetoInfoType (#3753)_ContourInfo to plot in plot_contour (#3755)no_trials option of prepare_study_with_trials (#3766)plot_contour files (#3767)ValueError for invalid returned type of target in _filter_nonfinite (#3768)plot_contour (#3769)Sampler.after_trial (#3775)stop_objective (#3786)plot_contour test (#3787)less_than_two and more_than_three options from prepare_study_with_trials (#3789)_get_node_value (#3818)type: ignore (#3832)QMCSampler (#3837)coverage directly (#3347, thanks @higucheese!)bayesmark benchmark results comparable to kurobako (#3584)warn_unused_ignores for mypy (#3627, thanks @harupy!)onnx and version constrained protobuf to document dependencies (#3658)mo-kurobako benchmark to CI (#3691)libomp for mac tests (#3728)bayesmark CI jobs (#3750)OMPI_MCA_rmaps_base_oversubscribe=yes before mpirun (#3758)budget option to benchmarks (#3774)n_concurrency option to benchmarks (#3776)n-runs instead of repeat to represent the number of studies in the bayesmark benchmark (#3780)mypy 0.971 (#3797)trial.report in PyTorchLightningPruningCallback (#3842)tox.ini consistent with checking (#3654)This release was made possible by the authors and the people who participated in the reviews and discussions.
@29Takuya, @HideakiImamura, @c-bata, @cfkazu, @contramundum53, @g-votte, @harupy, @higucheese, @himkt, @hvy, @keisuke-umezawa, @knshnb, @not522, @nzw0301, @sile, @toshihikoyanase, @wattlebirdaz, @xadrianzetx, @y0z
Add deprecated warning test to the multi-objective sampler test file
This is the release note of v3.0.0-b1.
We added a sampler comparison table on the samplers' documentation page. It includes supported options (parameter types, pruning, multi-objective optimization, constrained optimization, etc.), time complexity, and recommended budgets for each sampler. Please use this to select appropriate samplers for your tasks! See #3571 and #3593 for more details.
<img width="1224" alt="sampler_comparison_table" src="https://user-images.githubusercontent.com/38826298/172111648-eb56206f-8539-48ac-a0ab-b1cc199e2d22.png">
ShapleyImportanceEvaluatorOptuna now supports mean absolute SHAP value for evaluating parameter importances through integration with the SHAP library. SHAP value is a game-theoretic measure of parameter importance featuring nice theoretical properties (See paper for more information).
<img width="469" alt="168213146-465a8116-94f2-49c9-b4ce-ec12970d82f8" src="https://user-images.githubusercontent.com/38826298/172112327-cda4a831-e1aa-486d-a481-68a44b14637c.png">
To use mean absolute SHAP importances, an object of optuna.integration.shap.ShapleyImportanceEvaluator can be passed to evaluator argument in optuna.visualization.plot_param_importances or optuna.importance.get_param_importances.
import optuna
from optuna.integration.shap import ShapleyImportanceEvaluator
study = optuna.create_study()
study.optimize(objective, n_trials=100)
optuna.visualization.plot_param_importances(study, evaluator=ShapleyImportanceEvaluator())
See the #3507 for more details.
The benchmarking environment for black-box optimization algorithms on the GitHub Actions was introduced in the previous release. We have further enhanced its capabilities. The benchmarking functionality introduced can be run on all users' forks using GitHub Actions. You can also freely customize and run benchmarks on more computationally powerful clusters, for example, AWS, using the code in the optuna/benchmarks directory.
Optuna's algorithms can now be benchmarked using NASLib, the Neural Architecture Search benchmark library. For now we only support one dataset, NASBench 201, which deals with image recognition. Larger datasets and datasets from other areas such as natural language processing will be supported in the future.
| cifar10 | cifar100 | imagenet16-120 |
|---|---|---|
See README and #3465 for more information.
We are now able to benchmark our multi-objective optimization algorithms. They are not yet available on GitHub Actions, but you can use optuna/benchmarks/run_mo_kurobakmo.py directly. They will be available on GitHub Actions in the next release, so stay tuned! See #3271 and #3349 for more details.
This is the first version to officially support Python 3.10. All tests are passed including integration modules, with a few exceptions.
To use Optuna v3.0.0-b1 with RDBStorage that was created in the previous versions of Optuna, please run optuna storage upgrade to migrate your database.
# `YOUR_RDB_URL` is the URL of your database.
optuna storage upgrade –storage YOUR_RDB_URL
If you use RedisStorage, copy your study with RDBStorage using copy_study with the Optuna you used to create the study, thenrun optuna storage upgrade with Optuna v3.0.0-b0. After upgrading the storage, copy the study back as a new RedisStorage.
python -c ‘import optuna; optuna.copy_study(from_study_name=”example”, from_storage=”redis://localhost:6379”, to_storage=”sqlite:///upgrade.db”)
pip install –pre -U optuna
optuna storage upgrade –storage sqlite:///upgrade.db
python -c ‘import optuna; optuna.copy_study(from_study_name="example", from_storage="sqlite:///upgrade.db", to_study_name="new-example", to_storage="redis://localhost:6379")’
get_study_id_from_trial_id (#3538)targets argument to plot_pareto_plont of plotly backend (#3495, thanks @TakuyaInoue-github!)constraints_func in plot_pareto_front in matplotlib visualization (#3497, thanks @fukatani!)GridSampler reproducible (#3527, thanks @gasin!)ValueError with warning in GridSearchSampler (#3545)callbacks argument of OptunaSearchCV (#3577)RDBStorage (#3581)precision of sqlalchemy.Float in RDBStorage table definition (#3327)nan in trial.report (#3348, thanks @belldandyxtq!)set_trial_param() of RedisStorage faster (#3391, thanks @masap!)_set_best_trial() of RedisStorage faster (#3392, thanks @masap!)set_study_directions() of RedisStorage faster (#3393, thanks @masap!)plot_slice, plot_pareto_front, and plot_optimization_history (#3449, thanks @dubey-anshuman!)BaseStorage (#3475)torch.distributed calls from TorchDistributedTrial properties (#3490, thanks @nlgranger!)trial_values table in RDB (#3559)intermediate_value_type column to represent inf/-inf on RDBStorage (#3564)COLOR_SCALE inside import util context (#3492)-v option of optuna study set-user-attr command (#3499, thanks @nyanhi!)optuna.visualization.plot_param_importances and optuna.visualization.matplotlib.plot_param_importance (#3500, thanks @takoika!)--verbose and --quiet options in CLI (#3532, thanks @nyanhi!)ValueError with RuntimeError in get_best_trial (#3541)CategoricalDistribution by GridSampler (#3544)setup.py (#3517)document section (#3613)classifier to regressor in the code snippet of README.md (#3481)CONTRIBUTING.md (#3482)benchmarks/README.md for the bayesmark section (#3496)Study.stop as a criteria to stop creating trials in document (#3498, thanks @takoika!)study.optimize (#3505)README.md (#3508)FronzenTrial's docstring (#3514)GridSampler's seed option (#3568)youtube.com with youtube-nocookie.com (#3590)language from docs configuration (#3594)sklearn example (https://github.com/optuna/optuna-examples/pull/102, thanks @MasahitoKumada!)allennlp.yml (https://github.com/optuna/optuna-examples/pull/104)visualization.yml (https://github.com/optuna/optuna-examples/pull/109)protobuf in PyTorch Lightning example (https://github.com/optuna/optuna-examples/pull/116)matplotlib parallel coordinate test (#3368)matplotlib tests (#3414, thanks @divyanshugit!)inf test to intermediate values test (#3466)test_storages.py (#3480)optuna.visualization.plot_pareto_front (#3546)test_storages.py to another file (#3553)seed method of np.random.RandomState for reseeding and fix test_reseed_rng (#3569)test_get_observation_pairs (#3574)inf/nan objectives for ShapleyImportanceEvaluator (#3576)matplotlib.plot_parallel_coordinate (#3415)_log_completed_trial (#3551)copy.deepcopy() calls in importance module (#3554)check_trial_is_updatable (#3557)optuna.testing.integration.create_running_trial with study.ask (#3562)test_get_observation_pairs (#3574)constraints_func is specified (#3587)no-implicit-optional for mypy (#3599, thanks @harupy!)warn_redundant_casts for mypy (#3602, thanks @harupy!)TrialIntermediateValueModel (#3603)mypy checks of Alembic's get_current_head() method (#3608)reviewdog (#3357)virtualenv for benchmark extras (#3585)protobuf<4.0.0 to resolve Sphinx CI error (#3591)protobuf (#3598, thanks @harupy!)kurobako benchmark code to run it locally (#3468)fakeredis 1.7.4 release (#3549)fakeredis (#3561)fakeredis (#3607)This release was made possible by the authors and the people who participated in the reviews and discussions.
@HideakiImamura, @MasahitoKumada, @TakuyaInoue-github, @belldandyxtq, @c-bata, @captain-pool, @contramundum53, @divyanshugit, @drumehiron, @dubey-anshuman, @fukatani, @g-votte, @gasin, @harupy, @himkt, @hvy, @kei-mo, @keisuke-umezawa, @knshnb, @liaison, @masap, @nlgranger, @not522, @nyanhi, @nzw0301, @semiexp, @sile, @takoika, @toshihikoyanase, @xadrianzetx
…and IntLogUniformDistribution are deprecated. If you pass deprecated distributions to APIs such as Study.ask or create_trial, they are internally conv…
This is the release note of v3.0.0-b0.
Search space definitions, which consist of BaseDistribution and its child classes in Optuna, are greatly simplified. We have introduced FloatDistribution, IntDistribution, and CategoricalDistribution. If you use the suggest API and Study.optimize, the search space information is stored as these three distributions. Previous UniformDistribution, LogUniformDistribution, DiscreteUniformDistribution, IntUniformDistribution, and IntLogUniformDistribution are deprecated. If you pass deprecated distributions to APIs such as Study.ask or create_trial, they are internally converted to corresponding FloatDistribution or IntDistribution.
To use Optuna v3.0.0-b0 with RDBStorage that was created in the previous versions of Optuna, please run optuna storage upgrade to migrate your database.
If you use RedisStorage, copy your study with RDBStorage using copy_study with the Optuna you used to create the study, thenrun optuna storage upgrade with Optuna v3.0.0-b0. After upgrading the storage, copy the study back as a new RedisStorage.
python -c ‘import optuna; optuna.copy_study(from_study_name=”example”, from_storage=”redis://localhost:6379”, to_storage=”sqlite:///upgrade.db”)
pip install –pre -U optuna
optuna storage upgrade –storage sqlite:///upgrade.db
python -c ‘import optuna; optuna.copy_study(from_study_name="example", from_storage="sqlite:///upgrade.db", to_study_name="new-example", to_storage="redis://localhost:6379")’
Study.optimizeStudy.tell fails a trial when it is called with certain invalid combinations of state and values, instead of raising an error. This change aims to make Study.tell consistent with Study.optimize, which continues an optimization even if an objective returns an invalid value.
Study.tell now also returns the resulting trial (FrozenTrial) in order to allow inspecting how the arguments were interpreted.
Study.tell raises an exception when it is called with an invalid combination of state and values.
study.tell(study.ask(), values=None)
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# File "/…/optuna/optuna/study/study.py", line 579, in tell
# raise ValueError(
# ValueError: No values were told. Values are required when state is TrialState.COMPLETE.
Study.tell automatically fails the trial.
trial: FrozenTrial = study.tell(study.ask(), value=None)
assert trial.state == TrialState.FAIL
See #3144 for more details.
Study APIsWe are converting all positional arguments of create_study, delete_study, load_study, and copy_study to keyword-only arguments since the order of arguments were inconsistent. This is not yet a breaking-change, but if you use these features with positional arguments, then you will get a warning message to use them with keyword-only arguments.
In addition, we have fixed all of problems described in #2955, so we have stabled the Study APIs. Specifically, Study.add_trial, Study.add_trials, Study.enqueue_trial, and copy_study have been stabled.
See #3270 and #2955 for more details.
Several bugs in the visualization module have been resolved. For instance,
the parallel coordinates plot ignores trials with missing parameters (#3373) and the scale of the objective value is fixed (#3369). The edf plot filters trials with inf values (#3395 and #3435).
Before: Trials with missing parameters are wrongly connected to each other. <img width="1203" alt="158495812-acb399e5-d817-4cae-8c8b-c69bcc91efea" src="https://user-images.githubusercontent.com/38826298/162878578-fc162708-5c3b-45a0-b098-47445a1242a7.png">
After: Trials with missing parameters are removed from the plot. <img width="1191" alt="158495810-98a5ec29-581c-426a-a255-11d16ae1c144" src="https://user-images.githubusercontent.com/38826298/162878606-aa937868-b8f5-4b51-b09b-6aced6fd28ff.png">
{Float,Int}Distribution using alembic (#3113)_run_trial to study.tell (#3144)FloatDistribution and IntDistribution (#3246)CatBoostPruningCallback (#2734, thanks @tohmae!)Trial.system_attrs (#3223, thanks @belltailjp!)study.py (#3309)targets and deprecate axis_order in optuna.visualization.matplotlib.plot_pareto_front (#3341, thanks @shu65!)study.tell() (#3265, thanks @masap!)optuna.study.get_all_study_summaries() of RedisStorage fast (#3278, thanks @masap!)plotly visualization methods (#3376)filter_nonfinite (#3438)matplotlib contour plot (#3249, thanks @harupy!)fail_state_trials show warning when heartbeat is enabled (#3301)user_attrs and system_attrs in study summaries (#3352)matplotlib.plot_parallel_coordinate with log distributions (#3371)inf values from visualizations (#3395)best_index_ (#3410)px.colors.sequential.Blues that introduces pandas dependency (#3422)_is_reverse_scale (#3424)fastai job on Python 3.6 (#3412)click==8.1.0 that removed a deprecated feature (#3413)click==8.1.0 that removed a deprecated feature" (#3430)step behavior to new distributions (#3276)show_progress_bar (#3287)copy_study: it creates a copy regardless of its state (#3295)Raises doc section (#3315)QMCSampler in tutorial (#3320)plot_param_importances (#3332, thanks @ll7!)optuna study optimize in FAQ (#3364)best_trial (#3396, thanks @divyanshugit!)Study.best_trials in multi-objective optimization tutorial (#3443)haiku's CI (https://github.com/optuna/optuna-examples/pull/83)black 22.1.0 & run checks daily (https://github.com/optuna/optuna-examples/pull/84)hiplot example (https://github.com/optuna/optuna-examples/pull/86)CatBoostPruningCallback (https://github.com/optuna/optuna-examples/pull/92)fastai job on Python 3.6 (https://github.com/optuna/optuna-examples/pull/93)str in workflow files (https://github.com/optuna/optuna-examples/pull/95)SimulatedAnnealingSampler to support FloatDistribution (https://github.com/optuna/optuna-examples/pull/97)matplotlib_tests/test_param_importances (#3180, thanks @belldandyxtq!)plot_optimization_history methods consistent (#3234)RedisStorage (#3258, thanks @masap!)catalyst integration test (#3308)MLflowCallback tests (#3378)IntDistribution (#3181, thanks @nyanhi!)UniformDistribution, LogUniformDistribution and DiscreteUniformDistribution code paths (#3275)set_trial_state() and set_trial_values() into one function (#3323, thanks @masap!){Float, Int}Distributions (#3337, thanks @nyanhi!)get_trial_xxx abstract functions to base (#3338, thanks @belldandyxtq!)states (#3359, thanks @BasLaa!)RedisStorage (#3394, thanks @masap!)tests/integration_tests (#3408)filter_nonfinite (#3436)RetryFailedTrialCallback to optuna.storages.* (#3441)fail_stale_trials in each storage implementation (#3442, thanks @knshnb!)bayesmark (#3354)setuptools (#3427)v3.0.0b0.dev (#3289)question-and-help-support (#3305)v3.0.0a2 (#3314)v3.0.0-b0 (#3458)This release was made possible by the authors and the people who participated in the reviews and discussions.
@BasLaa, @CorentinNeovision, @HideakiImamura, @Hiroyuki-01, @andriyor, @belldandyxtq, @belltailjp, @contramundum53, @divyanshugit, @harupy, @higucheese, @himkt, @hppRC, @hvy, @kei-mo, @keisuke-umezawa, @keisukefukuda, @knshnb, @ll7, @masap, @not522, @nyanhi, @nzw0301, @shu65, @sile, @tohmae, @toshihikoyanase, @tsukudamayo, @xadrianzetx
This is the release note of v3.0.0-a2.
This is the release note of v3.0.0-a2.
Study.optimize Warning Configuration FixThis is a small release that fixes a bug that the same warning message was emitted more than once when calling Study.optimize.
fail_state_trials show warning when heartbeat is enabled (#3303)This release was made possible by the authors and the people who participated in the reviews and discussions.
@HideakiImamura, @himkt
Included are several new features, improved optimization algorithms, removals of deprecated interfaces and many quality of life improvements.
This is the release note of v3.0.0-a1.
Second alpha pre-release in preparation for the upcoming major version update v3.
Included are several new features, improved optimization algorithms, removals of deprecated interfaces and many quality of life improvements.
To read about the entire v3 roadmap, please refer to the Wiki.
While this is a pre-release, we encourage users to keep using the latest releases of Optuna, including this one, for a smoother transition to the coming major release. Early feedback is welcome!
Now, you can utilize a new sampling algorithm based on the Quasi-Monte Carlo method, optuna.samplers.QMCSampler. This is oftentimes a good alternative to the existing optuna.samplers.RandomSampler. The generated (sampled) sequences have lower discrepancies compared to the standard random sequences, which are sampled uniformly. The following figures show the performance comparison to other existing samplers. Note that this algorithm is only supported for python >= 3.7.
See #2423 for more details.
| Parkinson in HPOBench | Slice in HPOBench |
|---|---|
The Pareto front plot now supports visualization of constrained optimization. In Optuna, NSGAIISampler and BoTorchSampler allow constrained optimization by taking a function constraints_func as argument, then examine whether trials are feasible or not. The optuna.visualization.plot_pareto_front receives a similar function and uses this function to plot the trials in different colors depending on whether they violate the constraints or not.
See #3128 for more details.
def objective(trial):
# Binh and Korn function with constraints.
x = trial.suggest_float("x", -15, 30)
y = trial.suggest_float("y", -15, 30)
# Store the constraints as user attributes so that they can be restored after optimization.
c0 = (x - 5) ** 2 + y ** 2 - 25
c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7
trial.set_user_attr("constraints", (c0, c1))
v0 = 4 * x ** 2 + 4 * y ** 2
v1 = (x - 5) ** 2 + (y - 5) ** 2
return v0, v1
def constraints(trial):
return trial.user_attrs["constraints"]
if __name__ == "__main__":
sampler = optuna.samplers.NSGAIISampler(
constraints_func=constraints,
)
study = optuna.create_study(
directions=["minimize", "minimize"],
sampler=sampler,
)
study.optimize(objective, n_trials=1000)
optuna.visualization.plot_pareto_front(study, constraints_func=constraints).show()
We are actively working on cleaning up distributions for integer and floating-point. In Optuna v3, these distribution are unified to optuna.distributions.IntDistribution and optuna.distributions.FloatDistribution. v3.0.0-a1 contains several changes for this project and you will temporarily see UserWarning when you call Trial.suggest_int and Trial.suggest_float. We apologize for the inconvenience and the warning will be removed from the next release.
See #2941 for more information.
We make AllenNLP integration and FrozenTrial.create_trial stable.
See #3196 and #3228 for more information
type_checking.py (#3235)constraints_func in plot_pareto_front (#3128, thanks @semiexp!)skip_if_finished flag to Study.tell (#3150, thanks @xadrianzetx!)user_attrs argument to Study.enqueue_trial (#3185, thanks @knshnb!)RetryFailedTrialCallback (#3269, thanks @knshnb!)DiscreteUniformDistribution.q (#3283)create_trial (#3196)directions, user_attrs and system_attrs of study summaries (#3108)FloatDistribution across codebase (#3111, thanks @xadrianzetx!)json.loads to decode pruner configuration loaded from environment variables (#3114)timeout (#3115, thanks @xadrianzetx!)IntDistribution across codebase (#3126, thanks @nyanhi!)RedisStorage in CachedStorage (#3204, thanks @masap!)functools.wraps in track_in_mlflow decorator (#3216)RedisStorage fast when running multiple trials (#3262, thanks @masap!)Study.ask() (#3274, thanks @masap!)TPESampler when group=True (#3187, thanks @xuzijian629!)matplotlib.contour_plot (#3213, thanks @xadrianzetx!)matplotlib.contour_plot (#3218, thanks @xadrianzetx!)-inf and inf values in RDBStorage (#3238, thanks @xadrianzetx!)nan (#3286)torch related packages (#3156)pytorch-lightning>=1.5.0 (#3157)mlflow integration (#3170)nltk version (#3201)setuptools (#3207)setuptools (#3231)logging_callback only works in single process situation (#3143)FrozenTrial's docstring (#3161)README.md (#3167)Study.optimize from API page (#3171, thanks @xuzijian629!)study.enqueue_trial (#3172, thanks @knshnb!)distributions documentation (#3222, thanks @xadrianzetx!)Raises section of FloatDistribution docstring (#3248, thanks @xadrianzetx!){Float,Int}Distribution to docs (#3252)AllenNLPExecutor (#3253)DiscreteUniformDistribution.q (#3279)nltk version (https://github.com/optuna/optuna-examples/pull/75)setuptools (https://github.com/optuna/optuna-examples/pull/76)setuptools (https://github.com/optuna/optuna-examples/pull/79)visualization_tests/matplotlib_tests/test_slice.py (#3175, thanks @keisukefukuda!)matplotlib.plot_edf (#3178, thanks @makinzm!)plot_edf (#3188, thanks @makinzm!)matplotlib.contour_plot test (#3232, thanks @xadrianzetx!)plot_parallel_coordinate (#3266, thanks @MasahitoKumada!)FloatDistribution (#3166, thanks @xadrianzetx!)deprecated decorator of the feature of n_jobs (#3173, thanks @MasahitoKumada!)MaxTrialsCallback (#3261, thanks @knshnb!)_check_trial_id (#3264, thanks @masap!){Int,Float}Distribution (#3244)kurobako (#3155)This release was made possible by the authors and the people who participated in the reviews and discussions.
@BasLaa, @HideakiImamura, @MasahitoKumada, @TakuyaInoue-github, @akawashiro, @belldandyxtq, @g-votte, @himkt, @hvy, @keisuke-umezawa, @keisukefukuda, @knshnb, @kstoneriv3, @makinzm, @masap, @not522, @nyanhi, @nzw0301, @semiexp, @tohmae, @toshihikoyanase, @tupui, @xadrianzetx, @xuzijian629
Included are several new features, improved optimization algorithms, removals of deprecated interfaces and many quality of life improvements.
This is the release note of v3.0.0-a0.
First alpha pre-release in preparation for the upcoming major version update v3.
Included are several new features, improved optimization algorithms, removals of deprecated interfaces and many quality of life improvements.
To read about the entire v3 roadmap, please refer to the Wiki.
While this is a pre-release, we encourage users to keep using the latest releases of Optuna, including this one, for a smoother transition to the coming major release. Early feedback is welcome!
Optuna CLI speed and usability improvements. Previously, it took several seconds to launch a CLI command, #3000 significantly speeds up the commands by halving the module load time.
The usability of the ask-and-tell interface is also improved. The ask command allows users to define search space with short and simple JSON strings after #2905. The tell command supports --skip-if-finished which ignores duplicated reports of values and statuses instead of raising errors. It for instance improves robustness against pod retries on cluster environments. See #2905.
Before:
$ optuna ask --storage sqlite:///mystorage.db --study-name mystudy \
--search-space '{"x": {"name": "UniformDistribution", "attributes": {"low": 0.0, "high": 1.0}}}'
After:
$ optuna ask --storage sqlite:///mystorage.db --study-name mystudy \
--search-space '{"x": {"type": "float", "low": 0.0, "high": 1.0}}'
The optimization performance of NSGA-II has been greatly improved for real-valued problems. We introduce the crossover argument in NSGAIISampler. You can select several variants of the crossover option from uniform (default), blxalpha, sbx, vsbx, undx, and spx.
The following figure shows that the newly introduced crossover algorithms perform better than existing algorithms, that is, the uniform crossover algorithm and the Gaussian process based algorithm, in terms of biasness, convergence, and diversity. Note that the previous method, other implementations (in kurobako), and the default of the new method are based on uniform crossover.
See #2903 for more information.
The optimization history plot now supports visualization of multiple studies. It receives a list of studies. If the error_bar option is False, it outputs those histories in one figure. If the error_bar option is True, it calculates and shows the means and the standard deviations of those histories.
See #2807 for more details.
import optuna
def objective(trial):
return trial.suggest_float("x", 0, 1) ** 2
n_studies = 5
studies = [optuna.create_study(study_name=f"{i}th-study") for i in range(n_studies)]
for study in studies:
study.optimize(objective, n_trials=20)
# This generates the first figure.
fig = optuna.visualization.plot_optimization_history(studies)
fig.write_image("./multiple.png")
# This generates the second figure.
fig = optuna.visualization.plot_optimization_history(studies, error_bar=True)
fig.write_image("./error_bar.png")
The AllenNLP integration supports pruning in distributed environments. This change enables users to use the optuna_pruner callback option along with the distributed option as can be seen in the following training configuration. See #2977.
...
trainer: {
optimizer: 'adam',
cuda_device: -1,
callbacks: [
{
type: 'optuna_pruner',
}
],
},
distributed: {
cuda_devices: [-1, -1],
},
There are several implementations of BaseDistribution in Optuna, such as UniformDistribution, DiscreteUniformDistribution, IntUniformDistribution, CategoricalDistribution, This release includes part of ongoing work in reducing the number of these distribution classes to just FloatDistribution, IntDistribution, and CategoricalDistribution, aligning the classes to the trial suggest interface (suggest_float, suggest_int, and suggest_categorical). Please note that users are not recommended to use these distributions yet, because samplers haven’t been updated to support those. See #3063 for more details.
Some deprecated features including the optuna.structs module, LightGBMTuner.best_booster, and the optuna dashboard command are removed in #3057 and #3058. If you use such features please migrate to the new ones.
| Removed APIs | Corresponding active APIs |
|---|---|
optuna.structs.StudyDirection |
optuna.study.StudyDirection |
optuna.structs.StudySummary |
optuna.study.StudySummary |
optuna.structs.FrozenTrial |
optuna.trial.FrozenTrial |
optuna.structs.TrialState |
optuna.trial.TrialState |
optuna.structs.TrialPruned |
optuna.exceptions.TrialPruned |
optuna.integration.lightgbm.LightGBMTuner.best_booster |
optuna.integration.lightgbm.LightGBMTuner.get_best_booster |
optuna dashboard |
optuna-dashboard |
suggest APIs for floating-point parameters (#2990, thanks @xadrianzetx!)optuna dashboard (#3058)plot_optimization_history (#2807)MLflowCallback interface (#2912, thanks @xadrianzetx!)trial.user_attrs logging optional in MLflowCallback (#3043, thanks @xadrianzetx!)IntDistribution & FloatDistribution (#3063, thanks @nyanhi!)trial.user_attrs to pareto_front hover text (#3082, thanks @kasparthommen!)optuna tell with --skip-if-finished (#3131)BoTorchSampler (#2928)import optuna (#3000)_contains of IntLogUniformDistribution (#3005)matplotlib.plot_param_importances (#3012, thanks @xadrianzetx!)verbose_eval NoneN for LightGBMTuner/LightGBMTunerCV` to avoid conflict (#3014, thanks @chezou!)plot_contour (#3017)FixedTrial and FrozenTrial allowing not-contained parameters during suggest_* (#3018)optuna ask CLI receives --sampler-kwargs without --sampler (#3029)_get_removed_version_from_deprecated_version function (#3065, thanks @nuka137!)plotly.plot_param_importances (#3073, thanks @xadrianzetx!)plot_contour using SciPy's spsolve (#3092)sample_relative and fix type of return values of SkoptSampler and PyCmaSampler (#2897)GridSampler with RetryFailedTrialCallback or enqueue_trial (#2946)trial.values in MLflow integration (#2991)ValueError for invalid q in DiscreteUniformDistribution (#3001)trial.report during sanity check (#3002)matplotlib.plot_contour bug (#3046, thanks @IEP!)single distributions in fANOVA evaluator (#3085, thanks @xadrianzetx!)tensorflow and tensorflow-estimator versions to <2.7.0 (#3059)keras version to <2.7.0 (#3078)tensorflow (#3084)trial.report (#2980)pytorch-lightning (#2984)OptunaSearchCV about direction (#3007)n_trials in the docs (#3016, thanks @Rohan138!)optuna tell (#3052)logging.set_verbosity (#3061, thanks @drumehiron!)002_configurations.py in the Trial API page (#3067, thanks @makkimaki!)003_efficient_optimization_algorithms.py in the Trial API page (#3068, thanks @makkimaki!)set_user_attrs in Study to the user_attrs entry in Tutorial (#3069, thanks @MasahitoKumada!)suggest_float docstring (#3091, thanks @xadrianzetx!)ValueError and TypeErorr to Raises section of Trial.report (#3124, thanks @MasahitoKumada!)RetryFailedTrialCallback in pytorch_checkpoint example (https://github.com/optuna/optuna-examples/pull/59, thanks @xadrianzetx!)suggest_uniform with suggest_float (https://github.com/optuna/optuna-examples/pull/63)lightgbm (https://github.com/optuna/optuna-examples/pull/64)tensorflow and tensorflow-estimator versions to <2.7.0 (https://github.com/optuna/optuna-examples/pull/66)pytorch-lightning (https://github.com/optuna/optuna-examples/pull/67)README.md (https://github.com/optuna/optuna-examples/pull/68, thanks @solegalli!)pytorch-lightning (#2983)np.asarray in lightgbm test (#2997)suggest APIs across codebase (#3027, thanks @xadrianzetx!)return_cvbooster of LightGBMTuner consistent to the original value (#3070, thanks @abatomunkuev!)parametrize_sampler (#3080)tests/integration_tests/lightgbm_tuner_tests/test_optimize.py (#3086, thanks @nyanhi!)matplotlib.plot_slice (#3121)BaseStudy (#2986, thanks @twsl!)optuna.load_study in optuna ask CLI to omit direction/directions option (#2989)Trial warning message (#3008, thanks @xadrianzetx!)suggest APIs (#3054, thanks @xadrianzetx!)remove_version to the missing @deprecated argument (#3064, thanks @nuka137!)optuna.logging.get_verbosity (#3066, thanks @MasahitoKumada!){Float|Int}Distribution in NSGA-II crossover operators (#3139, thanks @xadrianzetx!)botorch to CI jobs on mac (#2988)mac-tests CI at a scheduled time (#3028)mypy version to 0.910 (#3123)README.md (#2999)tox.ini (#3025)This release was made possible by the authors and the people who participated in the reviews and discussions.
@Crissman, @HideakiImamura, @IEP, @MasahitoKumada, @Rohan138, @TakuyaInoue-github, @abatomunkuev, @avats-dev, @belldandyxtq, @chezou, @drumehiron, @g-votte, @himkt, @hvy, @kasparthommen, @keisuke-umezawa, @makkimaki, @masaaldosey, @masap, @not522, @nuka137, @nyanhi, @nzw0301, @shu65, @sidshrivastav, @sile, @solegalli, @tohmae, @toshihikoyanase, @twsl, @xadrianzetx, @yoshinobc, @ytsmiling
This is the release note of v2.10.1.
This is the release note of v2.10.1.
This is a patch release to resolve the issues on the document build. No feature updates are included.
youtube.com with youtube-nocookie.com (#3633)This release was made possible by the authors and the people who participated in the reviews and discussions.
@contramundum53, @toshihikoyanase
Replace deprecated argument lr with learning_rate in tf.keras
This is the release note of v2.10.0.
New subcommands optuna trials, optuna best-trial and optuna best-trials have been introduced to Optuna’s CLI for listing trials in studies with RDB storages. It allows direct interaction with trial data from the command line in various formats including human readable tables, JSON or YAML. See the following examples:
Show all trials in a study.
$ optuna trials --storage sqlite:///example.db --study-name example
+--------+---------------------+---------------------+---------------------+----------------+---------------------+----------+
| number | value | datetime_start | datetime_complete | duration | params | state |
+--------+---------------------+---------------------+---------------------+----------------+---------------------+----------+
| 0 | 0.6098421143538713 | 2021-10-01 14:36:46 | 2021-10-01 14:36:46 | 0:00:00.026059 | {'x': 'A', 'y': 6} | COMPLETE |
| 1 | 0.6584108953598753 | 2021-10-01 14:36:46 | 2021-10-01 14:36:46 | 0:00:00.023447 | {'x': 'A', 'y': 10} | COMPLETE |
| 2 | 0.612883262548314 | 2021-10-01 14:36:46 | 2021-10-01 14:36:46 | 0:00:00.021577 | {'x': 'C', 'y': 3} | COMPLETE |
| 3 | 0.09326753798819143 | 2021-10-01 14:36:46 | 2021-10-01 14:36:46 | 0:00:00.024183 | {'x': 'A', 'y': 0} | COMPLETE |
| 4 | 0.7316749689191168 | 2021-10-01 14:36:46 | 2021-10-01 14:36:46 | 0:00:00.021994 | {'x': 'C', 'y': 4} | COMPLETE |
+--------+---------------------+---------------------+---------------------+----------------+---------------------+----------+
Show the best trial as YAML.
$ optuna best-trial --storage sqlite:///example.db --study-name example --format yaml
datetime_complete: '2021-10-01 14:36:46'
datetime_start: '2021-10-01 14:36:46'
duration: '0:00:00.024183'
number: 3
params:
x: A
y: 0
state: COMPLETE
value: 0.09326753798819143
Show the best trials of multi-objective optimization and train a neural network with one of the best parameters.
$ STORAGE=sqlite:///example.db
$ STUDY_NAME=example-mo
$ optuna best-trials --storage $STORAGE --study-name $STUDY_NAME
+--------+-------------------------------------------+---------------------+---------------------+----------------+--------------------------------------------------+----------+
| number | values | datetime_start | datetime_complete | duration | params | state |
+--------+-------------------------------------------+---------------------+---------------------+----------------+--------------------------------------------------+----------+
| 0 | [0.23884292794146034, 0.6905832476748404] | 2021-10-01 15:02:32 | 2021-10-01 15:02:32 | 0:00:00.035815 | {'lr': 0.05318673615579818, 'optimizer': 'adam'} | COMPLETE |
| 2 | [0.3157886300888031, 0.05110976427394465] | 2021-10-01 15:02:32 | 2021-10-01 15:02:32 | 0:00:00.030019 | {'lr': 0.08044012012204389, 'optimizer': 'sgd'} | COMPLETE |
+--------+-------------------------------------------+---------------------+---------------------+----------------+--------------------------------------------------+----------+
$ optuna best-trials --storage $STORAGE --study-name $STUDY_NAME --format json > result.json
$ OPTIMIZER=`jq '.[0].params.optimizer' result.json`
$ LR=`jq '.[0].params.lr' result.json`
$ python train.py $OPTIMIZER $LR
See #2847 for more details.
Weights & Biases and MLflow integration modules support tracking multi-objective optimization. Now, they accept arbitrary numbers of objective values with metric names.
from optuna.integration import WeightsAndBiasesCallback
wandbc = WeightsAndBiasesCallback(metric_name=["mse", "mae"])
...
study = optuna.create_study(directions=["minimize", "minimize"])
study.optimize(objective, n_trials=100, callbacks=[wandbc])
from optuna.integration import MLflowCallback
mlflc = MLflowCallback(metric_name=["accuracy", "latency"])
...
study = optuna.create_study(directions=["minimize", "minimize"])
study.optimize(objective, n_trials=100, callbacks=[mlflc])
See #2835 and #2863 for more details.
optuna ask has been simplified. The first layer of nesting with the key “trial” is removed. Parsing can be simplified from jq ‘.trial.params’ to jq ‘.params’.WeightsAndBiasesCallback (#2835, thanks @xadrianzetx!)MLflowCallback (#2863, thanks @xadrianzetx!)optuna.visualization.matplotlib.plot_contour (#2810, thanks @xadrianzetx!)plot_parallel_coordinate (#2821, thanks @TakuyaInoue-github!)MLflowCallback (#2855, thanks @xadrianzetx!)plot_parallel_coordinate (#2856)sklearn.py (#2966, thanks @Garve!)datetime_complete in _CachedStorage (#2846)untransform of _SearchSpaceTransform with distribution.single() == True (#2947, thanks @yoshinobc!)keras 2.6.0 (#2851)tensorflow and keras version constraints (#2852)allennlp==2.7.0 (#2894)bounds' shape in the document (#2830)FrozenTrial (#2833)MLflowCallback (#2883)create_trial document (#2888)_CachedStorage (#2917):obj: for True, False, and None instead of inline code (#2922)constraints_func (#2930)keras==2.6.0 (https://github.com/optuna/optuna-examples/pull/44)lr with learning_rate in tf.keras (https://github.com/optuna/optuna-examples/pull/51)allennlp==2.7.0 (https://github.com/optuna/optuna-examples/pull/52)MLflowCallback in MLflow example (https://github.com/optuna/optuna-examples/pull/58, thanks @xadrianzetx!)1 (#2865, thanks @Yu212!)show_progress_bar of optimize (#2900, thanks @xadrianzetx!)namedtuple type name (#2961, thanks @sobolevn!)WeightsAndBiasesCallback (#2884, thanks @xadrianzetx!)alembic 1.7.0 type hint error (#2887)Trial._after_func method (#2899)namedtuple type name (#2961, thanks @sobolevn!)tensorflow and keras version constraints (#2852)test_lightgbm.py on macOS (#2896)This release was made possible by the authors and the people who participated in the reviews and discussions.
@01-vyom, @Crissman, @DeviousLab, @Garve, @HideakiImamura, @TakuyaInoue-github, @Yu212, @c-bata, @cowwoc, @himkt, @hvy, @jrbourbeau, @keisuke-umezawa, @not522, @nzw0301, @sobolevn, @toshihikoyanase, @xadrianzetx, @yoshinobc
This is the release note of v2.9.1.
This is the release note of v2.9.1.
The storage URI and the study name are no longer logged by optuna ask and optuna tell. The former could contain sensitive information.
ask and tell CLI subcommands (#2838)This release was made possible by the authors and the people who participated in the reviews and discussions.
@himkt, @hvy, @not522
The previously experimental multi-objective TPE Sampler MOTPESampler has also been deprecated and its capabilities are now absorbed by the standard TP…
This is the release note of v2.9.0.
Help us create the next version of Optuna! Please take a few minutes to fill in this survey, and let us know how you use Optuna now and what improvements you'd like. https://forms.gle/TtJuuaqFqtjmbCP67
The built-in CLI which you can use to upgrade storages or check the installed version with optuna --version, now provides experimental subcommands for the Ask-and-Tell interface. It is now possible to optimize using Optuna entirely from the CLI, without writing a single line of Python.
optuna askAsk for parameters using optuna ask, specifying the search space, storage, study name, sampler and optimization direction. The parameters and the associated trial number can be output as either JSON or YAML.
The following is an example outputting and piping the results to a YAML file.
$ optuna ask --storage sqlite:///mystorage.db \
--study-name mystudy \
--sampler TPESampler \
--sampler-kwargs '{"multivariate": true}' \
--search-space '{"x": {"name": "UniformDistribution", "attributes": {"low": 0.0, "high": 1.0}}, "y": {"name": "CategoricalDistribution", "attributes": {"choices": ["foo", "bar"]}}}' \
--direction minimize \
--out yaml \
> out.yaml
[I 2021-07-30 15:56:50,774] A new study created in RDB with name: mystudy
[I 2021-07-30 15:56:50,808] Asked trial 0 with parameters {'x': 0.21492964898919975, 'y': 'foo'} in study 'mystudy' and storage 'sqlite:///mystorage.db'.
$ cat out.yaml
trial:
number: 0
params:
x: 0.21492964898919975
y: foo
Specify multiple whitespace separated directions for multi-objective optimization.
optuna tellAfter computing the objective value based on the output of ask, you can report the result back using optuna tell and it will be stored in the study.
$ optuna tell --storage sqlite:///mystorage.db \
--study-name mystudy \
--trial-number 0 \
--values 1.0
[I 2021-07-30 16:01:13,039] Told trial 0 with values [1.0] and state TrialState.COMPLETE in study 'mystudy' and storage 'sqlite:///mystorage.db'.
Specify multiple whitespace separated values for multi-objective optimization.
See https://github.com/optuna/optuna/pull/2817 for details.
WeightsAndBiasesCallback is a new study optimization callback that allows logging with Weights & Biases. This allows utilizing Weight & Biases’ rich visualization features to analyze studies to complement Optuna’s visualization.
import optuna
from optuna.integration.wandb import WeightsAndBiasesCallback
def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 2) ** 2
wandb_kwargs = {"project": "my-project"}
wandbc = WeightsAndBiasesCallback(wandb_kwargs=wandb_kwargs)
study = optuna.create_study(study_name="mystudy")
study.optimize(objective, n_trials=10, callbacks=[wandbc])
See https://github.com/optuna/optuna/pull/2781 for details.
The Tree-structured Parzen Estimator (TPE) sampler has always been the default sampler in Optuna. Both it’s API and internal code has over time grown to accomodate for various needs such as independent and join parameter sampling (the multivariate parameter) , and multi-objective optimization (the MOTPESampler sampler). In this release, the TPE sampler has been refactored and its code greatly reduced. The previously experimental multi-objective TPE Sampler MOTPESampler has also been deprecated and its capabilities are now absorbed by the standard TPESampler.
This change may break code that depends on fixed seeds with this sampler. The optimization algorithms otherwise have not been changed.
Following demonstrates how you can now use the TPESampler for multi-objective optimization.
import optuna
def objective(trial):
x = trial.suggest_float("x", 0, 5)
y = trial.suggest_float("y", 0, 3)
v0 = 4 * x ** 2 + 4 * y ** 2
v1 = (x - 5) ** 2 + (y - 5) ** 2
return v0, v1
sampler = optuna.samplers.TPESampler() # `MOTPESampler` used to be required for multi-objective optimization.
study = optuna.create_study(
directions=["minimize", "minimize"],
sampler=sampler,
)
study.optimize(objective, n_trials=100)
Note that omitting the sampler argument or specifying None currently defaults to the NSGAIISampler for multi-objective studies instead of the TPESampler.
See https://github.com/optuna/optuna/pull/2618 for details.
MOTPESampler and TPESampler (#2688)GridSampler (#2783)warn_independent_sampling in TPESampler (#2786)constraint_fn to non-COMPLETE trials in NSGAII-sampler (#2791)TPESampler (#2816)AllenNLPExecutor reproducibility (#2717, thanks @MagiaSN!)repr and eval to restore pruner parameters in AllenNLP integration (#2731)Nan cast bug in TPESampler (#2739)infer_relative_search_space of TPE with the single point distributions (#2749)copy_study to the docs (#2737)RetryFailedTrialCallback.retried_trial_number (#2789)ID (#2798, thanks @belldandyxtq!)RDBStorage RuntimeError description (#2802, thanks @belldandyxtq!)MOTPEMultiObjectiveSampler (#2666)visualization.matplotlib.plot_intermediate_values (#2754, thanks @asquare100!)tests/visualization_tests/matplotlib/test_intermediate_plot.py (#2803)study directory (#2721)MOTPESampler (#2770)Checks (#2760)README.md (#2801)This release was made possible by the authors and the people who participated in the reviews and discussions.
@ytsmiling, @harupy, @asquare100, @hvy, @c-bata, @nzw0301, @lucafurrer, @belldandyxtq, @not522, @TakuyaInoue-github, @01-vyom, @himkt, @Crissman, @toshihikoyanase, @sile, @vanpelt, @HideakiImamura, @MagiaSN, @keisuke-umezawa, @Turakar, @xadrianzetx, @ytsmiling, @harupy, @asquare100, @hvy, @c-bata, @nzw0301, @lucafurrer, @belldandyxtq, @not522, @TakuyaInoue-github, @01-vyom, @himkt, @Crissman, @toshihikoyanase, @sile, @vanpelt, @HideakiImamura, @MagiaSN, @keisuke-umezawa, @Turakar, @xadrianzetx
Using the "log" key is deprecated in pytorch_lightning (#2611, thanks @sushi30!)
This is the release note of v2.8.0.
The number of Optuna examples has grown as the number of integrations have increased, and we’ve moved them to their own repository: optuna/optuna-examples.
In distributed environments, the TPE sampler may sample many points in a small neighborhood, because it does not have knowledge that other trials running in parallel are sampling nearby. To avoid this issue, we’ve implemented the Constant Liar (CL) heuristic to return a poor value for trials which have started but are not yet complete, to reduce search effort.
study = optuna.create_study(sampler=optuna.samplers.TPESampler(constant_liar=True))
The following history plots demonstrate how optimization can be improved using this feature. Ten parallel workers are simultaneously trying to optimize the same function which takes about one second to compute. The first plot has constant_liar=False, and the second with constant_liar=True, uses the Constant Liar feature. We can see that with Constant Liar, the sampler does a better job of assigning different parameter configurations to different trials and converging faster.
See #2664 for details.
The TPE sampler with multivariate=True now supports tree-structured search spaces. Previously, if the user split the search space with an if-else statement, as shown below, the TPE sampler with multivariate=True would fall back to random sampling. Now, if you set multivariate=True and group=True, the TPE sampler algorithm will be applied to each partitioned search space to perform efficient sampling.
See #2526 for more details.
def objective(trial):
classifier_name = trial.suggest_categorical("classifier", ["SVC", "RandomForest"])
if classifier_name == "SVC":
# If `multivariate=True` and `group=True`, the following 2 parameters are sampled jointly by TPE.
svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
svc_kernel = trial.suggest_categorical("kernel", ["linear", "rbf", "sigmoid"])
classifier_obj = sklearn.svm.SVC(C=svc_c, kernel=svc_kernel)
else:
# If `multivariate=True` and `group=True`, the following 3 parameters are sampled jointly by TPE.
rf_n_estimators = trial.suggest_int("rf_n_estimators", 1, 20)
rf_criterion = trial.suggest_categorical("rf_criterion", ["gini", "entropy"])
rf_max_depth = trial.suggest_int("rf_max_depth", 2, 32, log=True)
classifier_obj = sklearn.ensemble.RandomForestClassifier(n_estimators=rf_n_estimators, criterion=rf_criterion, max_depth=rf_max_depth)
...
sampler = optuna.samplers.TPESampler(multivariate=True, group=True)
Studies can now be copied across storages. The trial history as well as Study.user_attrs and Study.system_attrs are preserved.
For instance, this allows dumping a study in an MySQL RDBStorage into an SQLite file. Serializing the study this way, it can be shared with other users who are unable to access the original storage.
study = optuna.create_study(
study_name=”my-study”, storage=”mysql+pymysql://root@localhost/optuna"
)
study.optimize(..., n_trials=100)
# Creates a copy of the study “my-study” in an MySQL `RDBStorage` to a local file named `optuna.db`.
optuna.copy_study(
from_study_name="my-study",
from_storage="mysql+pymysql://root@localhost/optuna",
to_storage="sqlite:///optuna.db",
)
study = optuna.load_study(study_name=”my-study”, storage=”sqlite:///optuna.db”)
assert len(study.trials) >= 100
See #2607 for details.
optuna.storages.RetryFailedTrialCallback AddedUsed as a callback in RDBStorage, this allows a previously pre-empted or otherwise aborted trials that are detected by a failed heartbeat to be re-run.
storage = optuna.storages.RDBStorage(
url="sqlite:///:memory:",
heartbeat_interval=60,
grace_period=120,
failed_trial_callback=optuna.storages.RetryFailedTrialCallback(max_retry=3),
)
study = optuna.create_study(storage=storage)
See #2694 for details.
optuna.study.MaxTrialsCallback AddedUsed as a callback in study.optimize, this allows setting of a maximum number of trials of a particular state, such as setting the maximum number of failed trials, before stopping the optimization.
study.optimize(
objective,
callbacks=[optuna.study.MaxTrialsCallback(10, states=(optuna.trial.TrialState.COMPLETE,))],
)
See #2636 for details.
None as study_name when there is only a single study in load_study (#2608)GridSampler allowing not-contained parameters during suggest_* (#2663)LightGBMTuner and LightGBMTunerCV reproducible (#2431, thanks @tetsuoh0103!)visualization.matplotlib.plot_pareto_front (#2450, thanks @tohmae!)__str__ for samplers (#2539)n_min_trials argument for PercentilePruner and MedianPruner (#2556)None as study_name when there is only a single study in load_study (#2608)MaxTrialsCallback class to enable stopping after fixed number of trials (#2612)PatientPruner (#2636)optuna create-study) (#2640)TPESampler (#2664)visualization.matplotlib.plot_slice (#2709, thanks @Muktan!)PyTorchLightningPruningCallback to warn when an evaluation metric does not exist (#2157, thanks @bigbird555!)visualization.plot_contour (#2569)param_importances (#2576)visualization.matplotlib.plot_contour (#2593)optuna.visualization.matplotlib.plot_edf (#2603)optuna.visualization.matplotlib.plot_intermediate_values (#2606)MOTPEMultiObjectiveSampler a thin wrapper for MOTPESampler (#2615)matplotlib for consistency with plotly results (#2711, thanks @01-vyom!)target being specified (#2589)_contains (#2652)dump_best_config (#2681)AllenNLPExecutor multiple t… (Backport of #2717) (#2728)sklearn constraint (#2634)click==7.1.2 to GitHub workflows to solve AllenNLP import error (#2665)tensorflow 2.5.0 (#2674)example from setup.py (#2676)optuna.logging.disable_propagation (#2477, thanks @jeromepatel!)README.md (#2586)CmaEsSampler (#2591, thanks @turian!)ray-joblib.py to snakecase with underscores (#2594)If with if in a sentence (#2602)CmaEsSampler instead of TPESampler in the batch optimization example (#2610)BoTorchSampler page (#2631)WAITING details in docstring (#2683, thanks @jeromepatel!)optuna-examples (#2684)README.md (#2692)CONTRIBUTING.md (#2719)optuna/optuna (https://github.com/optuna/optuna-examples/pull/2)MaxTrialsCallback class to enable stopping after fixed number of trials (https://github.com/optuna/optuna-examples/pull/9)README.md (https://github.com/optuna/optuna-examples/pull/10)tensorflow 2.5.0 (https://github.com/optuna/optuna-examples/pull/13)tensorflow 2.5 (https://github.com/optuna/optuna-examples/pull/15)multi_objective in CI (https://github.com/optuna/optuna-examples/pull/16)CONTRIBUTING.md file ((https://github.com/optuna/optuna-examples/pull/21)n_jobs for study.optimize in examples/ (#2588, thanks @jeromepatel!)pytorch_lightning (#2611, thanks @sushi30!)optuna-examples (https://github.com/optuna/optuna-examples/pull/11 follow up (#2689)test_plot_pareto_front_unsupported_dimensions (#2578)matplotliv.plot_pareto_front for consistency (#2583)deterministic parameter to make LightGBM training reproducible (#2623)force_col_wise parameter of LightGBM in test cases of LightGBMTuner and LightGBMTunerCV (#2630, thanks @tetsuoh0103!)CudaCallback from the fastai test (#2641)optuna/visualization/matplotlib/edf.py (#2642)test_median.py (#2644)pruners_test (#2691, thanks @tsumli!)examples (#2554)matplotlib.contour (#2571)optuna.visualization.matplotlib.plot_pareto_front when axis_order is specified (#2577)_get_distribution from visualization/matplotlib/_param_importances.py (#2604)MOTPESampler from TPESampler (#2616)add_distributions in _SearchSpaceGroup (#2651)optuna.integrations (#2700)asv (#2673)master version to 2.8.0dev (#2562)formats.sh (#2637)command to check the existence of the libraries to avoid partially matching (#2653)CONTRIBUTING.md with optuna-examples (#2669)This release was made possible by the authors and the people who participated in the reviews and discussions.
@toshihikoyanase, @himkt, @Scitator, @tohmae, @crcrpar, @c-bata, @01-vyom, @sushi30, @tsumli, @not522, @tetsuoh0103, @jeromepatel, @bigbird555, @hvy, @g-votte, @nzw0301, @turian, @Crissman, @sile, @agarwalrounak, @Muktan, @Turakar, @HideakiImamura, @keisuke-umezawa, @0x41head, @toshihikoyanase, @himkt, @Scitator, @tohmae, @crcrpar, @c-bata, @01-vyom, @sushi30, @tsumli, @not522, @tetsuoh0103, @jeromepatel, @bigbird555, @hvy, @g-votte, @nzw0301, @turian, @nmasahiro, @Crissman, @sile, @agarwalrounak, @Muktan, @Turakar, @HideakiImamura, @keisuke-umezawa, @0x41head
Nothing published for this version
The previous optuna dashboard command is now deprecated.
This is the release note for v2.7.0.
optuna-dashboard RepositoryA new dashboard optuna-dashboard is being developed in a separate repository under the Optuna organization. Install it with pip install optuna-dashboard and run it with optuna-dashboard $STORAGE_URL.
The previous optuna dashboard command is now deprecated.
n_jobs Argument of Study.optimizeThe GIL has been an issue when using the n_jobs argument for multi-threaded optimization. We decided to deprecate this option in favor of the more stable process-level parallelization. Details available in the tutorial. Users who have been parallelizing at the thread level using the n_jobs argument are encouraged to refer to the tutorial for process-level parallelization.
If the objective function is not affected by the GIL, thread-level parallelism may be useful. You can achieve thread-level parallelization in the following way.
with ThreadPoolExecutor(max_workers=5) as executor:
for _ in range(5):
executor.submit(study.optimize, objective, 100)
Tutorial pages about the usage of the ask-and-tell interface (#2422) and best_trial (#2427) have been added, as well as an example that demonstrates parallel optimization using Ray (#2298) and an example to explain how to stop the optimization based on the number of completed trials instead of the total number of trials (#2449).
The code quality was improved in terms of bug fixes, third party library support, and platform support.
For instance, the bugs on warm starting CMA-ES and visualization.matplotlib.plot_optimization_history were resolved by #2501 and #2532, respectively.
Third party libraries such as PyTorch, fastai, and AllenNLP were updated. We have updated the corresponding integration modules and examples for the new versions. See #2442, #2550 and #2528 for details.
From this version, we are expanding the platform support. Previously, changes were tested in Linux containers. Now, we also test changes merged into the master branch in macOS containers as well (#2461).
n_jobs in Study.optimize (#2560)StudyDirection for create_study arguments (#2516)numpy.append (#2419, thanks @nyanhi!)after_trial for NSGAIISampler (#2436, thanks @jeromepatel!)visualization.matplotlib.plot_optimization_history for multi-objective (#2532)torch to 1.8.0 (#2442)install_requires (#2466)best_trial (#2427)max_trial_callback to optuna/examples (#2449, thanks @jeromepatel!)SuccessiveHalvingPruner page (#2489)load_if_exists and directions for consistency (#2491)n_jobs for OptunaSearchCV (#2545)Ray with joblib backend (#2298)examples/README.md (#2432, thanks @jeromepatel!)sh with bash in README of kubernetes examples (#2440)urllib patch for MNIST download (#2459, thanks @crcrpar!)Dockerfile of MLflow Kubernetes examples (#2472, thanks @0x41head!)pytorch_lightning_distributed.py to remove MNIST and PyTorch Lightning errors (#2514, thanks @0x41head!)OptunaPruningCallback in catalyst_simple.py (#2546, thanks @crcrpar!)MOTPESampler in parametrize_multi_objective_sampler (#2448)test_multi_objective.py (#2525)mypy errors produced by numpy==1.20.0 (#2300, thanks @0x41head!)_SearchSpaceTransform in RandomSampler (#2410, thanks @sfujiwara!)state of create_trial as COMPLETE (#2429)v2.2.0 (#2528)-f option to make clean command idempotent (#2439)master version to 2.7.0dev (#2444)CONTRIBUTING.md (#2463, thanks @crcrpar!)This release was made possible by authors, and everyone who participated in reviews and discussions.
@0x41head, @AmeerHajAli, @Crissman, @HideakiImamura, @c-bata, @crcrpar, @g-votte, @himkt, @hvy, @jeromepatel, @keisuke-umezawa, @not522, @nyanhi, @nzw0301, @parsiad, @sfujiwara, @sile, @toshihikoyanase, @y0z
This is the release note of v2.6.0.
This is the release note of v2.6.0.
Two new CMA-ES variants are available. Warm starting CMA-ES enables transferring prior knowledge on similar tasks. More specifically, CMA-ES can be initialized based on existing results of similar tasks. sep-CMA-ES is an algorithm which constrains the covariance matrix to be diagonal and is suitable for separable objective functions. See #2307 and #1951 for more details.
Example of Warm starting CMA-ES:
study = optuna.load_study(storage=”...”, study_name=”existing-study”)
study.sampler = CmaEsSampler(source_trials=study.trials)
study.optimize(objective, n_trials=100)
Example of sep-CMA-ES:
study = optuna.create_study(sampler=CmaEsSampler(use_separable_cma=True))
study.optimize(objective, n_trials=100)
Hyperparameter optimization for distributed neural-network training using PyTorch Distributed Data Parallel is supported. A new integration moduleTorchDistributedTrial, synchronizes the hyperparameters among all nodes. See #2303 for further details.
Example:
def objective(trial):
distributed_trial = optuna.integration.TorchDistributedTrial(trial)
lr = distributed_trial.suggest_float("lr", 1e-5, 1e-1, log=True)
…
RDBStorage ImprovementsThe RDBStorage now allows longer user and system attributes, as well as choices for categorical distributions (e.g. choices spanning thousands of bytes/characters) to be persisted. Corresponding column data types of the underlying SQL tables have been changed from VARCHAR to TEXT. If you want to upgrade from an older version of Optuna and keep using the same storage, please migrate your tables as follows. Please make sure to create any backups before the migration and note that databases that don’t support TEXT will not work with this release.
# Alter table columns from `VARCHAR` to `TEXT` to allow storing larger data.
optuna storage upgrade --storage <storage URL>
For more details, see #2395.
The heartbeat feature was introduced in v2.5.0 to automatically mark stale trials as failed. It is now possible to not only fail the trials but also execute user-specified callback functions to process the failed trials. See #2347 for more details.
Example:
def objective(trial):
… # Very time-consuming computation.
# Adding a failed trial to the trial queue.
def failed_trial_callback(study, trial):
study.add_trial(
optuna.create_trial(
state=optuna.trial.TrialState.WAITING,
params=trial.params,
distributions=trial.distributions,
user_attrs=trial.user_attrs,
system_attrs=trial.system_attrs,
)
)
storage = optuna.storages.RDBStorage(
url=...,
heartbeat_interval=60,
grace_period=120,
failed_trial_callback=failed_trial_callback,
)
study = optuna.create_study(storage=storage)
study.optimize(objective, n_trials=100)
The ask-and-tell interface allows specifying pre-defined search spaces through the new fixed_distributions argument. This option will keep the code short when the search space is known beforehand. It replaces calls to Trial.suggest_…. See #2271 for more details.
study = optuna.create_study()
# For example, the distributions are previously defined when using `create_trial`.
distributions = {
"optimizer": optuna.distributions.CategoricalDistribution(["adam", "sgd"]),
"lr": optuna.distributions.LogUniformDistribution(0.0001, 0.1),
}
trial = optuna.trial.create_trial(
params={"optimizer": "adam", "lr": 0.0001},
distributions=distributions,
value=0.5,
)
study.add_trial(trial)
# You can pass the distributions previously defined.
trial = study.ask(fixed_distributions=distributions)
# `optimizer` and `lr` are already suggested and accessible with `trial.params`.
print(trial.params)
RDBStorage data type updatesDatabases must be migrated for storages that were created with earlier versions of Optuna. Please refer to the highlights above.
For more details, see #2395.
datatime_start of enqueued trials.The datetime_start property of Trial, FrozenTrial and FixedTrial shows when a trial was started. This property may now be None. For trials enqueued with Study.enqueue_trial, the timestamp used to be set to the time of queue. Now, the timestamp is first set to None when enqueued, and later updated to the timestamp when popped from the queue to run. This has implications on StudySummary.datetime_start as well which may be None in case trials have been enqueued but not popped.
For more details, see #2236.
joblib internals removedjoblib was partially supported as a backend for parallel optimization via the n_jobs parameter to Study.optimize. This support has now been removed and internals have been replaced with concurrent.futures.
For more details, see #2269.
Optuna now officially supports AllenNLP v2. We also dropped the AllenNLP v0 support and the pruning support for AllenNLP v1. If you want to use AllenNLP v0 or v1 with Optuna, please install Optuna v2.5.0.
For more details, see #2412.
Study.ask method that allows define-and-run parameter suggestion (#2271)IntLogUniformDistribution for TensorBoard (#2362, thanks @nzw0301!)datetime_start (clean) (#2236, thanks @chenghuzi!)Study to suggest solutions (#2251)LightGBMTuner metrics for the case of higher is better (#2267, thanks @mavillan!)Storage (#2345)after_trial method in CmaEsSampler (#2359, thanks @jeromepatel!)low and high to float explicitly in distributions (#2360)after_trial for PyCmaSampler (#2365, thanks @jeromepatel!)after_trial for BoTorchSampler and SkoptSampler (#2372, thanks @jeromepatel!)after_trial for TPESampler (#2376, thanks @jeromepatel!)BoTorch >= 0.4.0 (#2386, thanks @nzw0301!)RDBStorage (#2395)after_trial for MOTPESampler (#2425, thanks @jeromepatel!)optuna.visualization.plot_contour of subplot case with categorical axes (#2297, thanks @nzw0301!)suggest_float (#2335, thanks @nzw0301!)GridSampler and HyperbandPruner (#2353)matplotlib.plot_parallel_coordinate with only one suggested parameter (#2354, thanks @nzw0301!)model_dir by _LightGBMBaseTuner (#2366, thanks @nyanhi!)low in _transform_from_uniform for TPE sampler (#2392, thanks @nzw0301!)optuna.visualization.plot_parallel_coordinate with categorical values (#2401, thanks @nzw0301!)mypy hotfix voiding latest NumPy 1.20.0 (#2292)jax from setup.py (#2308, thanks @nzw0301!)torch from PyPI for ReadTheDocs (#2361)botorch version (#2379)README.md (#2268)docs/source/tutorial for faster local documentation build (#2277)n_trials from example of GridSampler (#2280)sphinx.ext.imgconverter extension (#2323, thanks @KoyamaSohei!)high in the documentation of UniformDistribution and LogUniformDistribution (#2348)TFKerasPruningCallback (#2399, thanks @sfujiwara!)fig.show() in visualization code examples (#2403, thanks @harupy!)trainer.callback_metrics in the Pytorch Lightning example (#2294, thanks @TezRomacH!)DataModule (#2332, thanks @TezRomacH!)examples/kubernetes/mlflow/check_study.sh to match whole words (#2363, thanks @twolffpiggott!)failed_trial_callback (#2373)Dockerfile of Kubernetes simple example (#2375, thanks @0x41head!)GridSampler (#2285)parametrize_storage with StorageSupplier (#2404, thanks @nzw0301!)joblib with concurrent.futures for parallel optimization (#2269)LightGBMTuner (Follow-up #2267) (#2299)PyTorchLightningPruningCallback from Callback (#2326, thanks @TezRomacH!)suggest_float (#2344)contraints with constraints (#2378, thanks @nzw0301!)study.get_trials for states filtering (#2393, thanks @jeromepatel!)super().__init__ (#2402, thanks @nyanhi!)-f option from doctest pip installation (#2418)v2.6.0.dev (#2283)CONTRIBUTING.md (#2342)mypy error on Pytorch Lightning integration (#2349)botorch example (#2377, thanks @nzw0301!)-f option from documentation installation (#2407)This release was made possible by authors, and everyone who participated in reviews and discussions.
@0x41head, @Crissman, @HideakiImamura, @KoyamaSohei, @TezRomacH, @alexrobomind, @belldandyxtq, @c-bata, @chenghuzi, @crcrpar, @g-votte, @harupy, @himkt, @hvy, @jeromepatel, @keisuke-umezawa, @mavillan, @not522, @nyanhi, @nzw0301, @sfujiwara, @sile, @tktran, @toshihikoyanase, @twolffpiggott, @ydcjeff, @ytsmiling
Fix mypy local fails in tests/test_deprecated.py and tests/test_experimental.py
This is the release note of v2.5.0.
The ask-and-tell interface is a new complement to Study.optimize. It allows users to construct Trial instances without the need of an objective function callback, giving more flexibility in how to define search spaces, ask for suggested hyperparameters and how to evaluate objective functions. The interface is made out of two methods, Study.ask and Study.tell.
Study.ask returns a new Trial object.Study.tell takes either a Trial object or a trial number along with the result of that trial, i.e. a value and/or the state, and saves it. Since Study.tell accepts a trial number, a trial object can be disposed after parameters have been suggested. This allows objective function evaluations on a different thread or process.import optuna
from optuna.trial import TrialState
study = optuna.create_study()
# Use a Python for-loop to iteratively optimize the study.
for _ in range(100):
trial = study.ask() # `trial` is a `Trial` and not a `FrozenTrial`.
# Objective function, in this case not as a function but at global scope.
x = trial.suggest_float("x", -1, 1)
y = x ** 2
study.tell(trial, y)
# Or, tell by trial number. This is equivalent to `study.tell(trial, y)`.
# study.tell(trial.number, y)
# Or, prune if the trial seems unpromising.
# study.tell(trial, state=TrialState.PRUNED)
assert len(study.trials) == 100
Now, Optuna supports monitoring trial heartbeats with RDB storages. For example, if a process running a trial is killed by a scheduler in a cluster environment, Optuna will automatically change the state of the trial that was running on that process to TrialState.FAIL from TrialState.RUNNING.
# Consider running this script on several processes.
import optuna
def objective(trial):
(Very time-consuming computation)
# Recording heartbeats every 60 seconds.
# Other processes' trials where more than 120 seconds have passed
# since the last heartbeat was recorded will be automatically failed.
storage = optuna.storages.RDBStorage(url=..., heartbeat_interval=60, grace_period=120)
study = optuna.create_study(storage=storage)
study.optimize(objective, n_trials=100)
NSGA-II experimentally supports constrained optimization. Users can introduce constraints with the new constraints_func argument of NSGAIISampler.__init__.
The following is an example using this argument, a bi-objective version of the knapsack problem. We have 100 pairs of items and two knapsacks, and would like to maximize the profits of items within the weight limitation.
import numpy as np
import optuna
# Define bi-objective knapsack problem.
n_items = 100
n_knapsacks = 2
feasible_rate = 0.5
seed = 1
rng = np.random.RandomState(seed=seed)
weights = rng.randint(10, 101, size=(n_knapsacks, n_items))
profits = rng.randint(10, 101, size=(n_knapsacks, n_items))
constraints = (np.sum(weights, axis=1) * feasible_rate).astype(np.int)
def objective(trial):
xs = np.array([trial.suggest_categorical(f"x_{i}", (0, 1)) for i in range(weights.shape[1])])
total_weights = np.sum(weights * xs, axis=1)
total_profits = np.sum(profits * xs, axis=1)
# Constraints which are considered feasible if less than or equal to zero.
constraints_violation = total_weights - constraints
trial.set_user_attr("constraint", constraints_violation.tolist())
return total_profits.tolist()
def constraints_func(trial):
return trial.user_attrs["constraint"]
sampler = optuna.samplers.NSGAIISampler(population_size=10, constraints_func=constraints_func)
study = optuna.create_study(directions=["maximize"] * n_knapsacks, sampler=sampler)
study.optimize(objective, n_trials=200)
Study.ask, Study.tell) (#2158)constraints_func argument to NSGA-II (#2175)threading (#2190)Study.add_trials to simplify creating customized study (#2261)_CachedStorage (#2214)after_trial in PartialFixedSampler (#2209)trials_dataframe for multi-objective optimization with fail or pruned trials (#2265)calculate_weights_below method of MOTPESampler (#2274, thanks @y0z!)allennlp_*.py on GitHub Actions (#2226)mypy==0.790 (#2259)callback & (add|enqueue)_trial recipe (#2125)create_trial's documentation and tests richer (#2126)optuna/samplers/_base.py typo (#2239)examples/multi_objective/pytorch_simple.py (#2230)examples/multi_objective directory (#2244)examples/multi_objective/botorch_simple.py (#2245, thanks @nzw0301!)examples/mlflow (#2258, thanks @nzw0301!)create_trial's documentation and tests richer (#2126)SkoptSampler acquisition function in test to more likely converge (#2194)tests/test_study.py (#2218)tests/test_trial.py (#2219)STORAGE_MODES to testing/storage.py (#2231)study_tests/test_optimize.py (#2232)BoTorchSampler minor code fix reducing dictionary lookup and clearer type behavior (#2195)BoTorchSampler (#2197)study.get_trials instead of study._storage.get_all_trials (#2208)RDBStorage tests in github actions (#2200)pypi-publish.yml (#2187)mypy local fails in tests/test_deprecated.py and tests/test_experimental.py (#2191)CONTRIBUTING.md (#2192)This release was made possible by authors, and everyone who participated in reviews and discussions.
@Crissman, @HideakiImamura, @c-bata, @crcrpar, @g-votte, @himkt, @hvy, @keisuke-umezawa, @not522, @nzw0301, @sile, @srijan-deepsource, @tohmae, @toshihikoyanase, @y0z, @ytsmiling
optuna.multi_objective, used to be an experimental submodule for multi-objective optimization. This submodule is now deprecated. Changes required to m…
This is the release note of v2.4.0.
This is the first version to officially support Python 3.9. Everything is tested with the exception of certain integration modules under optuna.integration. We will continue to extend the support in the coming releases.
Multi-objective optimization in Optuna is now a stable first-class citizen. Multi-objective optimization allows optimizing multi objectives at the same time such as maximizing model accuracy while minimizing model inference time.
Single-objective optimization can be extended to multi-objective optimization by
directions instead of a single direction in optuna.create_study. Both parameters are supported for backwards compatibilityoptuna.create_study. If skipped, will default to the NSGAIISamplerSamplers that support multi-objective optimization are currently the NSGAIISampler, the MOTPESampler, the BoTorchSampler and the RandomSampler.
import optuna
def objective(trial):
# The Binh and Korn function. It has two objectives to minimize.
x = trial.suggest_float("x", 0, 5)
y = trial.suggest_float("y", 0, 3)
v0 = 4 * x ** 2 + 4 * y ** 2
v1 = (x - 5) ** 2 + (y - 5) ** 2
return v0, v1
sampler = optuna.samplers.NSGAIISampler()
study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler)
study.optimize(objective, n_trials=100)
# Get a list of the best trials.
best_trials = study.best_trials
# Visualize the best trials (i.e. Pareto front) in blue.
fig = optuna.visualization.plot_pareto_front(study, target_names=["v0", "v1"])
fig.show()
optuna.multi_objectiveoptuna.multi_objective, used to be an experimental submodule for multi-objective optimization. This submodule is now deprecated. Changes required to migrate to the new interfaces are subtle as described by the steps in the previous section.
With the introduction of multi-objective optimization, the database storage schema for the RDBStorage has been changed. To continue to use databases from v2.3, run the following command to upgrade your tables. Please create a backup of the database before.
optuna storage upgrade --storage <URL to the storage, e.g. sqlite:///example.db>
BoTorchSampler is an experimental sampler based on BoTorch. BoTorch is a library for Bayesian optimization using PyTorch. See example for an example usage.
For the first time in Optuna, BoTorchSampler allows constrained optimization. Users can impose constraints on hyperparameters or objective function values as follows.
import optuna
def objective(trial):
x = trial.suggest_float("x", -15, 30)
y = trial.suggest_float("y", -15, 30)
# Constraints which are considered feasible if less than or equal to zero.
# The feasible region is basically the intersection of a circle centered at (x=5, y=0)
# and the complement to a circle centered at (x=8, y=-3).
c0 = (x - 5) ** 2 + y ** 2 - 25
c1 = -((x - 8) ** 2) - (y + 3) ** 2 + 7.7
# Store the constraints as user attributes so that they can be restored after optimization.
trial.set_user_attr("constraint", (c0, c1))
return x ** 2 + y ** 2
def constraints(trial):
return trial.user_attrs["constraint"]
# Specify the constraint function when instantiating the `BoTorchSampler`.
sampler = optuna.integration.BoTorchSampler(constraints_func=constraints)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=32)
BoTorchSampler supports both single- and multi-objective optimization. By default, the sampler selects the appropriate sampling algorithm with respect to the number of objectives.
BoTorchSampler is customizable via the candidates_func callback parameter. Users familiar with BoTorch can change the surrogate model, acquisition function, and its optimizer in this callback to utilize any of the algorithms provided by BoTorch.
Visualization functions can now plot values other than objective values, such as inference time or evaluation by other metrics. Users can specify the values to be plotted by specifying the target argument. Even in multi-objective optimization, visualization functions can be available with the target argument along a specific objective.
The tutorial has been improved and new content for each Optuna’s key feature have been added. More contents will be added in the future. Please look forward to it!
Study and BaseStorage based on TrialState (#1943)fail_reason in trial system_attr (#1964)plot_contour and _get_contour_plot with Matplotlib backend (#1782, thanks @ytknzw!)plot_param_importances and _get_param_importance_plot with Matplotlib backend (#1787, thanks @ytknzw!)plot_slice and _get_slice_plot with Matplotlib backend (#1823, thanks @ytknzw!)PartialFixedSampler (#1892, thanks @norihitoishida!)Study and BaseStorage based on TrialState (#1943)node_rank and dry_run (#1959)BoTorchSampler (#1989)plot_pareto_front (#2000, thanks @okdshin!)plot_optimization_history with target values other than objective value (#2064)plot_contour with target values other than objective value (#2075)plot_parallel_coordinate with target values other than objective value (#2089)plot_slice with target values other than objective value (#2093)plot_edf with target values other than objective value (#2103)optuna.multi_objective.visualization.plot_pareto_front (#2110)ValueError if target is None and study is for multi-objective optimization for plot_contour (#2112)ValueError if target is None and study is for multi-objective optimization for plot_edf (#2117)ValueError if target is None and study is for multi-objective optimization for plot_optimization_history (#2118)plot_param_importances with target values other than objective value (#2119)ValueError if target is None and study is for multi-objective optimization for plot_parallel_coordinate (#2120)ValueError if target is None and study is for multi-objective optimization for plot_slice (#2121)NotImplementedError for trial.report and trial.should_prune during multi-objective optimization (#2135)ValueError in TPE and CMA-ES if study is being used for multi-objective optimization (#2136)ValueError if target is None and study is for multi-objective optimization for get_param_importances, BaseImportanceEvaluator.evaluate, and plot_param_importances (#2137)ValueError in integration samplers if study is being used for multi-objective optimization (#2145)include_package to AllenNLP for distributed setting (#2018)Study._is_multi_objective() to check whether study has multiple objectives (#2142, thanks @nyanhi!)TFKerasPruningCallback to warn when an evaluation metric does not exist (#2156, thanks @bigbird555!)Study.trials_dataframe for multi-objective optimization (#2181)weights_below in MOTPEMultiObjectiveSampler (#1979)matplotlib backend plot_parallel_coordinate (#2090)isnumerical to capture float values in plot_contour (#2096, thanks @nzw0301!)trial_values table (#2180)tests directory on install (#2015, thanks @130ndim!)setup.py requirements (#2051)xgboost<1.3 (#2084)blackdoc (#1982)codecov from CONTRIBUTING.md (#2005)plot_pareto_front (#2025)GridSampler (#2040)MOTPEMultiObjectiveSampler's example (#2045, thanks @norihitoishida!)pip install --find-links (#2065)lt symbol (#2068, thanks @KoyamaSohei!)RandomSampler in docs (#2071, thanks @akihironitta!)suggest_float with step argument (#2087)matplotlib.plot_parallel_coordinate example (#2097, thanks @nzw0301!)matplotlib.plot_param_importances example (#2098, thanks @nzw0301!)matplotlib.plot_slice example (#2099, thanks @nzw0301!)matplotlib.plot_contour example (#2100, thanks @nzw0301!)optuna.multi_objective deprecation (#2132)training_step of PyTorch Lightning example (#2043)examples/README.md (#2056, thanks @nai62!)enqueue_trial example (#2059)examples/multi_objective/plot_pareto_front.py to examples/visualization/plot_pareto_front.py (#2122)parametrize_sampler of test_samplers.py (#2020, thanks @norihitoishida!)trail_id + 123 -> trial_id (#2052)scipy==1.6.0 test failure with LogisticRegression (#2166)fail_reason in trial system_attr (#1964)_SearchSpaceTransform (#1988)dicts (#2007)__all__ to reexport modules explicitly (#2013)CmaEsSampler's warning message (#2019, thanks @norihitoishida!)structs.StudySummary against study.StudySummary (#2029)optuna.type_checking module (#2032)py35 from black config in pyproject.toml (#2035)session.query() (#2060)find_or_raise_by_id instead of find_by_id to raise if a study does not exist (#2061)LightGBMTuner (#2083)matplotlib.plot_slice example (#2099, thanks @nzw0301!)_run_trial refactoring (#2133)xgboost integration (#2143)set-env in GitHub Actions (#1992)checks from circleci (#2004)--diff option (#2031)tox.ini (#2024)CONTRIBUTING.md (#2159)pypi-publish.yml (#2188)This release was made possible by authors, and everyone who participated in reviews and discussions.
@130ndim, @Crissman, @HideakiImamura, @KoyamaSohei, @akihironitta, @alexrobomind, @bigbird555, @c-bata, @crcrpar, @eytan, @g-votte, @hal-314, @harupy, @himkt, @hvy, @keisuke-umezawa, @nai62, @norihitoishida, @not522, @nyanhi, @nzw0301, @okdshin, @pbmstrk, @sdaulton, @sile, @toshihikoyanase, @trivialfis, @ytknzw, @ytsmiling
This is the release note of v2.3.0.
This is the release note of v2.3.0.
TPE sampler now supports multi-objective optimization. This new algorithm is implemented in optuna.multi_objective and used viaoptuna.multi_objective.samplers.MOTPEMultiObjectiveSampler. See #1530 for the details.
LightGBMTunerCV returns the best boosterThe best booster of LightGBMTunerCV can now be obtained in the same way as the LightGBMTuner. See #1609 and #1702 for details.
The integration with PyTorch Lightning v1.0 is available. The pruning feature of Optuna can be used with the new version of PyTorch Lightning using optuna.integration.PyTorchLightningPruningCallback. See #597 and #1926 for details.
An example to illustrate how to use RAPIDS with Optuna is available. You can use this example to harness the computational power of the GPU along with Optuna.
optuna.multi_objective.samplers (#1530, thanks @y0z!)LGBMTunerCV booster (#1702, thanks @nyanhi!)plot_intermediate_values and _get_intermediate_plot with Matplotlib backend (#1762, thanks @ytknzw!)plot_optimization_history and _get_optimization_history_plot with Matplotlib backend (#1763, thanks @ytknzw!)plot_parallel_coordinate and _get_parallel_coordinate_plot with Matplotlib backend (#1764, thanks @ytknzw!)reseed_rng to NSGAIIMultiObjectiveSampler (#1938)MoTPEMultiObjectiveSampler (#1978)_MultivariateParzenEstimators (#1923, thanks @kstoneriv3!)plot_contour (#1929, thanks @carefree0910!)StudyDirection of mape in LightGBMTuner (#1966)with_trace method from docs (#1882, thanks @i-am-jeetu!)matplotlib.plot_edf (#1899)README.md (#1901)optuna.visualization and optuna.multi_objective.visualization (#1902)plot_edf figure in documentation by using matplotlib plot directive (#1905, thanks @harupy!)CmaEsSampler (#1909)matplotlib.plot_intermediate_values figure to doc (#1933, thanks @harupy!)matplotlib.plot_optimization_history figure to doc (#1934, thanks @harupy!)MOTPEMultiObjectiveSampler executable (#1953)Raises comments to samplers (#1965, thanks @yuk1ty!)examples/pytorch_lightning_simple.py (#1878, thanks @iamshnoo!)nest_trials for MLflowCallback works properly (#1932, thanks @harupy!)tag_study_user_attrs for MLflowCallback works properly (#1935, thanks @harupy!)Study.optimize (#1904)Study.trials_dataframe (#1907)optuna/logging.py (#1920, thanks @akihironitta!)_log_normal_cdf (#1922, thanks @kstoneriv3!)job-id of GitHub Actions workflows (#1898)mypy==0.782 (#1913)allennlp_jsonnet.py on GitHub Actions (#1915)store_artifacts step in document CircleCI job (#1962, thanks @harupy!)This release was made possible by authors, and everyone who participated in reviews and discussions.
@Crissman, @HideakiImamura, @Nanthini10, @akihironitta, @c-bata, @carefree0910, @crcrpar, @drobison00, @harupy, @himkt, @hvy, @i-am-jeetu, @iamshnoo, @keisuke-umezawa, @kstoneriv3, @nyanhi, @nzw0301, @resnant, @sile, @smly, @tktran, @toshihikoyanase, @y0z, @ytknzw, @yuk1ty
This is the release note of v2.2.0.
This is the release note of v2.2.0.
In this release, we drop support for Python 3.5. If you are using Python 3.5, please consider upgrading your Python environment to Python 3.6 or newer, or install older versions of Optuna.
TPESampler is updated with an experimental option to enable multivariate sampling. This algorithm captures dependencies among hyperparameters better than the previous algorithm. See #1767 for more details.
<img src="https://user-images.githubusercontent.com/3255979/95030825-40da5b80-06ed-11eb-84b1-fcc24dc1b70a.gif" width="480">
<!-- -->
<img src="https://user-images.githubusercontent.com/3255979/95030841-58b1df80-06ed-11eb-8e3c-a74e3687c78f.png" width="480">
<!-- -->
AllenNLPExecutor supports pruning. It is introduced in the official hyperparameter search guide by AllenNLP. Both AllenNLPExecutor and the guide were written by @himkt. See #1772.
optuna.visualization.matplotlib (#1756, thanks @ytknzw!)AllenNLPPruningCallback for AllenNLPExecutor (#1772)KerasPruningCallback to warn when an evaluation metric does not exist (#1759, thanks @bigbird555!)plot_edf and _get_edf_plot with Matplotlib backend (#1760, thanks @ytknzw!)LightGBMTuner (#1807, thanks @upura!)TensorBoardCallback (#1814, thanks @sfujiwara!)LightGBMTuner (#1822, thanks @nyanhi!)optuna.multi_objective.visualization.plot_pareto_front (#1824, thanks @nzw0301!)reseed_rng to RandomMultiobjectiveSampler (#1831, thanks @y0z!)IntLogUniformDistribution (#1788)mypy in an environment where some dependencies are installed (#1804)WFG._compute() (#1812, thanks @y0z!)CmaEsSampler (#1849)matplotlib directory to optuna.visualization.__init__.py (#1867)setup.py to drop Python 3.5 support (#1818, thanks @harupy!)setup.py (#1829, thanks @ytknzw!)plot_pareto_front preview path (#1808)multi_objective.visualization.plot_pareto_front (#1815, thanks @nzw0301!)__init__ from docs (#1820, thanks @upura!)README.md (#1825)isort in the contribution guidelines (#1842)isort (#1843)visualization.matpltlib to docs (#1847)Study.stop testcode (#1861)visualization.is_available (#1869)ThresholdPruner example (#1876, thanks @fsmosca!)optuna.logging.set_verbosity (#1884, thanks @nzw0301!)get_n_trials (#1568)isort to automatically sort import statements (#1695, thanks @harupy!)CmaEsSampler (#1777)logger member attributes from PyCmaSampler and CmaEsSampler (#1784)blackdoc (#1817)isort to visualization/matplotlib/ and multi_objective/visualization (#1830).scoring imports (#1864, thanks @norihitoishida!)matplotlib.* (#1868)--cache-from if trigger of docker image build is release (#1791)checks to GitHub Actions (#1838)--diff to black (#1840)This release was made possible by authors, and everyone who participated in reviews and discussions.
@HideakiImamura, @akihironitta, @bigbird555, @c-bata, @crcrpar, @fsmosca, @g-votte, @harupy, @himkt, @hvy, @keisuke-umezawa, @kstoneriv3, @norihitoishida, @nyanhi, @nzw0301, @sfujiwara, @sile, @sskarkhanis, @toshihikoyanase, @upura, @y0z, @ytknzw, @yuk1ty, @zchenry
Deprecate optuna.integration.KerasPruningCallback (#1670, thanks @VamshiTeja!)
This is the release note of v2.1.0.
Optuna v2.1.0 will be the last version to support Python 3.5. See #1067.
objective(study.best_trial)FrozenTrial used to subclass object but now implements BaseTrial. It can be used in places where a Trial is expected, including user-defined objective functions.
Re-evaluating the objective functions with the best parameter configuration is now straight forward. See #1503 for more details.
study.optimize(objective, n_trials=100)
best_trial = study.best_trial
best_value = objective(best_trial) # Did not work prior to v2.1.0.
CmaEsSampler comes with an experimental option to switch to IPOP-CMA-ES. This algorithm restarts the strategy with an increased population size after premature convergence, allowing a more explorative search. See #1548 for more details.
Comparing the new option with the previous CmaEsSampler and RandomSampler.
Optuna can be easily integrated with MLFlow on Kubernetes clusters. The example contained here is a great introduction to get you started with a few lines of commands. See #1464 for more details.
Type hint information is packaged following PEP 561. Users of Optuna can now run style checkers against the framework. Note that the applications which ignore missing imports may raise new type-check errors due to this change. See #1720 for more details.
Configuration files for AllenNLPExecutor may need to be updated. See #1544 for more details.
allennlp.common.params.infer_and_cast from AllenNLP integrations (#1544)optuna.integration.KerasPruningCallback (#1670, thanks @VamshiTeja!)FrozenTrial (#1503, thanks @nzw0301!)CmaEsSampler (#1548)TPESampler (#1562)WAITING trials in _CachedStorage (#1570)create_new_study method of storage classes (#1629, thanks @tohmae!)feature_pre_filter in LightGBMTuner (#1774)IntLogUniformDistribution (#1790)packaging in install_requires (#1551)fsspec<0.8.0 for Python 3.5 (#1596)packaging to >= 20.0 (#1599, thanks @Isa-rentacs!)lightgbm<3.0.0 to circumvent error with feature_pre_filter (#1773)StudySummary (#1533, thanks @nzw0301!)study-name (#1565, thanks @belldandyxtq!)README.md (#1573)BaseDistribution.single (#1593)README.md (#1597)CONTRIBUTING.md (#1601)examples/README.md (#1605)README.md (#1606)sphinx (#1613)MedianPruner) to the documentation (#1657, thanks @Chillee!)make clean (#1658)optuna.study.create_study (#1711, thanks @Ruketa!)optuna.study.load_study (#1712, thanks @bigbird555!)optuna.study.Study.optimize (#1726, thanks @norihitoishida!)Makefile (#1732, thanks @harupy!)optuna.study.delete_study (#1741, thanks @norihitoishida!)optuna.study.get_all_study_summaries (#1742, thanks @norihitoishida!)optuna.study.Study.set_user_attr (#1744, thanks @norihitoishida!)optuna.study.Study.user_attrs (#1745, thanks @norihitoishida!)optuna.study.Study.get_trials (#1746, thanks @norihitoishida!)optuna.multi_objective.study.MultiObjectiveStudy.optimize (#1747, thanks @norihitoishida!)optuna.trial (#1748)optuna.multi_objective.study.create_study (#1749, thanks @norihitoishida!)optuna.multi_objective.study.load_study (#1750, thanks @norihitoishida!)optuna.study.Study.stop (#1752, thanks @Ruketa!)examples/kubernetes/mlflow/README.md (#1540)plot_param_importances in example (#1555)optuna study optimize commands from examples (#1566, thanks @ritvik1512!)examples/kubernetes/* (#1584, thanks @VamshiTeja!)skorch pruning callback (#1668)tf.keras example (#1681, thanks @sfujiwara!)examples/pytorch_simple.py (#1725, thanks @wangxin0716!)_CachedStorage in test_study.py (#1575)tests/multi_objective as tests/multi_objective_tests (#1586)pytorch_lightning.data_loader decorator (#1667)trial (#1528, thanks @nzw0301!)WAITING trials in _CachedStorage (#1570)packaging to check the library version (#1610, thanks @VamshiTeja!)packaging.version (#1623)sample_from_categorical_dist (#1630)TPESampler (#1631, thanks @kstoneriv3!)n_ei_candidates for categorical parameters (#1637)optuna/integration/keras.py (#1642, thanks @airyou!)black in CONTRIBUTING.md (#1646)
1- Add type hints into optuna/cli.py (#1648, thanks @airyou!)optuna/dashboard.py, optuna/integration/__init__.py (#1653, thanks @airyou!)optuna/integration/_lightgbm_tuner (#1655, thanks @upura!)optuna/storages/__init__.py (#1661, thanks @akihironitta!)optuna/trial (#1662, thanks @upura!)optuna/testing (#1665, thanks @upura!)tests/storages_tests/rdb_tests (#1666, thanks @akihironitta!)optuna/samplers (#1673, thanks @akihironitta!)optuna.samplers._random (#1678, thanks @nyanhi!)optuna/integration/mxnet.py (#1679, thanks @norihitoishida!)optuna/pruners/_nop.py (#1680, thanks @Ruketa!)prunes/_percentile.py and prunes/_median.py (#1682, thanks @ytknzw!)args and kwargs (#1684, thanks @harupy!)optuna/pruners/_base.py and optuna/pruners/_successive_halving.py (#1685, thanks @ytknzw!)test_optimization_history.py (#1686, thanks @yosupo06!)tests/pruners_tests/test_median.py (#1687, thanks @polyomino-24!)visualization_tests (#1689, thanks @gasin!)trial from optuna.samplers._tpe.sampler._get_observation_pairs (#1692, thanks @ytknzw!)optuna/integration/chainer.py (#1693, thanks @norihitoishida!)optuna/integration/tensorflow.py (#1698, thanks @uenoku!)optuna/integration/chainermn.py (#1699, thanks @norihitoishida!)optuna/integration/xgboost.py (#1700, thanks @Ruketa!)tests/integration_tests (#1701, thanks @gasin!)Optional for keyword arguments that default to None (#1703, thanks @harupy!)tests/ (#1704, thanks @gasin!)optuna/integration (#1705, thanks @akihironitta!)LightGBMTuner (#1717, thanks @thigm85!)optuna/study.py into type annotations (#1724, thanks @harupy!)black==20.8b1 (#1730)optuna/integration/sklearn.py (#1735, thanks @akihironitta!)optuna/structs.py (#1743, thanks @norihitoishida!)optuna/samplers/_tpe/parzen_estimator.py (#1754, thanks @akihironitta!)allennlp_jsonnet.py example in CI (#1527)allennlp example directory in CI (#1585)actions/setup-python@v2 (#1594)cache to GitHub Actions Workflows (#1595)sphinx version to 3.0.4 (#1627, thanks @harupy!).dockerignore (#1633, thanks @harupy!)black in CONTRIBUTING.md (#1646)pyproject.toml for easier use of black (#1649)docs/Makefile (#1650)Several deprecated features (e.g., Study.study_id and Trial.trial_id) are removed. See #1346 for details.
This is the release note of v2.0.0.
The second major version of Optuna 2.0 is released. It accommodates a multitude of new features, including Hyperband pruning, hyperparameter importance, built-in CMA-ES support, grid sampler, and LightGBM integration. Storage access is also improved, significantly speeding up optimization. Documentation has been revised and navigation is made easier. See the blog for details.
The stable version of HyperbandPruner is available with a simpler interface and improved performance.
<img src="https://user-images.githubusercontent.com/5983694/88739399-6be88200-d175-11ea-9985-ce8c71d9538f.png" width="540px">
The stable version of the hyperparameter importance module is available.
FanovaImportanceEvaluator is now the default importance evaluator. This replaces the previous requirement for fanova with scikit-learn.visualization.plot_param_importances.The stable version of CmaEsSampler is available. This new CmaEsSampler can be used with pruning for major performance improvements.
The stable version of GridSampler is available through an intuitive interface for users familiar with Optuna. When the entire grid is exhausted, the optimization stops automatically, so you can specify n_trials=None.
The stable version of LightGBMTuner is available. The behavior regarding verbosity option has been improved. The random seed was fixed unexpectedly if the verbosity level is not zero, but now the user given seed is used correctly.
multi_objective.visualization.plot_pareto_front is available as an experimental feature.trial.create_trial and study.Study.add_trial are available as experimental features.Several deprecated features (e.g., Study.study_id and Trial.trial_id) are removed. See #1346 for details.
optuna.trial (#1371)LightGBMTuner (#1374)integration/chainermn.py (#1375)optuna/structs.py (#1377)optuna/study.py (#1379)Several features are deprecated.
optuna study optimize command (#1384)step argument in IntLogUniformDistribution (#1387, thanks @nzw0301!)Other.
BaseStorage.set_trial_param to return None instead of bool (#1327)suggest_float and suggest_int specifications on step and log arguments (#1329)BaseStorage.set_trial_intermediate_valute to return None instead of bool (#1337)optuna.integration.lightgbm_tuner private (#1378)IntLogUnioformDistribution.step during deprecation (#1438)LightGBMTuner verbosity level to the original LightGBM (#1504)CmaEsSampler (#1229)SkoptSampler (#1431)plot_pareto_front function (#1303)HyperbandPruner (#1435)Study.stop (#1450)GridSampler (#1451)LightGBMTuner (#1452)optuna.visualization.plot_param_importances (#1299)integration/CmaEsSampler to integration/PyCmaSampler (#1325)suggest_float and suggest_int specifications on step and log arguments (#1329)optuna.create_trial and Study.add_trial to create custom studies (#1335)deprecated (#1418)CatalystPruningCallback integration as experimental (#1465)optuna.visualization.plot_edf function (#1482)FanovaImportanceEvaluator as default importance evaluator (#1491)GridSampler (#1026)sklearn instead of fanova (#1106)NSGAIIMultiObjectiveSampler faster (#1257)log argument support for suggest_int of skopt integration (#1277, thanks @nzw0301!)read_trials_from_remote_storage method to Storage implementations (#1298)log argument for suggest_int of pycma integration (#1302)ImportError if bokeh version is 2.0.0 or newer (#1326)BaseStorage.set_trial_intermediate_valute to return None instead of bool (#1337)plot_param_importances figure (#1355)stacklevel for warnings.warn for more helpful warning message (#1419, thanks @harupy!)DeprecationWarning with FutureWarning in @deprecated (#1428)AllenNLPExecutor (#1449)LightGBMTuner (#1461)LightGBMTuner if verbosity == 1 (#1460)CategoricalDistribution (#1520)Several critical bugs are addressed in this release with the RDB storage, most related to distributed optimization.
sphinx update breaking existing type annotations (#1342)ChainerMNStudy.optimize (#1406)step to calculate range of IntUniformDistribution in PyCmaSampler (#1456)CachedStorage skipping trial param row insertion on cache miss (#1498)_CachedStorage and RDBStorage distribution compatibility check race condition (#1506)packaging in install_requires (#1561)python_requires in setup.py to clarify supported Python version (#1350, thanks @harupy!)classifiers in setup.py (#1358)keras 2.4.0 (#1386)sphinx version (#1393)cmaes (#1404)sphinx-rtd-theme and Python versions used on Read the Docs to CircleCI (#1434, thanks @harupy!)pfnopt (#1474)sphinx (#1485)packaging in install_requires (#1561)optuna (#1278)BaseStorage.set_trial_param (#1316)BaseStorage.get_best_trial and add unit tests (#1317).readthedocs.yml to use the same document dependencies on the CI and Read the Docs (#1354, thanks @harupy!)Colab to demonstrate a notebook instead of nbviewer (#1360)sphinx (#1369).. warning:: instead of .. note:: for the deprecation decorator (#1407)_templates/footer.html to _static/css/custom.css (#1439)custom.css to make it pretty and consistent (#1463, thanks @harupy!)CONTRIBUTING.md (#1466)CatalystPruningCallback in the documentation (#1468, thanks @harupy!)catch (#1473, thanks @harupy!)FrozenTrial (#1478)TensorBoardCallback to docs (#1486)step (#1489)plot_edf (#1510)timeout for relatively long-running examples (#1349)suggest_*uniform in examples with suggest_(int|float) (#1470)plot_param_importances test (#1328)test_experimental.py (#1332, thanks @harupy!)CmaEsSampler._get_trials() (#1433)pytorch_lightning.Trainer to disable checkpoint_callback (#1453)pytorch and torchvision (#1502)optuna._imports.try_import to DRY optional imports (#1315)plotly versions (#1338)optuna.visualization (#1359)optuna.pruners (#1361)optuna.samplers (#1362)logger to _trial's module variable (#1363)HyperbandPruner (#1366)__init__.py files (#1367, thanks @harupy!)optuna.storages (#1373)GridSampler (#1416)warnings.warn() or optuna.logging.Logger.warning() from codes which have both of them (#1421)deprecated by omitting removed version (#1422)CmaEsSampler (#1432)optuna.structs from MLflow integration (#1437)slice.py (#1267, thanks @bigbird555!)intermediate_values.py (#1268, thanks @bigbird555!)optimization_history.py (#1269, thanks @bigbird555!)utils.py (#1270, thanks @bigbird555!)test_logging.py (#1284, thanks @bigbird555!)test_chainer.py (#1286, thanks @bigbird555!)test_keras.py (#1287, thanks @bigbird555!)test_cma.py (#1288, thanks @bigbird555!)test_fastai.py (#1289, thanks @bigbird555!)test_integration.py (#1293, thanks @bigbird555!)test_mlflow.py (#1322, thanks @bigbird555!)test_mxnet.py (#1323, thanks @bigbird555!)optimize.py (#1364, thanks @bigbird555!)suggest_*uniform in examples with suggest_(int|float) (#1470)distributions.py (#1513)FloatingPointDistributionType (#1516)tests-python37 on CircleCI (#1348)doc-link from running on unrelated status update events (#1410, thanks @harupy!)ConfigSpace where Python 3.5 is dropped (#1471)circleci/python for dev image and install RDB servers (#1495)dockerimage.yml format (#1511)no-stale label from stale bot (#1321)no-stale label from stale bot (#1427)CONTRIBUTING.md (#1469, thanks @harupy!)2.0.0 (#1525)Several deprecated features (e.g., Study.study_id and Trial.trial_id) are removed. See #1346 for details.
A release candidate for the second major version of Optuna v2.0.0-rc0 is released! This release includes a lot of new features, cleaned up interfaces, performance improvements, internal refactorings and more. If you find any problems with this release candidate, please feel free to report them via GitHub Issues or Gitter.
The stable version of HyperbandPruner is available. It has a more simple interface and has seen performance improvement.
The stable version of the hyperparameter importance module is available.
FanovaImportanceEvaluator. While the previous implementation required fanova, this new FanovaImportanceEvaluator can be used with only scikit-learn.visualization.plot_param_importances.The stable version of CmaEsSampler is available. This new CmaEsSampler can be used with pruning, one of the Optuna’s important features, for great performance improvements.
The stable version of GridSampler is available and can be through an intuitive interface for users familiar with Optuna. When the entire grid is exhausted, the optimization also automatically stops so you can specify n_trials=None.
The stable version of LightGBMTuner is available. The behavior regarding verbosity option has been improved. The random seed was fixed unexpectedly if the verbosity level is not 0, but now the user given seed is used correctly.
multi_objective.visualization.plot_pareto_front is available as an experimental feature.trial.create_trial and study.Study.add_trial are available as experimental features.Several deprecated features (e.g., Study.study_id and Trial.trial_id) are removed. See #1346 for details.
optuna.trial. (#1371)LightGBMTuner. (#1374)integration/chainermn.py. (#1375)optuna/structs.py. (#1377)optuna/study.py. (#1379)Several features are deprecated.
optuna study optimize command. (#1384)step argument in IntLogUniformDistribution. (#1387, thanks @nzw0301!)Other.
BaseStorage.set_trial_param to return None instead of bool. (#1327)suggest_float and suggest_int specifications on step and log arguments. (#1329)BaseStorage.set_trial_intermediate_valute to return None instead of bool. (#1337)optuna.integration.lightgbm_tuner private. (#1378)IntLogUnioformDistribution.step during deprecation. (#1438)CmaEsSampler. (#1229)SkoptSampler. (#1431)plot_pareto_front function. (#1303)HyperbandPruner. (#1435)Study.stop. (#1450)GridSampler. (#1451)LightGBMTuner. (#1452)optuna.visualization.plot_param_importances. (#1299)integration/CmaEsSampler to integration/PyCmaSampler. (#1325)suggest_float and suggest_int specifications on step and log arguments. (#1329)optuna.create_trial and Study.add_trial to create custom studies. (#1335)deprecated. (#1418)CatalystPruningCallback integration as experimental. (#1465)GridSampler. (#1026)sklearn instead of fanova. (#1106)NSGAIIMultiObjectiveSampler faster. (#1257)log argument support for suggest_int of skopt integration. (#1277, thanks @nzw0301!)read_trials_from_remote_storage method to Storage implementations. (#1298)log argument for suggest_int of pycma integration. (#1302)ImportError if bokeh version is 2.0.0 or newer. (#1326)BaseStorage.set_trial_intermediate_valute to return None instead of bool. (#1337)plot_param_importances figure. (#1355)stacklevel for warnings.warn for more helpful warning message. (#1419, thanks @harupy!)DeprecationWarning with FutureWarning in @deprecated. (#1428)AllenNLPExecutor. (#1449)LightGBMTuner. (#1461)sphinx update breaking existing type annotations. (#1342)ChainerMNStudy.optimize. (#1406)step to calculate range of IntUniformDistribution in PyCmaSampler. (#1456)python_requires in setup.py to clarify supported Python version. (#1350, thanks @harupy!)classifiers in setup.py. (#1358)keras 2.4.0. (#1386)sphinx version. (#1393)cmaes. (#1404)sphinx-rtd-theme and Python versions used on Read the Docs to CircleCI. (#1434, thanks @harupy!)pfnopt. (#1474)optuna. (#1278)BaseStorage.set_trial_param. (#1316)BaseStorage.get_best_trial and add unit tests. (#1317).readthedocs.yml to use the same document dependencies on the CI and Read the Docs. (#1354, thanks @harupy!)Colab to demonstrate a notebook instead of nbviewer. (#1360)sphinx. (#1369).. warning:: instead of .. note:: for the deprecation decorator. (#1407)_templates/footer.html to _static/css/custom.css. (#1439)custom.css to make it pretty and consistent. (#1463, thanks @harupy!)CONTRIBUTING.md. (#1466)CatalystPruningCallback in the documentation. (#1468, thanks @harupy!)catch. (#1473, thanks @harupy!)timeout for relatively long-running examples. (#1349)plot_param_importances test. (#1328)test_experimental.py. (#1332, thanks @harupy!)CmaEsSampler._get_trials(). (#1433)pytorch_lightning.Trainer to disable checkpoint_callback. (#1453)optuna._imports.try_import to DRY optional imports. (#1315)plotly versions. (#1338)optuna.visualization. (#1359)optuna.pruners. (#1361)optuna.samplers. (#1362)logger to _trial's module variable. (#1363)HyperbandPruner. (#1366)__init__.py files. (#1367, thanks @harupy!)optuna.storages. (#1373)GridSampler. (#1416)warnings.warn() or optuna.logging.Logger.warning() from codes which have both of them. (#1421)deprecated by omitting removed version. (#1422)CmaEsSampler. (#1432)optuna.structs from MLflow integration. (#1437)slice.py. (#1267, thanks @bigbird555!)intermediate_values.py. (#1268, thanks @bigbird555!)optimization_history.py. (#1269, thanks @bigbird555!)utils.py. (#1270, thanks @bigbird555!)test_logging.py. (#1284, thanks @bigbird555!)test_chainer.py. (#1286, thanks @bigbird555!)test_keras.py. (#1287, thanks @bigbird555!)test_cma.py. (#1288, thanks @bigbird555!)test_fastai.py. (#1289, thanks @bigbird555!)test_integration.py. (#1293, thanks @bigbird555!)test_mlflow.py. (#1322, thanks @bigbird555!)test_mxnet.py. (#1323, thanks @bigbird555!)optimize.py. (#1364, thanks @bigbird555!)tests-python37 on CircleCI. (#1348)doc-link from running on unrelated status update events. (#1410, thanks @harupy!)ConfigSpace where Python 3.5 is dropped. (#1471)no-stale label from stale bot. (#1321)no-stale label from stale bot. (#1427)Stop suggesting using deprecated option in AllenNLP example.
This is the release note of v1.5.0.
LightGBM tuner, which provides efficient stepwise parameter tuning for LightGBM, supports cross-validation as an experimental feature with LightGBMTunerCV. See #1156 for details.
A sampler based on NSGA-II, a well-known multi-objective optimization algorithm, is now available as the default multi-objective sampler. The following benchmark result, on the ZDT1 function, shows that NSGA-II outperforms random sampling. Please refer to #1163 for further details.
The default hyperparameter importance evaluator is replaced with a naive mean decrease impurity algorithm. It uses the random forest feature importances in Scikit-learn and therefore requires this package. See #1253 for more details.
optuna.TrialPruned Aliasoptuna.TrialPruned is a new alias for optuna.exceptions.TrialPruned. It is now possible to write shorter and more readable code when pruning trials. See #1204 for details.
study.optimize. (#1025)--study-name instead of --study in CLI commands. (#1079, thanks @seungjaeryanlee!)LightGBMTuner. (#1156)suggest_int. (#1201, thanks @nzw0301!)optuna.exceptions.TrialPruned in __init__.py. (#1204)_get_observation_pairs for conditional parameters. (#1166, thanks @y0z!)HyperbandPruner. (#1196)RDBStorage to _CachedStorage. (#1263)_CachedStorage. (#1264)log argument for suggest_int of ChainerMN integration. (#1275, thanks @nzw0301!)Trial.suggest_int modifies high. (#1276)IntLogUniformDistribution. (#1279, thanks @himkt!)InMemoryStorage. (#1228)sklearn - skopt version incompatibility. (#1236)CmaEsSampler. (#1240)cmaes. (#1242)test_ to valid_ in docs and docstring. (#1167, thanks @himkt!)BaseStorage class doc. (#1174)BaseStorage method interfaces. (#1175)LightGBMTuner reference. (#1217)auto argument values in HyperbandPruner and SuccessiveHalvingPruner. (#1252)observation_key in XGBoostPruningCallback. (#1260)BaseStorage. (#1261)experimental decorator to decorate a class properly. (#1285, thanks @harupy!)dump_best_config in example. (#1225, thanks @himkt!)keras_integration.py. (#1301, thanks @zishiwu123!)sklearn - skopt version incompatibility. (#1236)trial.py. (#1210, thanks @himkt!)trial/*.py to trial/_*.py. (#1239)AllenNLPExecutor.__init__. (#1280)Trial.suggest_float. (#1292)n_brackets in HyperbandPruner. (#1294)LightGBMTuner and LightGBMTunerCV. (#1305)Add an argument of max_resource to HyperbandPruner and deprecate n_brackets.
This is the release note of v1.4.0.
Multi-objective optimization is available as an experimental feature. Currently, it only provides random sampling, but it will be continuously developed in the following releases. Feedback is highly welcomed. See #1054 for details.
A new Redis-based storage is available. It is a fast and flexible in-memory storage. It can also persist studies on-disk without having to configure relational databases. It is still an experimental feature, and your feedback is highly welcomed. See #974 for details.
Performance tuning has been applied to RDBStorage. For instance, it speeds up creating study lists by over 3000 times (i.e., 7 minutes to 0.13 seconds). See #1109 for details.
A new callback function is provided for MLFlow users. It reports Optuna’s optimization results (i.e., parameter values and metric values) to MLFlow. See #1028 for details.
A new integration module for AllenNLP is available. It enables you to reuse your jsonnet configuration files for hyperparameter tuning. See #1086 for details.
is_higher_better from TensorFlowPruningHook. (#1083, thanks @nuka137!)@abc.abstractmethod decorator to the abstract methods of BaseTrial and fixed ChainerMNTrial. (#1087, thanks @gorogoroumaru!)LogUniformDistribution for negative domains. (#1099)RedisStorage class to support storing activity on Redis. (#974, thanks @pablete!)study argument to optuna.integration.lightgbm.LightGBMTuner. (#1032)FrozenTrial and DataFrame. (#1071)LightGBMTuner. (#1076)number property to FixedTrial and BaseTrial. (#1077)DiscreteUniformDistribution in suggest_float. (#1081, thanks @himkt!)max_resource to HyperbandPruner and deprecate n_brackets. (#1138)HyperbandPruner. (#1141)IntersectionSearchSpace to speed up the search space calculation. (#1142)best_params in study. (#1150, thanks @himkt!)HyperbandPruner by deprecating min_early_stopping_rate_low. (#1159)KerasPruningCallback. (#1161, thanks @VladSkripniuk!)suggest_float with step in multi_objective. (#1205, thanks @nzw0301!)optuna.dashboard. (#1074)@abc.abstractmethod decorator to the abstract methods of BaseTrial and fixed ChainerMNTrial. (#1087, thanks @gorogoroumaru!)StudyDirection. (#1090)StudySummary. (#1095)TrialState and FrozenTrial. (#1101)optuna.structs to raise DeprecationWarning when using. (#1104)get_all_strudy_summaries function for RDB storages. (#1109)single() returns True when step or q is greater than high-low. (#1111)trial_id at study._append_trial(). (#1114)scipy for sampling from truncated normal in TPE sampler. (#1122)max_resource to HyperbandPruner. (#1171)n_brackets in HyperbandPruner. (#1188)ValueError that is not raised. (#1208, thanks @harupy!)MultiObjectiveStudy.optimize. (#1209)cmaes library. (#1082)HyperBandPruner. (#972)optuna/integration/*.py. (#1070, thanks @nuka137!)optuna/distributions.py. (#1089)FrozenTrial.distributions. (#1093)first.rst. (#1100, thanks @A03ki!)plot_intermediate_values example. (#1103)
sphinx version on RTD. (#1108)TPESampler. (#1144)optuna/storages/rdb/storage.py. (#1212, thanks @nuka137!)pytorch_simple.py to suggest lr from suggest_loguniform. (#1112)KerasPruningCallback example. (#1218)HyperbandPruner. (#1189)GridSampler.__init__. (#1102)mock with unittest.mock. (#1121)is_log logic in TPE sampler. (#1123)HyperbandPruner by deprecating min_early_stopping_rate_low. (#1159)Trial.system_attrs to store LightGBMTuner's results. (#1177)_TimeKeeper and use timeout of Study.optimize. (#1179)system_attrs as variables in LightGBMTuner. (#1192)colorlog after threading. (#1211, thanks @himkt!)IntUniformDistribution's step to UniformIntegerHyperparameter's q. (#1222, thanks @nzw0301!)torch with CUDA in CI. (#1118)torch with CUDA in CI by locking version. (#1124)llvmlite version for Python 3.5. (#1152)--cov option for pytest. (#1187, thanks @harupy!)actions/stale to never close ticket. (#1131)actions/stale on weekday mornings Tokyo time. (#1132)contribution-welcome and bug issues as stale. (#1216)This is the release note of v1.3.0.
This is the release note of v1.3.0.
A new built-in CMA−ES sampler is available. It is still an experimental feature, but we recommend trying it because it is much faster than the existing CMA-ES sampler from the integration submodule. See #920 for details.
Hyperparameter importances can be evaluated using optuna.importance.get_param_importances. This is an experimental feature that currently requires fanova. See #946 for details.
The per-trial log now shows the parameter configuration for the last trial instead of the so far best trial. See #965 for details.
step parameter on IntUniformDistribution. (#910, thanks @hayata-yamamoto!)ThresholdPruner. (#963, thanks @himkt!)suggest_float. (#1021, thanks @himkt!)SELECT FOR UPDATE while updating trial state. (#1014)n_warmup_steps documentation. (#980, thanks @PhilipMay!)code-block:: console. (#983)optuna/samplers/*.py and optuna/integration/*.py. (#999, thanks @nuka137!)optuna/integration/tensorflow.py. (#1019, thanks @nuka137!)RDBStorage. (#1022)intersection_search_space parameters. (#1053)allennlp example. (#949, thanks @himkt!)number column comment. (#1006)mypy==0.770 errors. (#1009)optuna/integration/pytorch_lightning.py. (#1024, thanks @nai62!)GridSampler. (#1027)autopep8 to black (string normalization separate commit). (#1030)sklearn.utils.safe_indexing for scikit-learn==0.24. (#1031, thanks @kuroko1t!)black error. (#1034)FATAL. (#1035)pytorch_lightning and bokeh. (#998)setup.cfg. (#985, thanks @pablete!).pytest_cache. (#991, thanks @harupy!)This is the release note of v1.2.0.
This is the release note of v1.2.0.
Study.enqueue_trial allows the user to specify fixed parameter values to be tried next instead of values from the suggestion algorithms. An example is available in the reference. See #520.
Note that this feature introduced an RDB schema change. If you have stored studies created by Optuna v1.1.0 or less in RDBs, please execute $ optuna storage upgrade --storage $URL after updating Optuna.
Grid search has been introduced as an experimental feature through GridSampler. See #665.
Trial.report to not update the trial value. (#854)Trial.report to not update the trial value. (#854)experimental warning. (#884)experimental decorator to accept optional name and apply to progress bar. (#918)n_startup_trials option to SkoptSampler. (#951)OptunaSearchCV by adding the method that was implemented in scikit-learn>=0.22.1. (#881, thanks @himkt!)Trial._check_distribution. (#934, thanks @PhilipMay!)LightGBMTuner to handle metrics with evaluation positions. (#912)DiscreteUniformDistribution. (#917)
Trial.suggest_discrete_uniform with the default sampler (i.e., TPESampler), optimization performance may degrade due to the issue (#916). Please update Optuna to this version or later.weakref.finalize instead of __del__ for RDBStorage. (#941)doctest to optuna/trial.py. (#882, thanks @nuka137!)enqueue_trial. (#927)cma.EvotionStrategy. (#929)parallel_coordinate. (#955, thanks @keisuke-umezawa!)doctest to optuna/exceptions.py. (#958, thanks @nuka137!)optuna.structs.TrialPruned (followup of #958). (#959)doctest to optuna/prunes/*.py. (#964, thanks @nuka137!)xgboost.cv using XGBoostPruningCallback. (#907, thanks @yutayamazaki!)__init__.py to tests/samplers_tests/tpe_tests. (#945, thanks @keisuke-umezawa!)numpy.ndarray as Python lists are no longer accepted in xgboost==1.0. (#947)_check_distribution. (#937, thanks @PhilipMay!)optuna.integration.lightgbm_tuner.train. (#943)This is the release note of v1.1.0.
This is the release note of v1.1.0.
Hyperband, an extension of the successive halving pruning algorithm, has been introduced as an experimental feature through HyperbandPruner. It is compatible with RandomSampler and TPESampler. See #809.
SuccessiveHalvingPruner minimum resource heuristicWhen min_resource is omitted, instead of defaulting to 1, the SuccessiveHalvingPruner now uses a heuristic to guess a suitable value. See #812.
pool_pre_ping=True for MySQL to avoid connection errors. (#806)min_resource in SuccessiveHalvingPruner. (#812)TPESampler to support HyperbandPruner. (#828)XGBoostPruningCallback to be compatible with xgboost.cv. (#865, thanks @yutayamazaki!)PyTorchIgnitePruningHandler report correct epoch. (#847)max_depth=-1. (#872)plot_intermediate_values. (#889)KerasPruningCallback pruning example code. (#832, thanks @harupy!)HyperbandPruner. (#875)examples/README.md. (#896)--pruning option to pytorch_lightning_simple.py. (#794, thanks @yutayamazaki!)FastAIPruningCallback. (#848)pytorch_lightning==0.6.0. (#858)typing external dependency. (#840, thanks @hbredin!).DS_Store. (#829)torchvision>=0.5.0 with Pillow version fix. (#839, thanks @hugovk!)Fix deprecated property access.
This is the release note of v1.0.0. See here for the complete list of solved issues and merged PRs.
The first major version of Optuna v1.0 is released. It includes a stable API, significant performance improvements by reduced memory allocations and database storage accesses. Documentation has been revised and Jupyter Notebook examples are available on Google Colab. The visualization API has become customizable such that layouts can be modified.
Due to the end-of-life (EOL) of Python 2 in January 1, 2020, Optuna has dropped the support for Python 2 and has now shifted to support Python 3.5.1 and above.
A developer blog has started. The first post is about this release and future roadmaps.
trials_dataframe to allow returning DataFrame with flattened columns. (#736)__eq__. (#726)study.best_trial for RDB. (#729, thanks @chris-chris!)structs.py. (#768)SourceFileLoader instead of imp module. (#778)InMemoryStorage with multi-processing. (#780)doctest. (#702)conda. (#739, thanks @crcrpar!)configurations.rst. (#747)intersphinx. (#754)BasePruner in docs. (#788, thanks @crcrpar!)BasePruner.prune. (#793, thanks @crcrpar!)tutorial/index.rst. (#816, thanks @harupy!)OptunaSearchCV. (#722, thanks @yutayamazaki!)scikit-image example. (#730, thanks @tohmae!)optuna in examples. (#746)quickstart.ipynb. (#753)examples/pytorch_simple.py. (#779, thanks @crcrpar!)should_prune in chainermn. (#764, thanks @crcrpar!)test_callbacks function. (#769)test_callbacks test thread-safe. (#773)Study unit test parameterizations. (#774)cache_mode argument to fix a broken test. (#790)optuna/visualization.py. (#681, thanks @crcrpar!)setup.py. (#742)StudySummary.__ne__. (#756)RDBStorage. (#765)logger to study's module variable. (#770, thanks @crcrpar!)cython from requirements. (#781)scipy. (#801)scikit-learn (<=0.22.0). (#826)pillow to avoid torchvision's issue. (#827)optuna.dashboard. (#698)optuna.samplers. (#699)optuna.integration. (#705)enable_cache option from RDBStorage. (#706)integration.__init__.py condition. (#712)six. (#714)__future__. (#715)bokeh-allow-websocket-origins required. (#716)study_id private. (#718)FrozenTrial.__lt__ to sort trials without key. (#719)setup.py. (#724)trials_dataframe to allow returning DataFrame with flattened columns. (#736)BaseDistribution.__hash__ to take __class__ into account. (#743)float, str and castable to float in categorical distribution. (#758)TrialState in trials dataframes. (#771)product_search_space function. (#772)Study.trials_dataframe(). (#775)_To avoid breaking existing code however, optuna.structs.TrialPruned is still available but marked as deprecated._
This is the release note of v0.19.0. See here for the complete list of solved issues and merged PRs.
The GitHub organization of this repository has been changed from pfnet, the organization for Preferred Networks, Inc. to optuna in order to widen the community, growing the project as an open source software.
optuna.exceptions has been introduced. Now, all exceptions defined in Optuna, including TrialPruned, have been moved out from optuna.structs to this new submodule. This is a part of a larger refactoring that we are currently working on to clean up the interfaces and make Optuna even easier to use. To avoid breaking existing code however, optuna.structs.TrialPruned is still available but marked as deprecated.
Due to the end-of-life (EOL) of Python 2 in January 2020, Optuna will drop Python 2 support in December 2019. This decision was made considering the following facts:
Trial.report. (#701)PercentilePruner. (#693)optuna.exceptions module. (#691){FrozenTrial,Trial}.trial_id private. (#663){Median,Percentile}Pruner. (#660)time_budget option on lightgbm_tuner.train(). (#684, thanks @momijiame!)Trial.should_prune. (#690)RDBStorage constructor. (#689)trial.report() doc. (#687)Optuna Contributors in the docs. (#676)Trial class. (#668)pytorch-lightning==0.5.3. (#700)optuna.visualization tests. (#630, thanks @crcrpar!)pytorch_lightning. (#694)This is the release note of v0.18.1. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.18.1. See here for the complete list of solved issues and merged PRs.
import optuna failure on Python2.7 environment without LightGBM installed. (#674)This is the release note of v0.18.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.18.0. See here for the complete list of solved issues and merged PRs.
Due to the end-of-life (EOL) of Python 2 in January 2020, Optuna will drop Python 2 support in December 2019.
This decision was made considering the following facts:
We plan to drop Python 2 support in the first release in December 2019.
OptunaConfig. (#653)optimize by default. (#638)TrialModel.where_study_id. (#640)trial_id. (#535, thanks @oda!)force_garbage_collection for every trial. (#533, thanks @oda!)PULL_REQUEST_TEMPLATE.md file. (#651)CONTRIBUTING.md file. (#649)OptunaSearchCV. (#639)BaseStudy.trials doc to specify order constraint. (#634)Study. (#591)--pruning option to the PyTorch Ignite example. (#633)tensorflow==1.15.0. (#620)plot_slice() of slice plot example in backquotes. (#608, thanks @crcrpar!)test_optimize_with_catch. (#650)ValueError test to test_get_contour_plot(). (#616, thanks @crcrpar!)pip progress bar from CircleCI log. (#648)install-examples step name to anchor. (#618)None check with assertion. (#610)This is the release note of v0.17.1. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.17.1. See here for the complete list of solved issues and merged PRs.
OptunaSearchCV document isn't generated. (#592)OptunaSearchCV. (#583)Change the deprecated product_search_space to intersection_search_space. (#532, thanks @oda!)
This is the release note of v0.17.0. See here for the complete list of solved issues and merged PRs.
Due to the end-of-life (EOL) of Python 2 in January 2020, Optuna will drop Python 2 support in December 2019.
This decision was made considering the following facts:
We plan to drop Python 2 support in December 2019, but the date has not been determined yet. The detailed schedule will be published in the next release.
NopPruner. (#555, thanks @Muragaruhae)tf.keras. (#530, thanks @sfujiwara!)delete_study API for storage package. (#524, thanks @c-bata!)Study._append_trial method. (#464)OptunaSearchCV that provides sklearn compatible API (experimental). (#357, thanks @Y-oHr-N!)datetime_start property to FixedTrial. (#548)test_keras_pruning_callback to be compatible with keras==2.3.0. (#537)product_search_space to intersection_search_space. (#532, thanks @oda!)datetime_start to ChainerMN integration. (#528, thanks @tanapiyo!)RDBStorage. (#525, thanks @tadan18!)datetime_start during a trial. (#523, thanks @tanapiyo!)TracebackType. (#547)suggest_discrete_uniform. (#544).gitignore to ignore files related to examples. (#578, thanks @crcrpar!)MXNetPruningCallback. (#570, thanks @crcrpar!)This is the release note of v0.16.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.16.0. See here for the complete list of solved issues and merged PRs.
_id suffix from BaseStorage.create_new_{study,trial}_id methods. (#506)InTrialStudy class. (#491)keras_simple.py to examples/README.md. (#508)create_study function. (#502, thanks @scouvreur!)This is the release note of v0.15.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.15.0. See here for the complete list of solved issues and merged PRs.
FronzenTrial.params_in_internal_repr field. (#462)BaseStudy.storage field private. (#472)Study and InTrialStudy. (#470)TPESampler. (#476)TPESampler document. (#474)BaseSampler document. (#456)This is the release note of v0.14.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.14.0. See here for the complete list of solved issues and merged PRs.
TPESampler faster. (#466, thanks @oda!)intersection_search_space function. (#461)TPESampler. (#445)TPESampler pruning aware. (#439)suggest_categorical is called with choices containing int and str. (#449)cma.py and trial.py. (#463, thanks @nmasahiro and @c-bata!)This is the release note of v0.13.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.13.0. See here for the complete list of solved issues and merged PRs.
BaseDistribution.single methods. (#431)product_search_space function. (#430)connect_args in RDBStorage constructor with engine_kwargs argument. (#425)This is the release note of v0.12.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.12.0. See here for the complete list of solved issues and merged PRs.
FixedTrial cannot handle categorical parameters correctly. (#402)optuna.load_study() function instead of optuna.Study() constructor. (#407)This is the release note of v0.11.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.11.0. See here for the complete list of solved issues and merged PRs.
distributions property to trial classes. (#383)This is the release note of v0.10.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.10.0. See here for the complete list of solved issues and merged PRs.
RDBStorage. (#349)ChainerMNStudy. (#348)TPESampler and ParzenEstimator. (#373)FronzenTrial could be created with low probability. (#361)KerasPruningCallback and TensorFlowPruningHook to document. (#371)Deprecate Trial.trial_id attribute.
This is the release note of v0.9.0. See here for the complete list of solved issues and merged PRs.
maximize. (#331)load_study function. (#330)Trial.number. (#285)Trial.trial_id with Trial.number in dashboard. (#345)Trial.trial_id attribute. (#344)Trial.trial_id with Trial.number in reference, tutorial, and examples. (#342, #341, #340, #339, #338, #337)Study.__init__. (#335)optuna.integration for Python 3.x. (#334)rdb.rst. (#328, thanks @usernameandme!)Trial.suggest_discrete_uniform(). (#321)This is the release note of v0.8.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.8.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.7.0. See here for the complete list of solved issues and merged PRs.
This is the release note of v0.7.0. See here for the complete list of solved issues and merged PRs.
--disallow-untyped-defs mypy flag. (#296)Your coding agent can read these notes before it upgrades. Set up the MCP server →