This is the release note of v5.0.0-rc1.

Installation

Since this is a pre-release, do not forget to specify the version number to upgrade Optuna.

pip install optuna==5.0.0rc1

Highlights

Massive default sampler algorithm enhancement

Multivariate TPE with a constant liar strategy and an enhanced bandwidth computation (Watanabe 2023) has become the default algorithm for single-objective optimization. Multi-Objective TPE has been adopted as the new default sampler for multi-objective optimization, replacing NSGA-II. We conducted comprehensive benchmarking, carefully selected default options, and modified implementation details to maximize optimization performance.

Conditional PED-ANOVA (KDD 2026)

Our paper, Conditional PED-ANOVA: Hyperparameter Importance in Hierarchical & Dynamic Search Spaces, has been accepted to the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2026)!

Optuna v5.0 now supports this feature and adopts it as the default algorithm for hyperparameter importance.

import optuna

# Starting in Optuna v5.0, the default hyperparameter importance algorithm
# supports conditional search spaces like the one below:
def objective(trial):
    classifier_name = trial.suggest_categorical("classifier", ["SVC", "RandomForest"])
    if classifier_name == "SVC":
        svc_c = trial.suggest_float("svc_c", 1e-10, 1e10, log=True)
        classifier_obj = sklearn.svm.SVC(C=svc_c, gamma="auto")
    else:
        rf_max_depth = trial.suggest_int("rf_max_depth", 2, 32, log=True)
        classifier_obj = sklearn.ensemble.RandomForestClassifier(
            max_depth=rf_max_depth, n_estimators=10
        )
    return …

study = optuna.create_study()
study.optimize(objective)

optuna.importance.get_param_importances(study)

New APIs for Constrained Optimization

Starting in Optuna v5.0, the interface for constrained optimization has changed:

• trial.set_constraint() is added to set constraint values.
• trial.constraints is added to get constraints.

Along with these new apis, constraints_func argument on samplers is now deprecated.

import optuna

def objective(trial):
    trial.set_constraint(“c0”, c0)
    trial.set_constraint(“c1”, c1)
    …

# The constraints_func argument is now deprecated.
sampler = optuna.samplers.TPESampler()
study = optuna.create_study(sampler=samplera)
study.optimize(objective)
print(study.best_trial.constraints)

Breaking Changes

• Remove optuna.multi_objective module (#6686)
• Remove deprecated integration wrappers for AllenNLP, Chainer, and MXNet (#6693)
• Stabilize GPSampler (#6715)
• Enable constant_liar by default (#6738)
• Support categorical parameters in QMCSampler (#6742, thanks @saivedant169!)
• Enable multivariate by default in TPESampler (#6746)
• Fix: remove deprecated positional-arg compatibility from study and trial APIs (#6747, thanks @yen-0!)
• Make PedAnovaImportanceEvaluator the default importance evaluator (#6748)
• Deprecate constraints_func (#6773)
• Change to treat cases without constraints as feasible (#6774)
• Normalize trial timestamps to UTC in RDBStorage and JournalStorage (#6776)
• Remove axis_order argument from plot_pareto_front (#6781)
• Remove system_attrs from StudySummary (#6782)
• Remove categorical_distance_func from TPESampler (#6783)

New Features

• Implement qLogEI acquisition function (#6640)
• Add support for conditional search spaces in PED-ANOVA (#6682)
• Add conditional gp regressor (#6721)
• Add mutation to ga (#6724, thanks @hrntsm!)
• Use MOTPE-like split in PED-ANOVA for multi-objective studies when target is None (#6728)
• Add constraints property to Trial (#6736)
• Add set_constraint method to Trial (#6754)

Enhancements

• Fix QMCSampler fallback to independent sampling in distributed setups (#6638, thanks @Rishabh-git10!)
• Speed up BruteForceSampler by avoiding full tree build based on tree size check (#6646)
• Speed up BruteForceSampler by candidates caching (#6650)
• Split BruteForceSampler refactoring [3/3] (#6657)
• Refactor PED-ANOVA (#6681)
• Add lazy tree node to BruteForceSampler for speedup (#6705)
• Raise ValueError in PedAnovaImportanceEvaluator for multi-objective studies without target (#6716)
• Fix for the GPSampler OMP issue (#6753)
• Make TPESampler the default sampler for multi-objective optimization (#6766)

Bug Fixes

• Handle insufficient trials without raising ValueError in importances (#6720)
• Fix a concurrency issue when setting study attributes (#6751)

Documentation

• Add ablation study tutorial using BruteForceSampler (#6652)
• Simplify the document for deprecated CmaEsSampler options (#6694)
• Move deprecated TPESampler options to the end (#6696)
• Fix typos in docstrings and duplicated author names in SPXCrossover citation (#6698, thanks @Divyansh-ag14!)
• Fix wording in specify params tutorial (#6711, thanks @Ryo2611!)
• Add a documentation about constrained TPESampler (#6712)
• Enhance AutoSampler citation path (#6714)
• Update sampler table (#6717)
• docs: clarify n_warmup_steps boundary in MedianPruner and PercentilePruner (#6733, thanks @vin0san!)
• Fix trial report doc (#6735)
• chore: consolidate instruction for external dependency (#6749)
• Update sampler table for constrained optimization (#6775)

Tests

• Refactoring importance tests (#6725)
• Add importance evaluator test cases to optuna.testing (#6765)

Code Fixes

• Refactor batched distribution classes to reduce branch duplication (#6689)
• Reduce redundancy in test_brute_force.py (#6706)
• Replace SciPy with Torch in gp.py (#6710)
• Fix and refactor params validation in PED-ANOVA (#6729)
• Show all study names upon load/create study failure (#6750)
• Changed type of study's direction into Literal | StudyDirection (#6762, thanks @yen-0!)
• Fix unused mypy ignores in optuna/visualization/matplotlib/_rank.py (#6767)
• Avoid unnecessary trial values in storage tests (#6769)
• Follow-up #6762: Fix mypy errors (#6778)

Continuous Integration

• Specify workflow permissions explicitly (#6687)
• Capture Windows stdout (#6699)
• Fix logging tests broken by pytest 9.1.0 caplog behavior change (#6719)
• Hotfix for Sphinx CI (#6726)
• Fix mypy errors with latest NumPy stubs (#6730)
• Skip flaky gRPC journal storage tests (#6770)

Other

• Bump up to version number v5.0.0.dev (#6685)
• Add attestations: false to fix release workflow (#6690)
• Update README with Optuna 4.9.0 release news (#6701)
• Add CODEOWNERS file (#6718)
• Bump the version up to 5.0.0rc1 (#6784)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@Alnusjaponica, @Divyansh-ag14, @Rishabh-git10, @Ryo2611, @c-bata, @gen740, @himkt, @hrntsm, @kAIto47802, @nabenabe0928, @not522, @porink0424, @saivedant169, @sawa3030, @vin0san, @y0z, @yen-0

This is the release note of v4.9.0.

Highlights

Enhance Multi-Objective Constrained Parallel Optimization in GPSampler

@sawa3030 introduces parallelization enhancements to GPSampler, leveraging the Kriging Believer approach for constrained and multi-objective optimization (#6481). This improvement allows for more efficient exploration when multiple trials are running concurrently.

The GP surrogate is updated by assigning temporary objective function values ​​to the running trials.

For more technical details and benchmarks, please check out our blog post: Improving Optuna’s GPSampler Parallelization by Considering Running Trials.

Deprecate Several Features

The following features are deprecated in v4.9.0 and scheduled for removal in v6.0.0.

optuna

• Several arguments in TPESampler (#6635)
  • prior_weight, consider_magic_clip, consider_endpoints, gamma, weights, hyperopt_parameters: These internal parameters are being deprecated to simplify the interface, as the default settings are optimal for most use cases.
  • warn_independent_sampling: Deprecated because TPESampler now robustly supports both independent and joint sampling, making this warning obsolete.
  • categorical_distance_func: This advanced feature will be migrated to OptunaHub in the future.
• x0 and sigma0 options in CmaEsSampler (#6624)
  • These options have been deprecated because they require a deep understanding of CmaEsSampler's internals to be configured effectively.
• optuna.terminator module (#6668)
  • This feature will be migrated to OptunaHub in the future.
• RetryFailedTrialCallback (#6670)
  • This class has been renamed to RetryHeartbeatStaleTrialCallback to better reflect its behavior and avoid confusion with general trial retries (#6085).
• optuna.integration module
  • The optuna.integration module currently acts as a shortcut to the external optuna_integration package for backward compatibility. Please import directly from the optuna_integration package going forward.

optuna-integration

• PyCmaSampler: Please use Optuna's native CmaEsSampler instead.
• CometCallback: This feature will be migrated to OptunaHub in the future.
• MLflowCallback: This feature will be migrated to OptunaHub in the future.
• TensorBoardCallback: This feature will be migrated to OptunaHub in the future.
• TrackioCallback: This feature will be migrated to OptunaHub in the future.
• WeightsAndBiasesCallback: This class has already been migrated to OptunaHub.

Breaking Changes

• Deprecate PyCmaSampler (optuna/optuna-integration#276)
• Add deprecation message for CometCallback (optuna/optuna-integration#280)
• Add deprecation message for MLflowCallback (optuna/optuna-integration#281)
• Add deprecation message for TensorBoardCallback (optuna/optuna-integration#282)
• Add deprecation message for WeightsAndBiasesCallback (optuna/optuna-integration#283)
• Add deprecation message for TrackioCallback (optuna/optuna-integration#284)
• Fix importance computation for single-distributed params (#6500)
• Make QMCSampler stateless (#6616)
• Deprecate x0 and sigma0 options in CmaEsSampler (#6624)
• Deprecate a set of TPESampler arguments (#6635)
• Deprecate legacy imports from optuna.integration (#6667)
• Deprecate optuna.terminator module (#6668)
• Rename RetryFailedTrialCallback to RetryHeartbeatStaleTrialCallback (#6670)

Enhancements

• Enhance Multi-Objective Constrained Parallel Optimization in GPSampler (#6481)
• Remove prior_mu from compute_sigmas (#6574)
• Fix sampling bias in BruteForceSampler (#6627, thanks @Rishabh-git10!)
• Refactor BruteForceSampler (#6645)
• Speed up BruteForceSampler by using any instead of count (#6647)
• Fix the return type of BruteForceSampler (#6648)
• Split BruteForceSampler refactoring [2/3] (#6656)
• fix: collect all infeasible values in error message instead of early return (#6661, thanks @AshutoshDevpura!)

Bug Fixes

• Fix categorical None handling in slice plots (#6621)
• fix: handle JSONDecodeError in TPESampler._get_params to avoid race condition (#6628, thanks @AshutoshDevpura!)
• Split BruteForceSampler refactoring [2/3] (#6656)

Documentation

• Update LightGBM links (#6511, thanks @jameslamb!)
• Improve doc for best_trial/best_trials in constrained optimization (#6522)
• Update CmaEsSampler docstring regarding categorical support (#6625)
• Fix RST note directive typo in QMCSampler docstring (#6631, thanks @RudrenduPaul!)
• Enhance Generative Engine Optimization (GEO) of many objective and constraint handling (#6639)
• Add NSGA-III to FAQ about constraint (#6641)
• Update BruteForceSampler and GridSampler information in docs (#6651)
• Enhance FAQ about killing trials (#6653)

Examples

• Explicitly install tensorboard in the tensorboard example CI (optuna/optuna-examples#356)
• Remove examples which uses deprecated integration modules (optuna/optuna-examples#357)

Code Fixes

• Use broadcast_object_list instead of a custom method (optuna/optuna-integration#274)
• Correct return type annotations for Axes in visualization functions (#6504, thanks @kvr06-ai!)
• Fix type checking in trial folder (#6510, thanks @sateeshkumarb!)
• Use TYPE_CHECKING in importance/_mean_decrease_impurity.py (#6514, thanks @saivedant169!)
• Use TYPE_CHECKING in importance/_ped_anova/evaluator.py (#6515, thanks @saivedant169!)
• Use TYPE_CHECKING in samplers/_cmaes.py (#6516, thanks @saivedant169!)
• Use TYPE_CHECKING in samplers/_base.py (#6517, thanks @saivedant169!)
• Use TYPE_CHECKING in pruners/_wilcoxon.py (#6518, thanks @saivedant169!)
• Use TYPE_CHECKING in pruners/_percentile.py (#6520, thanks @saivedant169!)
• Use TYPE_CHECKING in nsgaii/_constraints_evaluation.py (#6521, thanks @saivedant169!)
• Use TYPE_CHECKING in nsgaii/_elite_population_selection_strategy.py (#6523, thanks @saivedant169!)
• Use TYPE_CHECKING in nsgaii/_after_trial_strategy.py (#6524, thanks @saivedant169!)
• Use TYPE_CHECKING in nsgaii/_child_generation_strategy.py (#6526, thanks @saivedant169!)
• Move type-only imports in optuna.samplers._partial_fixed to TYPE_CHECKING (#6527, thanks @t7r0n!)
• Fix TC006 cast annotation in optuna._gp.acqf (#6528, thanks @t7r0n!)
• Fix TC006 cast annotation in optuna.trial._frozen (#6529, thanks @t7r0n!)
• Fix TC006 cast annotation in tests.test_distributions (#6530, thanks @t7r0n!)
• Fix TC006 cast annotation in optuna.study._multi_objective (#6531, thanks @t7r0n!)
• Fix TC006 cast annotation in optuna.terminator.improvement.emmr (#6532, thanks @t7r0n!)
• Fix TC006 cast annotation in optuna.terminator.erroreval (#6533, thanks @t7r0n!)
• Fix TC006 cast annotation in optuna.study.study (#6534, thanks @t7r0n!)
• Move type-only Callable import in optuna.testing.threading behind TYPE_CHECKING (#6535, thanks @t7r0n!)
• Move type-only TracebackType import in optuna.testing.tempfile_pool behind TYPE_CHECKING (#6536, thanks @t7r0n!)
• Move type-only stdlib imports in optuna.testing.storages behind TYPE_CHECKING (#6537, thanks @t7r0n!)
• test: move type-only imports in optuna.testing.trials behind TYPE_CHECKING (#6538, thanks @t7r0n!)
• test: move type-only imports in optuna.testing.pytest_samplers behind TYPE_CHECKING (#6539, thanks @t7r0n!)
• test: move type-only imports in optuna.testing.pytest_storages behind TYPE_CHECKING (#6540, thanks @t7r0n!)
• test: move type-only imports in optuna.testing.samplers behind TYPE_CHECKING (#6541, thanks @t7r0n!)
• Quote cast type expression in tests/test_distributions.py (#6543, thanks @t7r0n!)
• Move type-only datetime import in tests/trial_tests/test_trial.py (#6544, thanks @t7r0n!)
• Move type-only imports to TYPE_CHECKING in samplers/_tpe/sampler.py (#6545, thanks @yasumorishima!)
• Move type-only imports to TYPE_CHECKING in nsgaii/_crossover.py (#6546, thanks @yasumorishima!)
• Remove dead code: unused functions, methods, and variables (#6547, thanks @duriantaco!)
• Fix typing import issues in scott_parzen_estimator (#6548, thanks @rpathade!)
• fix(types): use TYPE_CHECKING for imports in samplers/_cmaes.py (#6550, thanks @Aliipou!)
• fix(types): use TYPE_CHECKING for BaseDistribution import in _transform.py (#6552, thanks @Aliipou!)
• refactor: move BaseDistribution import to TYPE_CHECKING in search_space/group_decomposed.py (#6553, thanks @Aliipou!)
• Use TYPE_CHECKING for imports in importance/_ped_anova/scott_parzen_estimator.py (#6554, thanks @Aliipou!)
• fix(types): use TYPE_CHECKING for imports in samplers/_brute_force.py (#6555, thanks @Aliipou!)
• Move type-only imports to TYPE_CHECKING in samplers/_nsgaiii/_elite_population_selection_strategy.py (#6557, thanks @Aliipou!)
• Move type-only imports to TYPE_CHECKING in samplers/_nsgaiii/_sampler.py (#6558, thanks @Aliipou!)
• Move Study import to TYPE_CHECKING in _timeline.py (#6559, thanks @rpathade!)
• Use TYPE_CHECKING in optuna/samplers/_ga/_base.py (#6560, thanks @nightcityblade!)
• Use TYPE_CHECKING in nsgaii/_crossovers/_base.py (#6561, thanks @saivedant169!)
• Use TYPE_CHECKING in nsgaii/_crossovers/_blxalpha.py (#6562, thanks @saivedant169!)
• Use TYPE_CHECKING in nsgaii/_crossovers/_uniform.py (#6563, thanks @saivedant169!)
• Move imports to TYPE_CHECKING in tests/test_multi_objective (#6564, thanks @acabellom!)
• Use TYPE_CHECKING in nsgaii/_sampler.py (#6565, thanks @saivedant169!)
• Use TYPE_CHECKING in optuna.samplers._qmc (#6566, thanks @hnshah!)
• Use TYPE_CHECKING in visualization/_rank.py (#6567, thanks @nightcityblade!)
• Use TYPE_CHECKING in visualization/_parallel_coordinate.py (#6568, thanks @nightcityblade!)
• Use TYPE_CHECKING in visualization/_slice.py (#6569, thanks @nightcityblade!)
• Use TYPE_CHECKING in visualization/_intermediate_values.py (#6570, thanks @nightcityblade!)
• Use TYPE_CHECKING in visualization/_hypervolume_history.py (#6571, thanks @nightcityblade!)
• Use TYPE_CHECKING in visualization/_contour.py (#6572, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/_edf.py (#6575, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/_optimization_history.py (#6576, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/_hypervolume_history.py (#6577, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/_pareto_front.py (#6578, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/_terminator_improvement.py (#6579, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/_rank.py (#6580, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/_slice.py (#6581, thanks @saivedant169!)
• Fixed spelling/typo in error message (#6583, thanks @craigulliott!)
• Use TYPE_CHECKING in visualization/_parallel_coordinate.py (#6584, thanks @saivedant169!)
• Use TYPE_CHECKING in optuna/visualization/matplotlib/_utils.py (#6585, thanks @nightcityblade!)
• Use TYPE_CHECKING in optuna/visualization/matplotlib/_edf.py (#6586, thanks @nightcityblade!)
• Use TYPE_CHECKING in optuna/visualization/matplotlib/_contour.py (#6587, thanks @nightcityblade!)
• Use TYPE_CHECKING in optuna/visualization/matplotlib/_pareto_front.py (#6588, thanks @nightcityblade!)
• Use TYPE_CHECKING in optuna/visualization/matplotlib/_param_importances.py (#6589, thanks @nightcityblade!)
• Fix missing blank line after TYPE_CHECKING in visualization/_intermediate_values.py (#6591, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/matplotlib/_optimization_history.py (#6592, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/matplotlib/_parallel_coordinate.py (#6593, thanks @saivedant169!)
• Use TYPE_CHECKING in visualization/matplotlib/_rank.py (#6594, thanks @saivedant169!)
• Address integration type checking in optuna.integration.__init__.py (#6597)
• Remove unused type ignore comment (#6607)
• fix: move FrozenTrial import into TYPE_CHECKING block in storages/_callbacks.py (#6609, thanks @satishkc7!)
• fix: move type-only imports to TYPE_CHECKING in StudySummary (#6610, thanks @Zelys-DFKH!)
• Enhance the error message of QMCSampler (#6614)
• Enhance the warning message for TPESampler with multivariate=True (#6618)
• Simplify some GP processing and update staled comments (#6642)
• Enhance LSP experience by adding explicit references for optuna modules (#6671)
• Remove unused variable in optuna.storages._rdb.storage.py (#6672)

Continuous Integration

• Exclude Python 3.9 from fastaiv2 CI (optuna/optuna-integration#273)
• Pinned every action for pypi-publish to a commit SHA (optuna/optuna-integration#285)
• Restore TestPyPI repository URL for scheduled publishes (optuna/optuna-integration#286)
• Pinned every action for pypi-publish to a commit SHA (#6675)
• Restore TestPyPI repository URL for scheduled publishes (#6677)

Other

• Bump the version up to v4.9.0.dev (optuna/optuna-integration#268)
• Bump up version (

Note truncated.

This is the release note of v4.8.0.

Highlights

Support for Constant Liar Strategy to GPSampler

A constant liar strategy for efficient parallelization has been introduced to GPSampler by @sawa3030. The figures (left: v4.7.0, right: v4.8.0) show that the overlap of search points has decreased, and a wider variety of solutions are being explored. The experiment uses n_jobs = 10 and n_trials = 100. Currently, this feature supports single-objective and unconstrained optimization. Further extensions are coming in v4.9.0.

SHAP-like beeswarm plot visualization

@yasumorishima introduces the new visualization to OptunaHub. Please refer to https://hub.optuna.org/visualization/plot_beeswarm/ for details.

New Features

• Add Trackio Integration for Optuna (optuna/optuna-integration#259, thanks @ParagEkbote!)
• Add constant liar strategy to GPSampler (#6430)

Enhancements

• Validate artifact_id in FileSystemArtifactStore to prevent path traversal (#6432, thanks @RinZ27!)
• fix: correct inverted warning message in pareto front plot (#6498, thanks @aerosta!)

Bug Fixes

• Fix shared callback state in parallel OptunaSearchCV with LightGBM (optuna/optuna-integration#260, thanks @Quant-Quasar!)
• Fix GPSampler crash when torch default device is CUDA (#6418, thanks @VedantMadane!)
• Fix combination between PartialFixedSampler and TPESampler with group decomposed search space (#6428)
• Fix TPESampler with multivariate and constant_liar (#6505)

Documentation

• Add documentation indicating that WilcoxonPruner requires scipy (#6477)
• Remove version & language selectors in the sidebar of docs (#6482)

Examples

• Apply black 26.1.0 (optuna/optuna-examples#348)
• Remove CI workflow for allennlp since no longer maintained (optuna/optuna-examples#351)
• Reduce the frequency of scheduled CI triggers (optuna/optuna-examples#352)
• Remove the scheduled trigger for aim CI (optuna/optuna-examples#353)
• Add constraint to transformers (optuna/optuna-examples#355)

Tests

• Add SamplerTestCase class in optuna.testing package (#6424)
• Move test_before_trial and test_after_trial_* to test_trial.py and test_study.py, respectively (#6429)

Code Fixes

• Move type-only imports to TYPE_CHECKING in _param_importances.py (#6423, thanks @dotz0ver!)
• Use future.annotations in matplotlib/_intermediate_values (#6425, thanks @Lakshman142!)
• Reformat files with the latest version of ruff (#6426)
• Replace .format() with f-strings in _parallel_coordinate.py (#6431, thanks @yasumorishima!)
• Use TYPE_CHECKING for import in pruners/_base.py (#6434, thanks @yasumorishima!)
• Use f-strings in optuna/storages/_base.py (#6435, thanks @edwiniac!)
• Use f-strings in _contour.py (#6436, thanks @edwiniac!)
• Use f-strings in _intermediate_values.py (#6437, thanks @edwiniac!)
• Use f-strings in cli.py (#6438, thanks @edwiniac!)
• Use f-strings in optuna/testing/storages.py (#6439, thanks @edwiniac!)
• Use f-strings in optuna/storages/journal/_storage.py (#6440, thanks @edwiniac!)
• Use f-strings in _imports.py (#6442, thanks @edwiniac!)
• Use f-strings in _deprecated.py (#6443, thanks @edwiniac!)
• Use TYPE_CHECKING for import in pruners/_hyperband.py (#6447, thanks @yasumorishima!)
• Use TYPE_CHECKING for typing-only imports in test_timeline.py (#6451, thanks @KRMed!)
• Refactor: Use f string in multi_objective tutorial (#6455, thanks @ryota717!)
• refactor: move type-only imports into TYPE_CHECKING in samplers/_grid.py (#6466, thanks @dhruvildarji!)
• refactor: move BaseDistribution into TYPE_CHECKING in search_space/intersection.py (#6467, thanks @dhruvildarji!)
• Simplify Union type alias in optuna/samplers/_cmaes.py (#6478, thanks @roli-lpci!)
• Use f-string in _convert_positional_args.py (#6483, thanks @toroleapinc!)
• Add acqf import inside a TYPE_CHECKING block (#6484, thanks @acabellom!)
• Refactor __repr__ in trial/_frozen.py to use f-strings (#6485, thanks @Bhavyag1337!)
• Use TYPE_CHECKING in pruners/_threshold.py (#6487, thanks @LuciferDono!)
• Use f-string in tutorial/10_key_features/005_visualization.py (#6489, thanks @maheer14!)
• Move JSONSerializable import to TYPE_CHECKING in study/study.py (#6490, thanks @yasumorishima!)
• Use f-string and {var_name=} instead of .format and var_name={var_name} (#6494, thanks @buddy0452004!)
• Move optuna import to TYPE_CHECKING in pruners/_patient.py (#6501, thanks @yasumorishima!)
• Move application import to TYPE_CHECKING in pruners/_patient.py (#6502, thanks @nightcityblade!)
• Move application import to TYPE_CHECKING in pruners/_successive_halving.py (#6503, thanks @nightcityblade!)
• Replace .format() with f-strings in 002_configurations.py (#6506, thanks @acabellom!)
• Fix type checking (#6507, thanks @sateeshkumarb!)

Continuous Integration

• Fix test for FastAIV2PruningCallback (optuna/optuna-integration#265)
• Fix windows-test failure due to NamedTemporaryFilePool (#6417)
• Fix checks-optional CI (#6422)
• Fix mypy errors in GP module (#6492)

Other

• Bump up version number to 4.8.0.dev (optuna/optuna-integration#263)
• Bump the version up to v4.8.0 (optuna/optuna-integration#269)
• Bump up version to 4.8.0.dev (#6414)
• Update news section for the v4.7 release (#6420)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@Bhavyag1337, @KRMed, @Lakshman142, @LuciferDono, @ParagEkbote, @Quant-Quasar, @RinZ27, @VedantMadane, @acabellom, @aerosta, @buddy0452004, @c-bata, @dhruvildarji, @dotz0ver, @edwiniac, @gen740, @kAIto47802, @maheer14, @nabenabe0928, @nightcityblade, @not522, @roli-lpci, @ryota717, @sateeshkumarb, @sawa3030, @toroleapinc, @y0z, @yasumorishima

This is the release note of v4.7.0.

Highlights

Two New Multi-Objective Samplers Added to OptunaHub!

@hrntsm introduces two new multi-objective samplers—SPEA-II (Strength Pareto Evolutionary Algorithm 2) and HypE (Hypervolume Estimation Algorithm)—to OptunaHub. SPEA-II is an improved multi-objective evolutionary algorithm that differs from NSGA-II in its selection mechanism. HypE is a fast, hypervolume-based evolutionary algorithm designed for many-objective optimization problems. Please refer to the following pages for more details:

• SPEA-II: https://hub.optuna.org/samplers/speaii/
• HypE: https://hub.optuna.org/samplers/hype/

PedAnovaImportanceEvaluator Now Supports Local Hyperparameter Importance Computation

The target_quantile and region_quantile arguments have been introduced to PedAnovaImportanceEvaluator. This change allows you to investigate local hyperparameter importance rather than the global one with region_quantile < 1.0. See the original paper for the technical details.

Enhancements

• Introduce stacklevel-aware custom warnings (#6293)
• Cache distributions to skip consistency check (#6301)
• Add warnings when JournalStorage lock acquisition is delayed (#6361)
• Add support for local HPI in PED-ANOVA (#6362)

Bug Fixes

• Fix log PDF of discrete trunc log-norm distribution for TPESampler (#6258)
• Fix coefficient in PED-ANOVA (#6358)
• Fix GPSampler crash when default torch device is CUDA (#6397, thanks @Quant-Quasar!)

Documentation

• Add SECURITY.md (#6317)
• Add a note for future dev of exclusive HV (#6318)
• Update GPSampler documentation to include D-BE optimization details (#6347, thanks @Kaichi-Irie!)
• Revert PR #6354 to enable -W option on Sphinx build (#6373)

Examples

• Disable scheduled runs for PyTorch and visualization temporarily (optuna/optuna-examples#337)
• Fix skorch example: Replace unavailable OpenML MNIST (optuna/optuna-examples#338, thanks @sotagg!)
• Pin minio version to <=7.2.18 to fix CI & stop daily CI running (optuna/optuna-examples#339)
• fix spark example (optuna/optuna-examples#342, thanks @fritshermans!)
• Pin scikit-learn to < 1.6.0 for lightgbm (optuna/optuna-examples#343)
• Remove python 3.9 due to EOL Status (optuna/optuna-examples#344, thanks @ParagEkbote!)
• Add IPython as a dependency of fastai example (optuna/optuna-examples#347)

Tests

• Fix TC006 violation in tests/visualization_tests/test_utils.py (#6387, thanks @jiayusu!)

Code Fixes

• Replace .format() with f-string in _setup_studies (#6326, thanks @haitham404!)
• Update _upload.py for TYPE_CHECKING (#6327, thanks @satyarth7srivastava!)
• Replace .format() with f-string in progress_bar.py (#6328, thanks @Nayil97!)
• Use f-strings in optuna/samplers/_cmaes.py (#6331, thanks @swativdusane!)
• Replace .format() with f-string in _parallel_coordinate.py (#6333, thanks @satyarth7srivastava!)
• Refactor/fstring storage rdb (#6336, thanks @gadmin7!)
• Migrate to ruff from black/blackdoc/isort/flake8 (#6341)
• Replace .format with f-strings in optuna/importance/_base (#6342, thanks @VihaanMotwani!)
• updated _terminator_improvement.py for TYPE_CHECKING (#6343, thanks @satyarth7srivastava!)
• Replace .format with f-string in _param_importances.py (#6345, thanks @Harshadev-24!)
• Replace .format() with f-strings in several modules (#6348, thanks @varundevr!)
• Replace more .format() calls with f-strings (#6349, thanks @varundevr!)
• Replace .format with f-string in tutorial/20_recipes/004_cli.py (#6350, thanks @RektPunk!)
• Replace .format with f-string in optuna/study/_optimize.py (#6351, thanks @RektPunk!)
• Format optuna/ files with Ruff (#6352)
• Refactor: Use f-string in 001_rdb.py (#6356, thanks @sotagg!)
• Format tests/ and tutorials/ files with Ruff (#6360)
• Remove redundant _color_supported() check (#6363)
• Add StorageTestCase class in optuna.testing package (#6369)
• Change string formatting in optuna/pruners/_hyperband.py (#6370, thanks @eleannapapaio!)
• Refactor test_study.py to use f-string instead of .format() (#6372, thanks @nepersoned!)
• Fix mypy error for np.select (#6374)
• Change string formatting for _successive_halving.py (#6375, thanks @spenam!)
• Fix type annotations for optuna/trial/_frozen.py (#6377, thanks @spenam!)
• fix type annotations for optuna/study/study.py (#6378, thanks @spenam!)
• Fix type annotations for tests/study_tests/test_study.py (#6379, thanks @spenam!)
• Move type-only imports to TYPE_CHECKING in test_visualizations.py (#6380, thanks @Sip4818!)
• Move type-only imports to TYPE_CHECKING in _constrained_optimization.py (#6381, thanks @Sip4818!)
• Replace format with f-string (#6383, thanks @varundevr!)
• Move type-only imports to TYPE_CHECKING in _multi_objective.py (#6385, thanks @Sip4818!)
• Move FrozenTrial import under TYPE_CHECKING for _study_summary.py file (#6386, thanks @Sip4818!)
• Move type-check imports to TYPE_CHECKING in optuna/terminator/callback.py (#6388, thanks @Sip4818!)
• Using f-string instead of .format() (#6389, thanks @Lakshman142!)
• Use f-strings in optuna/_experimental.py (#6390, thanks @Rohan0497!)
• Fix invalid StorageTestCase scenarios involving trial state and values (#6391)
• Move type-hint import inside Type-Checking block in optuna\terminator\erroreval.py (#6395, thanks @Sip4818!)
• Move type-check imports to TYPE_CHECKING in optuna\terminator\improvement\emmr.py (#6396, thanks @Sip4818!)
• Move typing-only imports under TYPE_CHECKING in matplotlib/_slice.py (#6399, thanks @kapishyadav!)
• Use logger.warning instead of optuna_warn for lock-acquisition delay notifications (#6400)
• Update string formatting in optuna/samplers/_grid.py (#6401, thanks @kapishyadav!)
• Updating storages/_in_memory.py to use f-strings (#6404, thanks @jrings!)
• Move type-hint imports into type-checking block in optuna\terminator\improvement\evaluator.py (#6405, thanks @Sip4818!)
• Move type-hint imports into type-checking block in median_erroreval.py (#6408, thanks @Sip4818!)
• Replace .format() with f-string in _rank.py (#6409, thanks @jwalith!)
• Replace .format() with f-string in test_hyperband.py (#6411, thanks @Banjiola!)
• Replace .format() with f-string in _fixed.py (#6412, thanks @VedantMadane!)

Continuous Integration

• Fix for CI (optuna/optuna-integration#258)
• Add ipython as a dependency of fastai (optuna/optuna-integration#261)
• Remove FastAIV2PruningCallback from docs to resolve warning (optuna/optuna-integration#264)
• Dispose SQLAlchemy's engine in storage_tests/test_with_server.py (#6330)
• Dispose SQLAlchemy's engine in storage_tests/test_cached_storage.py (#6337)
• Dispose SQLAlchemy's engine in storage_tests/rdb_tests/test_storage.py (#6338)
• Support Python 3.14 (#6339)
• Temporarily disable -W option on Sphinx build (#6354)

Other

• Bump up version (optuna/optuna-integration#256)
• Bump up version number to 4.7.0 (optuna/optuna-integration#262)
• Bump up to the version number v4.7.0.dev (#6325)
• Update the news section on README (#6335)
• Remove Codecov usage (#6340)
• Remove formats.sh and tidy up CONTRIBUTING.md (#6353)
• Remove asv and the speed benchmark workflow (#6393)
• Bump up version to v4.7.0 (#6413)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@Alnusjaponica, @Banjiola, @Harshadev-24, @HideakiImamura, @Kaichi-Irie, @Lakshman142, @Nayil97, @ParagEkbote, @Quant-Quasar, @RektPunk, @Rohan0497, @Sip4818, @VedantMadane, @VihaanMotwani, @c-bata, @eleannapapaio, @fritshermans, @fusawa-yugo, @gadmin7, @gen740, @haitham404, @jiayusu, @jrings, @jwalith, @kAIto47802, @kapishyadav, @nabenabe0928, @nepersoned, @not522, @nzw0301, @satyarth7srivastava, @sawa3030, @sotagg, @spenam, @swativdusane, @toshihikoyanase, @varundevr, @y0z

This is the release note of v4.6.0.

Highlights

Optuna Dashboard LLM Integration

Optuna Dashboard is a web-based tool that helps you easily explore and visualize your Optuna optimization history. The latest release, v0.20.0, introduces LLM integration, enabling the natural language-based Trial filtering and automatic Plotly chart generation. Please refer to the release blog for more details.

Further Speed Enhancements for GPSampler

GPSampler becomes significantly faster owing to parallelized multi-start acquisition function optimization via PyTorch batching, and to optimized NumPy operations.

Full Support for Multi-objective and Constrained Optimization in AutoSampler

We have fully implemented sampler selection rules for multi-objective and constrained optimization in AutoSampler. For more details, please see our blog post, "AutoSampler: Full Support for Multi-Objective & Constrained Optimization."

Additions of Robust Bayesian Optimization Packages

Robust Bayesian optimization methods have been added to OptunaHub. Robust Bayesian optimization enables suggesting more robust parameters against input perturbations. This is especially helpful for Sim2Real transfer scenarios.

Breaking Changes

• Drop Python 3.8 & Support Python 3.13 (optuna/optuna-integration#253)
• Change TrialState.__repr__ and TrialState.__str__ (#6281, thanks @ktns!)
• Drop Python 3.8 (#6302)

Enhancements

• Use iterator for lazy evaluation in journal storage’s read_logs (#6144)
• Cache pair-wise distances to speed up GPSampler (#6244)
• Speed up LogEI implementation (#6248)
• Speed up EHVI by optimizing tensor operation order (#6257)
• Use the decremental approach in the hypervolume contribution calculation (#6264)
• Use cached trials in TPESampler's sample_relative (#6265)
• Remove find_or_raise_by_id in _set_trial_value_without_commit (#6266)
• Speed up GPSampler by Batching Acquisition Function Evaluations (#6268, thanks @Kaichi-Irie!)
• Use cached study direction and trial for _CachedStorage's get_best_trial (#6270)
• Add upsert in _set_trial_attr_without_commit for PostgreSQL (#6282, thanks @jaikumarm!)
• Add states argument to _read_trials_from_remote_storage (#6288)
• Use cached trials for intersection search space calculation (#6291)
• Replace np.linalg.inv with np.linalg.cholesky to speed up GPSampler for numpy>=2.0.0 (#6296)

Bug Fixes

• Skip trial validation on copy_study (#6249)
• Fix incremental update algorithm in _CachedStorage's _read_trials_from_remote_storage (#6310)
• Add safety guard for exhaustive search (#6321)

Documentation

• Add AutoSampler to the sampler comparison table in the API reference (#6260, thanks @Kaichi-Irie!)
• Update the GPSampler document to reflect support for constrained multi-objective optimization (#6262)
• Add a link to the metric TPE paper in the TPESampler document (#6263)
• Update announcement (#6285)
• Update the table of Samplers in docs (#6287, thanks @fusawa-yugo!)
• Fix the table of samplers in the docs (#6290)

Examples

• Add example of OpenTelemetry in Optuna Dashboard (optuna/optuna-examples#330)
• [hotfix] Fix transformers example by adding the version constraint on transformers (optuna/optuna-examples#332)
• Drop Python 3.8 (optuna/optuna-examples#334)
• Remove Version Constraint for Transformers (optuna/optuna-examples#335, thanks @ParagEkbote!)

Tests

• Add unit tests for batched L-BFGS-B (#6274, thanks @Kaichi-Irie!)

Code Fixes

• Update target version of black from Python 3.8 to 3.9 (optuna/optuna-integration#254)
• Move fit_kernel_params to GPRegressor (#6243)
• Modify TYPE_CHECKING in _brute_force.py (#6259, thanks @Kaichi-Irie!)
• Move SciPy to the lazy import section in _gp/scipy_blas_thread_patch.py (#6269, thanks @Kaichi-Irie!)
• Make the interface of batched_lbfgsb module compatible with scipy.optimize (#6273, thanks @Kaichi-Irie!)
• Fix type checking in optuna.study._frozen.py (#6275, thanks @GabrielRomaoG!)
• Move typing-only imports under TYPE_CHECKING in optuna.importance.__init__ (#6278, thanks @euangoodbrand!)
• Move typing-only imports under TYPE_CHECKING in FanovaImportanceEvaluator (#6279, thanks @euangoodbrand!)
• Move typing-only imports under TYPE_CHECKING in /study/_optimize.py (#6280, thanks @euangoodbrand!)
• Use TYPE_CHECKING in optuna/pruners/_nop.py (#6297, thanks @AddyM!)
• Use TYPE_CHECKING in optuna/samplers/_random.py (#6298, thanks @AddyM!)
• Speed up squared distance computation (#6300)
• Refactor emmr (#6304)
• Fix string format of optuna/distributions.py (#6306)
• Fix string format of tests/samplers_tests/tpe_tests/test_truncnorm.py (#6307)
• Update black target to Python 3.9 (#6308)
• Fix string format for optuna/study/study.py (#6309, thanks @unKnownNG!)
• Changed the old .format code to the new f string format in the test_journal.py (#6312, thanks @Zrahay!)
• Update string formatting in visualization/_pareto_front.py (#6314, thanks @dross20!)
• Use f-string in 001_first.py (#6315, thanks @satyarth7srivastava!)
• Use f-strings in _intermediate_values.py (#6316, thanks @nihalsiddiqui7!)
• Refactor .format to f-string in _percentile.py (#6323, thanks @Jongwan93!)

Continuous Integration

• Update sklearn.py to fix mypy checks (optuna/optuna-integration#249)
• Fix CI (optuna/optuna-integration#250)
• Fix fragile test_parallel_optimize_with_sleep (#6241)
• Fix type checking in GP for CI (#6276)
• Fix CI (#6284)
• Limit the blackdoc version (#6289)
• Migrate .coveragerc to pyproject.toml (#6292, thanks @ParagEkbote!)
• Explicitly close DB connections when discarding SQLAlchemy's Engine (#6303)

Other

• Bump up the version number to 4.6.0.dev (optuna/optuna-integration#245)
• Add __version__ to init (optuna/optuna-integration#247)
• Migrate .coveragerc to pyproject.toml (optuna/optuna-integration#252, thanks @ParagEkbote!)
• Bump up version (optuna/optuna-integration#255)
• Bump up version to v4.6.0.dev (#6252)
• Update NEWS section (#6319)
• Bump up to version number 4.6.0 (#6324)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@AddyM, @GabrielRomaoG, @Jongwan93, @Kaichi-Irie, @ParagEkbote, @Zrahay, @c-bata, @contramundum53, @dross20, @euangoodbrand, @fusawa-yugo, @gen740, @jaikumarm, @kAIto47802, @ktns, @nabenabe0928, @nihalsiddiqui7, @not522, @satyarth7srivastava, @sawa3030, @toshihikoyanase, @unKnownNG, @y0z

This is the release note of v4.5.0.

Highlights

GPSampler for constrained multi-objective optimization

GPSampler is now able to handle multiple objective and constraints simultaneously using the newly introduced constrained LogEHVI acquisition function.

The figures below show the difference between GPSampler (LogEHVI, unconstrained) vs GPSampler (constrained LogEHVI, new feature). The 3-dimensional version of the C2DTLZ2 benchmark problem we used is a problem where some areas of the Pareto front of the original DTLZ2 problem are made infeasible by constraints. Therefore, even if constraints are not taken into account, it is possible to obtain the Pareto front. Experimental results show that both LogEHVI and constrained LogEHVI can approximate the Pareto front, but the latter has significantly fewer infeasible solutions, demonstrating its efficiency.

Significant speedup of TPESampler

TPESampler is significantly (about 5x as listed in the table below) faster! It enables a larger number of trials in each study. The speedup was achieved through a series of enhancements in constant factors.

The following table shows the speed comparison of TPESampler between v4.4.0 and v4.5.0. The experiments were conducted using multivariate=True on a search space with 3 continuous parameters and 3 numerical discrete parameters. Each row shows the runtime for each number of objectives and each column shows each number of trials to be evaluated. Each runtime is shown along with the standard error over 3 random seeds. The numbers in parentheses represent the speedup factor in comparison to v4.4.0. For example, (5.1x) means the runtime of v4.5.0 is 5.1 times faster than that of v4.4.0.

Significant speedup of plot_hypervolume_history

plot_hypervolume_history is essential to assess the performance of multi-objective optimization, but it was unbearably slow when a large number of trials are evaluated on a many-objective (The number of objectives > 3) problem. v4.5.0 addressed this issue by incrementally updating the hypervolume instead of calculating each hypervolume from scratch.

The following figure shows the elapsed times of hypervolume history plot in Optuna v4.4.0 and v4.5.0 using a four-objective problem. The x-axis represents the number of trials and the y-axis represents the elapsed times for each setup. The blue and red lines are the results of v4.4.0 and v4.5.0, respectively.

CmaEsSampler now supports 1D search space

Up until Optuna v4.4, CmaEsSampler could not handle one-dimensional space and fell back to random search. Optuna v4.5 now allows the CMA-ES algorithm to be used for one-dimensional space.

The optunahub library is available on conda-forge

Now, you can install the optunahub library via conda-forge as follows.

conda install conda-forge::optunahub

New Features

• Add ConstrainedLogEHVI (#6198)
• Add support for constrained multi-objective optimization in GPSampler (#6224)
• Support 1D Search Spaces in CmaEsSampler (#6228)

Enhancements

• Move optuna._lightgbm_tuner module (optuna/optuna-integration#233, thanks @milkcoffeen!)
• Fix numerical issue warning on qehvi_candidates_func (optuna/optuna-integration#242, thanks @LukeGT!)
• Calculate hypervolume in HSSP using sum of contributions (#6130)
• Use hypervolume difference as upperbound of contribs in HSSP (#6131)
• Refactor tell_with_warning to avoid unnecessary get_trial call (#6133)
• Print fully qualified name of experimental function by default (#6162, thanks @ktns!)
• Include scipy-stubs in the type-check dependencies (#6174, thanks @jorenham!)
• Warn when GPSampler falls back to RandomSampler (#6179, thanks @sisird864!)
• Handle slowdown of GPSampler due to L-BFGS in SciPy v1.15 (#6191)
• Use the Newton method instead of bisect in ndtri_exp (#6194)
• Speed up erf for TPESampler (#6200)
• Avoid duplications in _log_gauss_mass evaluations (#6202)
• Remove unnecessary NumPy usage (#6215)
• Use subset comparator to judge if trials are included in search space (#6218)
• Speed up log pdf in _BatchedTruncNormDistributions by vectorization (#6220)
• Speed up WFG by skipping is_pareto_front and using simple Python loops (#6223)
• Vectorize ndtri_exp (#6229)
• Speed up plot_hypervolume_history (#6232)
• Speed up HSSP 4D+ by using a decremental approach (#6234)
• Use lru_cache to skip HSSP (#6240, thanks @fusawa-yugo!)
• Add hypervolume computation for a zero size array (#6245)

Bug Fixes

• Fix: Resolve PG17 incompatibility for ENUMS in CASE statements (#6099, thanks @vcovo!)
• Fix ill-combination of journal and gRPC (#6175)
• Fix a bug in constrained GPSampler (#6181)
• Fix TPESampler with multivariate and constant_liar (#6189)

Installation

• Remove version constraint for torch with Python 3.13 (#6233)

Documentation

• Add missing spaces in error message about inconsistent intermediate values (optuna/optuna-integration#239, thanks @Greesb!)
• Improve parallelization document (#6123)
• Add FAQ for case sensitivity problem with MySQL (#6127, thanks @fusawa-yugo!)
• Add an FAQ entry about specifying optimization parameters (#6157)
• Update link to survey (#6169)
• Add introduction of OptunaHub in docs (#6171, thanks @fusawa-yugo!)
• Add GPSampler as a sampler that supports constraints (#6176, thanks @1kastner!)
• Remove optuna-fast-fanova references from documentation (#6178)
• Fix stale docs in GP-related modules (#6184)
• Embed link to OptunaHub in documentation (#6192, thanks @fusawa-yugo!)
• Update README.md (#6222, thanks @muhammadibrahim313!)
• Add a note about unrelated changes in PR (#6226)
• Document the default evaluator in Optuna Dashboard (#6238)

Examples

• Add Spark example using ask-and-tell interface (optuna/optuna-examples#328, thanks @dhyeyinf!)

Tests

• Fix test_log_completed_trial_skip_storage_access (#6208)

Code Fixes

• Clean up GP-related docs (#6125)
• Refactor return style #6136 (#6151, thanks @unKnownNG!)
• Refactor KernelParamsTensor towards cleaner GP-related modules (#6152)
• Rename KernelParamsTensor to GPRegressor (#6153)
• Refactor returns in v3.0.0.d.py (#6154, thanks @dross20!)
• Refactor acquisition function minimally (#6166)
• Implement Type-Checking for optuna/_imports.py (#6167, thanks @AdrianStrymer!)
• Fix type checking in optuna.artifacts._download.py (#6177, thanks @dross20!)
• Integrate is_categorical to search space (#6182)
• Fix type checking in optuna.artifacts._list_artifact_meta.py (#6187, thanks @dross20!)
• Introduce the independent sampling warning template (#6188)
• Use warning template for independent sampling in GPSampler (#6195)
• Refactor SearchSpace in GP (#6197)
• Refactor _truncnorm (#6201)
• Use TYPE_CHECKING in optuna/_gp/acqf.py to avoid circular imports (#6204, thanks @CarvedCoder!)
• Use TYPE_CHECKING in optuna/_gp/optim_mixed.py to avoid circular imports (#6205, thanks @Subodh-12!)
• Flip the sign of constraints in GPSampler (#6213)
• Implement NSGA-III using BaseGASampler (#6219)
• Replace torch.newaxis with None for old PyTorch (#6237)

Continuous Integration

• Fix CI (optuna/optuna-integration#238)
• Add sklearn version constraint (optuna/optuna-integration#243)
• Fix README for blackdoc==0.3.10 (#6150)
• Fix CI (#6161)
• Add pytest-xdist to speed up the CI (#6170)
• Delete test_get_timeline_plot_with_killed_running_trials (#6210)
• Fix fragile test test_experimental (#6211)
• Mark fragile test as xfail (#6217)
• Add unit tests for constrained multi-objective GPSampler (#6235)
• Fix CI (#6246)

Other

• Update the example list in the README (optuna/optuna-integration#234, thanks @ParagEkbote!)
• Bump up version number to 4.5.0.dev (optuna/optuna-integration#237)
• Bump up the version number to 4.5.0 (optuna/optuna-integration#244)
• Bump up version number to v4.5.0.dev (#6149)
• Update News section in README (#6159)
• Bump up version to v4.5.0 (#6251)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@1kastner, @AdrianStrymer, @CarvedCoder, @Greesb, @HideakiImamura, @LukeGT, @ParagEkbote, @Subodh-12, @c-bata, @contramundum53, @dhyeyinf, @dross20, @fusawa-yugo, @gen740, @hvy, @jorenham, @kAIto47802, @ktns, @milkcoffeen, @muhammadibrahim313, @nabenabe0928, @not522, @nzw0301, @sawa3030, @sisird864, @toshihikoyanase, @unKnownNG, @vcovo, @y0z

This is the release note of v4.4.0.

Highlights

In addition to new features, bug fixes, and improvements in documentation and testing, version 4.4 introduces a new tool called the Optuna MCP Server.

Optuna MCP Server

The Optuna MCP server can be accessed by any MCP client via uv — for instance, with Claude Desktop, simply add the following configuration to your MCP server settings file. Of course, other LLM clients like VSCode or Cline can also be used similarly. You can also access it via Docker. If you want to persist the results, you can use the — storage option. For details, please refer to the repository.

{
  "mcpServers": {
    … (Other MCP Servers' settings)
    "Optuna": {
      "command": "uvx",
      "args": [
        "optuna-mcp"
      ]
    }
  }
}

Gaussian Process-Based Multi-objective Optimization

Optuna’s GPSampler, introduced in version 3.6, offers superior speed and performance compared to existing Bayesian optimization frameworks, particularly when handling objective functions with discrete variables. In Optuna v4.4, we have extended this GPSampler to support multi-objective optimization problems. The applications of multi-objective optimization are broad, and the new multi-objective capabilities introduced in this GPSampler are expected to find applications in fields such as material design, experimental design problems, and high-cost hyperparameter optimization.

GPSampler can be easily integrated into your program and performs well against the existing BoTorchSampler. We encourage you to try it out with your multi-objective optimization problems.

sampler = optuna.samplers.GPSampler()
study = optuna.create_study(directions=["minimize", "minimize"], sampler=sampler)

New Features in OptunaHub

During the development period of Optuna v4.4, several new features were also introduced to OptunaHub, the feature-sharing platform for Optuna:

• A sampler utilizing Google Vizier is now newly available.
• The CMA-ES-based sampler with restart strategy, which was previously part of the Optuna core, has been migrated to OptunaHub, making it simpler and more user-friendly.
• A benchmark problem solving aircraft design as a black-box optimization task has been added, further enhancing the convenience of algorithm development using OptunaHub.
• A visualization feature has also been added, allowing users to see how the acquisition function of the default TPESampler evolves as trials progress.
• A novel mutation operation has been added to the MOEA/D evolutionary computation algorithm for multi-objective optimization.

Breaking Changes

• Update consider_prior Behavior and Remove Support for False (#6007)
• Remove restart_strategy and inc_popsize to simplify CmaEsSampler (#6025)
• Make all arguments of TPESampler keyword-only (#6041)

New Features

• Add a module to preprocess solutions for hypervolume improvement calculation (#6039)
• Add AcquisitionFuncParams for LogEHVI (#6052)
• Support Multi-Objective Optimization GPSampler (#6069)
• Add n_recent_trials to plot_timeline (#6110, thanks @msdsm!)

Enhancements

• Adapt TYPE_CHECKING of samplers/_gp/sampler.py (#6059)
• Avoid deepcopy in _tell_with_warning (#6079)
• Add _compute_3d for hypervolume computation (#6112, thanks @shmurai!)
• Improve performance of plot_hypervolume_history (#6115, thanks @shmurai!)
• add deprecated/removed version specification to calls of convert_positional_args (#6117, thanks @shmurai!)
• Optimize Study.best_trial performance by avoiding unnecessary deep copy (#6119, thanks @msdsm!)
• Refactor and speed up HV3D (#6124)
• Add assume_pareto for hv calculation in _calculate_weights_below_for_multi_objective (#6129)

Bug Fixes

• Update vsbx (#6033, thanks @hrntsm!)
• Fix request.values in OptunaStorageProxyService (#6044, thanks @hitsgub!)
• Fix a bug in distributed optimization using NSGA-II/III (#6066, thanks @leevers!)
• Fix: fetch all trials in BruteForceSampler for HyperbandPruner (#6107)

Documentation

• Add Pycma Example (optuna/optuna-integration#226, thanks @ParagEkbote!)
• Add SHAP Example (optuna/optuna-integration#227, thanks @ParagEkbote!)
• Document Behavior of optuna.pruners.MedianPruner and optuna.pruners.PatientPruner (#6055, thanks @ParagEkbote!)
• Change the link of tutorial docs of optunahub (#6063, thanks @fusawa-yugo!)
• Update the documentation string of GPSampler (#6081)
• Add a warning about the combination of gRPC Proxy and Journal Storage (#6097)
• Cosmetic fix to the terminator documents (#6100)
• Note in docstring that heartbeat mechanism is experimental (#6111, thanks @lan496!)
• Update docstrings of _get_best_trial to follow coding conventions (#6122)

Examples

• Add Example for Comet (optuna/optuna-examples#305, thanks @ParagEkbote!)
• Adding an OpenML example (optuna/optuna-examples#310, thanks @SubhadityaMukherjee!)
• Add workflow dispatch (optuna/optuna-examples#311)
• Update PyTorch Checkpoint Example using tempfile (optuna/optuna-examples#313, thanks @ParagEkbote!)
• [hotfix] Fix Dask-ML example by adding the version constraint on numpy (optuna/optuna-examples#315)
• Setup Pre-Commit (optuna/optuna-examples#316, thanks @ParagEkbote!)
• Remove Python 3.9 from haiku CI (optuna/optuna-examples#318)
• Add a transformers example (optuna/optuna-examples#322, thanks @ParagEkbote!)
• Add transformer item to RAEDME.md (optuna/optuna-examples#323)
• Remove version constraints of tensorflow and numpy (optuna/optuna-examples#324)
• Update pre-commit hooks (optuna/optuna-examples#326, thanks @lan496!)
• Add preferential optimization picture (optuna/optuna-examples#327, thanks @milkcoffeen!)

Tests

• Add float precision tests for storages (#6040)
• Refactor test_base_gasampler.py (#6104)
• chore: run tests for importance only with in-memory (#6109)
• Improve test cases for n_recent_trials of plot_timeline (follow-up #6110) (#6116)
• Performance optimization for test_study.py by removing redundancy (#6120)

Code Fixes

• Optional mypy check (#6028)
• Update Type-Checking for optuna/_experimental.py (#6045, thanks @ParagEkbote!)
• Update Type-Checking for optuna/importance/_base.py (#6046, thanks @ParagEkbote!)
• Update Type-Checking for optuna/_convert_positional_args.py (#6050, thanks @ParagEkbote!)
• Update Type-Checking for optuna/_deprecated.py (#6051, thanks @ParagEkbote!)
• Update Type-Checking for optuna/_gp/gp.py (#6053, thanks @ParagEkbote!)
• Add validate eta in sbx (#6056, thanks @hrntsm!)
• Remove CmaEsAttrKeys and _attr_keys for Simplification (#6068)
• Replace np.isnan with math.isnan (#6080)
• Refactor warning handling of _tell_with_warning (#6082)
• Implement Type-Checking for optuna/distributions.py (#6086, thanks @AdrianStrymer!)
• Update TYPE_CHECKING for optuna/_gp/gp.py (#6090, thanks @Samarthi!)
• Support mypy 1.16.0 (#6102)
• Emit ExperimentalWarning if heartbeat is enabled (#6106, thanks @lan496!)
• Simplify tuple return in optuna/visualization/_terminator_improvement.py (#6139, thanks @Prashantdhaka23!)
• Refactor return standardization in optim_mixed.py (#6140, thanks @Ajay-Satish-01!)
• Simplify tuple return in test_trial.py (#6141, thanks @saishreyakumar!)
• Refactor return statement style in optuna/storages/_rdb/models.py for consistency among the codebase (#6143, thanks @Shubham05122002!)

Continuous Integration

• Add type ignore in wandb (optuna/optuna-integration#228)
• Fix to prevent daily checks-optional CI on the fork repositories (#6103)
• Fix CI (#6137)
• [hotfix] Add version constraint on blackdoc (#6145)

Other

• Bump up version number to v4.4.0.dev (optuna/optuna-integration#220)
• Add pre commit config (optuna/optuna-integration#231, thanks @milkcoffeen!)
• Bump up version number to 4.4.0 (optuna/optuna-integration#236)
• Bump up version to 4.4.0dev (#6038)
• Update the news section in README.md (#6049)
• Fix README for v5 roadmap (#6094)
• Update pre-commit hooks (#6108, thanks @lan496!)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@AdrianStrymer, @Ajay-Satish-01, @Alnusjaponica, @Copilot, @HideakiImamura, @ParagEkbote, @Prashantdhaka23, @Samarthi, @Shubham05122002, @SubhadityaMukherjee, @c-bata, @contramundum53, @copilot-pull-request-reviewer[bot], @fusawa-yugo, @gen740, @himkt, @hitsgub, @hrntsm, @kAIto47802, @lan496, @leevers, @milkcoffeen, @msdsm, @nabenabe0928, @not522, @nzw0301, @saishreyakumar, @sawa3030, @shmurai, @toshihikoyanase, @y0z

This is the release note of v4.3.0.

Highlights

This has various bug fixes and improvements to the documentation and more.

Breaking Changes

• [fix] lgbm 4.6.0 compatibility (optuna/optuna-integration#207, thanks @ffineis!)

Enhancements

• Accept custom objective in LightGBMTuner (optuna/optuna-integration#203, thanks @sawa3030!)
• Improve time complexity of IntersectionSearchSpace (#5982, thanks @GittyHarsha!)
• Add _prev_waiting_trial_number in InMemoryStorage to improve the efficiency of _pop_waiting_trial_id (#5993, thanks @sawa3030!)
• Add arguments of versions to convert_positional_args (#6009, thanks @fusawa-yugo!)
• Add wait_server_ready method in GrpcStorageProxy (#6010, thanks @hitsgub!)
• Remove warning messages for Matplotlib-based plot_contour and plot_rank (#6011)
• Fix type checking in optuna._callbacks.py (#6030)
• Enhance SBXCrossover (#6008, thanks @hrntsm!)

Bug Fixes

• Convert storage into InMemoryStorage before copying to the local (optuna/optuna-integration#213)
• Fix contour plot of matplotlib (#5892, thanks @fusawa-yugo!)
• Fix threading lock logic (#5922)
• Use _LazyImport for grpcio package (#5954)
• Prevent Lock Blocking by Adding Timeout to JournalStorage (#5971, thanks @sawa3030!)
• Fix a minor bug in GPSampler for objective that returns inf (#5995)
• Fix a bug that a gRPC server doesn't work with JournalStorage (#6004, thanks @fusawa-yugo!)
• Fix _pop_waiting_trial_id for finished trial (#6012)
• Resolve the issue where BruteForceSampler fails to suggest all combinations (#5893)

Documentation

• Follow recent changes in optuna/optuna's document sphinx config (optuna/optuna-integration#197)
• Fix links to external modules (optuna/optuna-integration#198)
• Update CONTRIBUTING.md (optuna/optuna-integration#200, thanks @sawa3030!)
• Update comment in .readthedocs.yml (#5976)
• Add comments on the reproducibility of HyperBandPruner (#6018)

Examples

• [hotfix] Add the version constraint on dask (optuna/optuna-examples#296)
• [hotfix] Add the version constraint on dask for dask-ml (optuna/optuna-examples#297)
• Extends execution span of hiplot and sklearn (optuna/optuna-examples#298, thanks @fusawa-yugo!)
• Apply black to fix CI (optuna/optuna-examples#300)
• Bump up to 3.12 for CI (optuna/optuna-examples#301)
• [hotfix] Add the version constraint on lightgbm (optuna/optuna-examples#302)
• Fix Skorch Example (optuna/optuna-examples#303, thanks @ParagEkbote!)
• Add version constraint for tensorflow-related CI (optuna/optuna-examples#304)
• Temporarily skip Python 3.9 in fastai example (optuna/optuna-examples#308)
• Run the skorch example in the CI (optuna/optuna-examples#309)
• Fix fastai Example (optuna/optuna-examples#312)

Tests

• Use JournalStorage in test_cli.py (#5990, thanks @sawa3030!)

Code Fixes

• Add BaseGASampler (#5864)
• Fix comments in pyproject.toml (#5972)
• Remove FirstTrialOnlyRandomSampler (#5973, thanks @mehakmander11!)
• Remove _check_and_set_param_distribution (#5975, thanks @siddydutta!)
• Remove testing/distributions.py (#5977, thanks @mehakmander11!)
• Remove _StudyInfo's param_distribution in _cached_storage.py (#5978, thanks @tarunprabhu11!)
• Introduce UpdateFinishedTrialError to raise an error when attempting to modify a finished trial (#6001, thanks @sawa3030!)
• Deprecate consider_prior in TPESampler (#6005, thanks @sawa3030!)
• Improve Code Readability by Following PEP8 Standards (#6006, thanks @sawa3030!)
• Made error message for create_study's direction easier to understand optuna.study (#6021, thanks @sinano1107!)

Continuous Integration

• Hotfix ci (optuna/optuna-integration#199)
• Add flake8 in CI (optuna/optuna-integration#201, thanks @sawa3030!)
• Remove test cases that uses UnsupportedDistribution (optuna/optuna-integration#208)
• Fix a mypy error when using numpy>=2.2.4 (optuna/optuna-integration#212)
• Fix a bug of lightgbm tuner for Python 3.8 users (optuna/optuna-integration#214)
• Add a version constraint on xgboost (optuna/optuna-integration#217)
• Run (optuna/optuna-integration#218)
• Ensure gRPC server readiness before proceeding to prevent test failures (#5938, thanks @sawa3030!)
• Apply black to fix CI (#5952)
• Add workflow_dispatch trigger to all the CI (#6019)
• Fix CI (#6026)

Other

• Bump up version number to 4.3.0.dev (optuna/optuna-integration#192)
• Bump the version up to v4.2.1 (optuna/optuna-integration#195)
• Set repository url (optuna/optuna-integration#196, thanks @ktns!)
• Bump up version number to v4.3.0 (optuna/optuna-integration#221)
• Bump the version up to v4.3.0.dev (#5927)
• Add the article to the news section (#5928)
• Update news section for 4.2.0 release (#5934)
• Update News (#5936)
• Update README with the new blog entry (#5980)
• Add GPSampler blog to the announcement (#6014)
• Add grpc blog to README (#6020)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@Alnusjaponica, @GittyHarsha, @HideakiImamura, @ParagEkbote, @c-bata, @contramundum53, @ffineis, @fusawa-yugo, @gen740, @hitsgub, @hrntsm, @kAIto47802, @ktns, @mehakmander11, @nabenabe0928, @not522, @nzw0301, @porink0424, @sawa3030, @siddydutta, @sinano1107, @tarunprabhu11, @toshihikoyanase, @y0z

This is the release note of v4.2.1. This release includes a bug fix addressing an issue where Optuna was unable to import if an older version of the grpcio package was installed.

Bug

• [backport] Use _LazyImport for grpcio package (#5965)

Other

• Bump up version number to v4.2.1 (#5964)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.
@c-bata @HideakiImamura @nabenabe0928

This is the release note of v4.2.0. In conjunction with the Optuna release, OptunaHub 0.2.0 is released. Please refer to the release note of OptunaHub 0.2.0 for more details.

Highlights of this release include:

• 🚀gRPC Storage Proxy for Scalable Hyperparameter Optimization
• 🤖 SMAC3: Support for New State-of-the-art Optimization Algorithm by AutoML.org (@automl)
• 📁 OptunaHub Now Supports Benchmark Functions
• 🧑‍💻 Gaussian Process-Based Bayesian Optimization with Inequality Constraints
• 🧑‍💻 c-TPE: Support Constrained TPESampler

Highlights

gRPC Storage Proxy for Scalable Hyperparameter Optimization

The gRPC storage proxy is a feature designed to support large-scale distributed optimization. As shown in the diagram below, gRPC storage proxy sits between the optimization workers and the database server, proxying the calls of Optuna’s storage APIs.

<img width="1241" alt="grpc-proxy" src="https://github.com/user-attachments/assets/00220bc6-75da-4ee3-b9ff-a2b3440ac207" />

In large-scale distributed optimization settings where hundreds to thousands of workers are operating, placing a gRPC storage proxy for every few tens can significantly reduce the load on the RDB server which would otherwise be a single point of failure. The gRPC storage proxy enables sharing the cache about Optuna studies and trials, which can further mitigate load. Please refer to the official documentation for further details on how to utilize gRPC storage proxy.

SMAC3: Random Forest-Based Bayesian Optimization Developed by AutoML.org

SMAC3 is a hyperparameter optimization framework developed by AutoML.org, one of the most influential AutoML research groups. The Optuna-compatible SMAC3 sampler is now available thanks to the contribution to OptunaHub by Difan Deng (@dengdifan), one of the core members of AutoML.org. We can now use the method widely used in AutoML research and real-world applications from Optuna.

# pip install optunahub smac
import optuna
import optunahub
from optuna.distributions import FloatDistribution

def objective(trial: optuna.Trial) -> float:
    x = trial.suggest_float("x", -10, 10)
    y = trial.suggest_float("y", -10, 10)
    return x**2 + y**2

smac_mod = optunahub.load_module("samplers/smac_sampler")
n_trials = 100
sampler = smac_mod.SMACSampler(
    {"x": FloatDistribution(-10, 10), "y": FloatDistribution(-10, 10)},
    n_trials=n_trials,
)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=n_trials)

Please refer to https://hub.optuna.org/samplers/smac_sampler/ for more details.

OptunaHub Now Supports Benchmark Functions

Benchmarking the performance of optimization algorithms is an essential process indispensable to the research and development of algorithms. The newly added OptunaHub Benchmarks in the latest version v0.2.0 of optunahub is a new feature for Optuna users to conduct benchmarks conveniently.

# pip install optunahub>=4.2.0 scipy torch
import optuna
import optunahub

bbob_mod = optunahub.load_module("benchmarks/bbob")
smac_mod = optunahub.load_module("samplers/smac_sampler")
sphere2d = bbob_mod.Problem(function_id=1, dimension=2)

n_trials = 100
studies = []
for study_name, sampler in [
    ("random", optuna.samplers.RandomSampler(seed=1)),
    ("tpe", optuna.samplers.TPESampler(seed=1)),
    ("cmaes", optuna.samplers.CmaEsSampler(seed=1)),
    ("smac", smac_mod.SMACSampler(sphere2d.search_space, n_trials, seed=1)),
]:
    study = optuna.create_study(directions=sphere2d.directions,
        sampler=sampler, study_name=study_name)
    study.optimize(sphere2d, n_trials=n_trials)
    studies.append(study)

optuna.visualization.plot_optimization_history(studies).show()

In the above sample code, we compare and display the performance of the four kinds of samplers using a two-dimensional Sphere function, which is part of a group of benchmark functions widely used in the black-box optimization research community known as Blackbox Optimization Benchmarking (BBOB).

<img width="1250" alt="bbob" src="https://github.com/user-attachments/assets/89b09090-dd37-4679-9fa8-af987f138dcc" />

Gaussian Process-Based Bayesian Optimization with Inequality Constraints

We worked on its extension and adapted GPSampler to constrained optimization in Optuna v4.2.0 since Gaussian process-based Bayesian optimization is a very popular method in various research fields such as aircraft engineering and materials science. We show the basic usage below.

# pip install optuna>=4.2.0 scipy torch
import numpy as np
import optuna

def objective(trial: optuna.Trial) -> float:
    x = trial.suggest_float("x", 0.0, 2 * np.pi)
    y = trial.suggest_float("y", 0.0, 2 * np.pi)
    c = float(np.sin(x) * np.sin(y) + 0.95)
    trial.set_user_attr("c", c)
    return float(np.sin(x) + y)

def constraints(trial: optuna.trial.FrozenTrial) -> tuple[float]:
    return (trial.user_attrs["c"],)

sampler = optuna.samplers.GPSampler(constraints_func=constraints)
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=50)

Please try out GPSampler for constrained optimization especially when only a small number of trials are available!

c-TPE: Support Constrained TPESampler

Although Optuna has supported constrained optimization for TPESampler, which is the default Optuna sampler, since v3.0.0, its algorithm design and performance comparison have not been verified academically. OptunaHub now supports c-TPE, which is another constrained optimization method for TPESampler. Importantly, the algorithm design and its performance comparison are publicly reviewed to be accepted to IJCAI, a top-tier AI international conference. Please refer to https://hub.optuna.org/samplers/ctpe/ for details.

New Features

• Enable GPSampler to support constraint functions (#5715)
• Update output format options in CLI to include the value choice (#5822, thanks @iamarunbrahma!)
• Add gRPC storage proxy server and client (#5852)

Enhancements

• Introduce client-side cache in GrpcStorageProxy (#5872)

Bug Fixes

• Fix CI (https://github.com/optuna/optuna-integration/pull/185)
• Fix ticks in Matplotlib contour plot (#5778, thanks @sulan!)
• Adding check in cli.py to handle an empty database (#5828, thanks @willdavidson05!)
• Avoid the input validation fail in Wilcoxon signed ranked test for Scipy 1.15 (#5912)
• Fix the default sampler of load_study function (#5924)

Documentation

• Update OptunaHub example in README (#5763)
• Update distributions.rst to list deprecated distribution classes (#5764)
• Remove deprecation comment for step in IntLogUniformDistribution (#5767)
• Update requirements for OptunaHub in README (#5768)
• Use inline code rather than italic for step (#5769)
• Add notes to ask_and_tell tutorial - batch optimization recommendations (#5817, thanks @SimonPop!)
• Fix the explanation of returned values of get_trial_params (#5820)
• Introduce sphinx-notfound-page for better 404 page (#5898)
• Follow-up #5872: Update the docstring of run_grpc_proxy_server (#5914)
• Modify doc-string of gRPC-related modules (#5916)

Examples

• Adapt docker recipes to Python 3.11 (https://github.com/optuna/optuna-examples/pull/292)
• Add version constraint for wandb (https://github.com/optuna/optuna-examples/pull/293)

Tests

• Add unit tests for retry_history method in RetryFailedTrialCallback (#5865, thanks @iamarunbrahma!)
• Add tests for value format in CLI (#5866)
• Import grpc lazy to fix the CI (#5878)

Code Fixes

• Fix annotations for distributions.py (#5755, thanks @KannanShilen!)
• Simplify type annotations for tests/visualization_tests/test_pareto_front.py (#5756, thanks @boringbyte!)
• Fix type annotations for test_hypervolume_history.py (#5760, thanks @boringbyte!)
• Simplify type annotations for tests/test_cli.py (#5765, thanks @boringbyte!)
• Simplify type annotations for tests/test_distributions.py (#5773, thanks @boringbyte!)
• Simplify type annotations for tests/samplers_tests/test_qmc.py (#5775, thanks @boringbyte!)
• Simplify type annotations for tests/sampler_tests/tpe_tests/test_sampler.py (#5779, thanks @boringbyte!)
• Simplify type annotations for tests/samplers_tests/tpe_tests/test_multi_objective_sampler.py (#5781, thanks @boringbyte!)
• Simplify type annotations for tests/samplers_tests/tpe_tests/test_parzen_estimator.py (#5782, thanks @boringbyte!)
• Simplify type annotations for tests/storage_tests/journal_tests/test_journal.py (#5783, thanks @boringbyte!)
• Simplify type annotations for tests/storage_tests/rdb_tests/create_db.py (#5784, thanks @boringbyte!)
• Simplify type annotations for tests/storage_tests/rdb_tests/ (#5785, thanks @boringbyte!)
• Simplify type annotations for tests/test_deprecated.py (#5786, thanks @boringbyte!)
• Simplify type annotations for tests/test_convert_positional_args.py (#5787, thanks @boringbyte!)
• Simplify type annotations for tests/importance_tests/test_init.py (#5790, thanks @boringbyte!)
• Simplify type annotations for tests/storages_tests/test_storages.py (#5791, thanks @boringbyte!)
• Simplify type annotations for tests/storages_tests/test_heartbeat.py (#5792, thanks @boringbyte!)
• Simplify type annotations optuna/cli.py (#5793, thanks @willdavidson05!)
• Simplify type annotations for tests/trial_tests/test_frozen.py (#5794, thanks @boringbyte!)
• Simplify type annotations for tests/trial_tests/test_trial.py (#5795, thanks @boringbyte!)
• Simplify type annotations for tests/trial_tests/test_trials.py (#5796, thanks @boringbyte!)
• Refactor some funcs in NSGA-III (#5798)
• Simplify type annotations optuna/_transform.py (#5799, thanks @JLX0!)
• Make the assertion messages in test_trial.py readable (#5800)
• Simplify type annotations for tests/pruners_tests/test_hyperband.py (#5801, thanks @boringbyte!)
• Simplify type annotations for tests/pruners_tests/test_median.py (#5802, thanks @boringbyte!)
• Simplify type annotations for tests/pruners_tests/test_patient.py (#5803, thanks @boringbyte!)
• Use study.ask() in tests instead of create_new_trial (#5807, thanks @unKnownNG!)
• Simplify type annotations for tests/pruners_tests/test_percentile.py (#5808, thanks @boringbyte!)
• Simplify type annotations for tests/pruners_tests/test_successive_halving.py (#5809, thanks @boringbyte!)
• Simplify type annotations for tests/study_tests/test_optimize.py (#5810, thanks @boringbyte!)
• Simplify type annotations for tests/hypervolume_tests/test_hssp.py (#5812, thanks @boringbyte!)
• Refactor the MOTPE split (#5813)
• Simplify type annotations for optuna/_callbacks.py (#5818, thanks @boringbyte!)
• Simplify type annotations for optuna/samplers/_random.py (#5819, thanks @boringbyte!)
• Simplify type annotations for optuna/samplers/_gp/sampler.py (#5823, thanks @boringbyte!)
• Simplify type annotations for optuna/samplers/nsgaii/_crossovers/_sbx.py (#5824, thanks @boringbyte!)
• Simplify type annotations for optuna/samplers/nsgaii/_crossovers/_spx.py (#5825, thanks @boringbyte!)
• Simplify type annotations for optuna/samplers/nsgaii/_crossovers/_undx.py (#5826, thanks @boringbyte!)
• Simplify type annotations for optuna/samplers/nsgaii/_crossovers/_vsbx.py (#5827, thanks @boringbyte!)
• Simplify type annotations for optuna/samplers/nsgaii/_crossover.py (#5831, thanks @boringbyte!)
• Simplify type annotations for optuna/samplers/testing/threading.py (#5832, thanks @boringbyte!)
• Simplify type annotations for optuna/pruners/_patient.py (#5833, thanks @boringbyte!)
• Use study.ask() in tests/pruners_tests/test_percentile.py (#5834, thanks @fusawa-yugo!)
• Simplify type annotations for optuna/search_space/group_decomposed.py (#5836, thanks @boringbyte!)
• Simplify type annotations for optuna/search_space/intersection.py (#5837, thanks @boringbyte!)
• Simplify type annotations for optuna/storages/journal/_base.py (#5838, thanks @boringbyte!)
• Simplify type annotations for optuna/storages/journal/_redis.py (#5840, thanks @boringbyte!)
• Simplify type annotations for optuna/storages/journal/_storage.py (#5841, thanks @boringbyte!)
• Simplify type annotations for optuna/study/_dataframe.py (#5842, thanks @boringbyte!)
• Fix logger.warn() deprecation warning in GP module (#5843, thanks @iamarunbrahma!)
• Simplify type annotations for optuna/study/_optimize.py (#5844, thanks @boringbyte!)
• Simplify type annotations for optuna/study/_tell.py (#5845, thanks @boringbyte!)
• Fix mypy errors due to numpy 2.2.0 (#5848)
• Simplify type annotations for optuna/visualization/matplotlib/_contour.py (#5851, thanks @boringbyte!)
• Refactor the fix for MyPy errors due to NumPy v2.2.0 (#5853)
• Simplify type annotations for optuna/visualization/matplotlib/_parallel_coordinate.py (#5854, thanks @boringbyte!)
• Simplify type annotations for optuna/visualization/matplotlib/_param_importances.py (#5855, thanks @boringbyte!)
• Use study.ask() in tests/pruners_tests/test_successive_halving.py (#5856, thanks @willdavidson05!)
• Simplify type annotations for optuna/visualization/matplotlib/_pareto_front.py (#5857, thanks @boringbyte!)
• Simplify type annotations for optuna/visualization/matplotlib/_rank.py (#5858, thanks @boringbyte!)
• Simplify type annotations for optuna/visualization/matplotlib/_slice.py (#5859, thanks @boringbyte!)
• Refactor plot contour (#5867)
• Refactor MOTPE weighting (#5871)
• Simplify type annotations for optuna/visualization/_utils.py (#5876, thanks @boringbyte!)
• Simplify type annotations for optuna/visualization/_contour.py (#5877, thanks @boringbyte!)
• Simplify type annotations for optuna/_gp/gp.py (#5879, thanks @boringbyte!)
• Simplify type annotations for optuna/study/_tell.py (#5880, thanks @boringbyte!)
• Simplify type annotations for optuna/storages/_heartbeat.py (#5882, thanks @boringbyte!)
• Simplify type annotations for optuna/storages/_rdb/storage.py (#5883, thanks @boringbyte!)
• Simplify type annotations for optuna/storages/_rdb/alembic/versions/v3.0.0.a.py (#5884, thanks @boringbyte!)
• Simplify type annotations for optuna/storages/_rdb/alembic/versions/v3.0.0.c.py (#5885, thanks @boringbyte!)
• Simplify type annotations for optuna/storages/_rdb/alembic/versions/v3.0.0.d.py (#5886, thanks @boringbyte!)
• Simplify type annotations for optuna/_deprecated.py (#5887, thanks @boringbyte!)
• Simplify type annotations for optuna/_experimental.py (#5888, thanks @boringbyte!)
• Simplify type annotations for optuna/_imports.py (#5889, thanks @boringbyte!)
• Simplify type annotations for optuna/visualization/_slice.py (#5894, thanks @boringbyte!)
• Simplify type annotations for optuna/visualization/_parallel_coordinate.py (#5895, thanks @boringbyte!)
• Simplify type annotations for optuna/visualization/_rank.py (#5896, thanks @boringbyte!)
• Simplify type annotations for optuna/visualization/_param_importances.py (#5897, thanks @boringbyte!)
• Simplify type annotations for optuna/_callbacks.py (#5899, thanks @boringbyte!)
• Simplify type annotations for optuna/storages/journal/_file.py (#5900, thanks @boringbyte!)
• Simplify type annotations for tests/storages_tests/test_with_server.py (#5901, thanks @boringbyte!)
• Simplify type annotations for tests/test_multi_objective.py (#5902, thanks @boringbyte!)
• Simplify type annotations for tests/artifacts_tests/test_gcs.py (#5903, thanks @boringbyte!)
• Simplify type annotations for tests/samplers_tests/test_grid.py (#5904, thanks @boringbyte!)
• Simplify type annotations for optuna/study/_dataframe.py (#5905, thanks @boringbyte!)
• Simplify type annotations for tests/visualization_tests/test_optimization_history.py (#5906, thanks @boringbyte!)
• Simplify type annotations for tests/visualization_tests/test_intermediate_plot.py (#5907, thanks @boringbyte!)
• Simplify type annotations for multiple test files in tests/visualization_tests/ (#5908, thanks @boringbyte!)
• Avoid the port number conflict in the copy study tests (#5913)
• Simplify annotations in tests/study_tests/test_study.py (#5923, thanks @sawa3030!)

Continuous Integration

• Fix a GitHub action workflow for publishing to PyPI (https://github.com/optuna/optuna-integration/pull/181)
• Fix CI by adding a version constraint (https://github.com/optuna/optuna-integration/pull/186)
• Rename test_cache_is_invalidated and remove assert study._thread_local.cached_all_trials is None (#5733)
• Use Python 3.12 for docs build CI (#5742)
• Fix a GitHub action workflow for publishing to PyPI (#5759)
• Install older kaleido to fix CI errors (#5771)
• Add Python 3.12 to the Docker image CI (#5789)
• Move mypy related entries in setup.cfg to pyproject.toml (#5861)

Other

• Bump up version number to v4.2.0.dev (https://github.com/optuna/optuna-integration/pull/178)
• Bump up version number to 4.2.0 (https://github.com/optuna/optuna-integration/pull/191)
• Add CITATION.cff (#5746)
• Update news to add the autosampler article (#5748)
• Bump the version up to v4.2.0.dev (#5752)
• Update the news section for Optuna 4.1 release (#5758)
• Update pre-commit configuration file (#5847)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@HideakiImamura, @JLX0, @KannanShilen, @SimonPop, @boringbyte, @c-bata, @fusawa-yugo, @gen740, @himkt, @iamarunbrahma, @kAIto47802, @ktns, @mist714, @nabenabe0928, @not522, @nzw0301, @porink0424, @sawa3030, @sulan, @unKnownNG, @willdavidson05, @y0z

This is the release note of v4.1.0. Highlights of this release include:

• 🤖 AutoSampler: Automatic Selection of Optimization Algorithms
• 🚀 More scalable RDB Storage Backend
• 🧑‍💻 Five New Algorithms in OptunaHub (MO-CMA-ES, MOEA/D, etc.)
• 🐍 Support Python 3.13

The updated list of tested and supported Python releases is as follows:

• Optuna 4.1: supported by Python 3.8 - 3.13
• Optuna Integration 4.1: supported by Python 3.8 - 3.12
• Optuna Dashboard 0.17.0: supported by Python 3.8 - 3.13

Highlights

AutoSampler: Automatic Selection of Optimization Algorithms

<img width="750" alt="Blog-1" src="https://github.com/user-attachments/assets/f3ecd366-7e7b-49d6-ae38-f35301fa7d11">

AutoSampler automatically selects a sampler from those implemented in Optuna, depending on the situation. Using AutoSampler, as in the code example below, users can achieve optimization performance equal to or better than Optuna's default without being aware of which optimization algorithm to use.

$ pip install optunahub cmaes torch scipy

import optuna
import optunahub

auto_sampler_module = optunahub.load_module("samplers/auto_sampler")
study = optuna.create_study(sampler=auto_sampler_module.AutoSampler())

See the Medium blog post for details.

Enhanced RDB Storage Backend

This release incorporates comprehensive performance tuning on Optuna’s RDBStorage, leading to significant performance improvements. The table below shows the comparison results of execution times between versions 4.0 and 4.1.

For fair comparison, all experiments were repeated 10 times, and the mean execution time was compared. Additional detailed benchmark settings include the following:

• Objective Function: Each trial consists of 10 parameters and 10 user attributes
• Storage: MySQL 8.0 (with PyMySQL)
• Sampler: RandomSampler
• Execution Environment: Kubernetes Pod with 5 cpus and 8Gi RAM

Please note, due to extensive execution time, the figure for v4.0.0 with 50,000 trials represents the average of 7 runs instead of 10.

<details> <summary>Benchmark Script</summary>

import optuna
import time
import os
import numpy as np

optuna.logging.set_verbosity(optuna.logging.ERROR)
storage_url = "mysql+pymysql://user:password@<ipaddr>:<port>/<dbname>"
n_repeat = 10

def objective(trial: optuna.Trial) -> float:
    s = 0
    for i in range(10):
        trial.set_user_attr(f"attr{i}", "dummy user attribute")
        s += trial.suggest_float(f"x{i}", -10, 10) ** 2
    return s


def bench(n_trials):
    elapsed = []
    for i in range(n_repeat):
        start = time.time()
        study = optuna.create_study(
            storage=storage_url,
            sampler=optuna.samplers.RandomSampler()
        )
        study.optimize(objective, n_trials=n_trials, n_jobs=10)
        elapsed.append(time.time() - start)
        optuna.delete_study(study_name=study.study_name, storage=storage_url)
    print(f"{np.mean(elapsed)=} {np.std(elapsed)=}")


for n_trials in [1000, 10000, 50000]:
    bench(n_trials)

</details>

Five New Algorithms in OptunaHub (MO-CMA-ES, MOEA/D, etc.)

The following five new algorithms were added to OptunaHub!

• Multi-objective CMA-ES (MO-CMA-ES)by @y0z
• MOEA/D sampler by @hrntsm
• MAB Epsilon-Greedy Sampler by @ryota717
• NSGAII sampler with Initial Trials by @hrntsm
• CMA-ES with User Prior by @nabenabe0928

MO-CMA-ES is an extension of CMA-ES for multi-objective optimization. Its search mechanism is based on multiple (1+1)-CMA-ES and inherits good invariance properties from CMA-ES, such as invariance against rotation of the search space.

<img width="400" alt="mocmaes" src="https://github.com/user-attachments/assets/5289d2fd-81c9-40a4-b575-daf5d3cedc3f">

MOEA/D solves a multi-objective optimization problem by decomposing it into multiple single-objective problems. It allows for the maintenance of a good diversity of solutions during optimization. Please take a look at the article from Hiroaki NATSUME(@hrntsm) for more details.

<img width="400" alt="moead" src="https://github.com/user-attachments/assets/3ab83a64-b789-425b-800c-2c1f09e5a752">

Enhancements

• Update sklearn.py by addind catch to OptunaSearchCV (https://github.com/optuna/optuna-integration/pull/163, thanks @muhlbach!)
• Reduce SELECT statements by passing study_id to check_and_add in TrialParamModel (#5702)
• Introduce UPSERT in set_trial_user_attr (#5703)
• Reduce SELECT statements of _CachedStorage.get_all_trials by fixing filtering conditions (#5704)
• Reduce SELECT statements by removing unnecessary distribution compatibility check in set_trial_param() (#5709)
• Introduce UPSERT in set_trial_system_attr (#5741)

Bug Fixes

• Accept Mapping as param_distributions in OptunaSearchCV (https://github.com/optuna/optuna-integration/pull/172, thanks @yu9824!)
• Allow use of OptunaSearchCV with cross_val_predict (https://github.com/optuna/optuna-integration/pull/174, thanks @yu9824!)
• Fix GPSampler's suggestion failure within torch.no_grad() context manager (#5671, thanks @kAIto47802!)
• Fix a concurrency issue in GPSampler (#5737)
• Fix a concurrency issue in QMCSampler (#5740)

Installation

• Drop the support for Python 3.7 and update package metadata for Python 3.13 support (#5727)
• Remove dependency specifier for installing SciPy (#5736)

Documentation

• Add badges of PyPI and Conda Forge (https://github.com/optuna/optuna-integration/pull/167)
• Update news (#5655)
• Update news section in README.md (#5657)
• Add InMemoryStorage to document (#5672, thanks @kAIto47802!)
• Add the news about blog of JournalStorage to README.md (#5674)
• Fix broken link in the artifacts tutorial (#5677, thanks @kAIto47802!)
• Add pandas installation guide to RDB tutorial (#5685, thanks @kAIto47802!)
• Add installation guide to multi-objective tutorial (#5686, thanks @kAIto47802!)
• Update document and notice interoperability between NSGA-II and Pruners (#5688)
• Fix typo in EMMREvaluator (#5694)
• Fix RegretBoundEvaluator document (#5696)
• Update news section in README.md (#5705)
• Add link to related blog post in emmr.py (#5707)
• Update FAQ entry for model preservation using Optuna Artifact (#5716, thanks @chitvs!)
• Escape \D for Python 3.12 with sphinx build (#5735)
• Avoid using functions in sphinx_gallery_conf to remove document build error with Python 3.12 (#5738)
• Remove all generated directories in docs/source recursively (#5739)
• Add information about AutoSampler to the docs (#5745)

Examples

• Update CI config for fastai (https://github.com/optuna/optuna-examples/pull/279)
• Introduce optuna.artifacts to the PyTorch checkpoint example (https://github.com/optuna/optuna-examples/pull/280, thanks @kAIto47802!)
• Fix inline code in README files in kubernetes directory (https://github.com/optuna/optuna-examples/pull/282)
• Test ray example with Python 3.12 (https://github.com/optuna/optuna-examples/pull/284)
• Add a link to molecule LLM notebook (https://github.com/optuna/optuna-examples/pull/285)
• Fix syntax error in YAML file of rapids CI (https://github.com/optuna/optuna-examples/pull/286)
• Another fix for syntax error in YAML file of rapids CI (https://github.com/optuna/optuna-examples/pull/287)
• Add checking for Python 3.12 (https://github.com/optuna/optuna-examples/pull/288, thanks @kAIto47802!)
• Add Python 3.12 to tfkeras CI and remove warning message (https://github.com/optuna/optuna-examples/pull/289)
• Add Python 3.12 to lightgbm CI (https://github.com/optuna/optuna-examples/pull/290)
• Remove Python 3.7 from the workflow (https://github.com/optuna/optuna-examples/pull/291)

Code Fixes

• Add a comment for an unexpected bug in CategoricalDistribution (#5683)
• Add more information about the hack in WFG (#5687)
• Simplify type annotations to _imports.py (#5692, thanks @Prabhat-Thapa45!)
• Use __future__.annotations in optuna/_experimental.py (#5714, thanks @Jonathan43!)
• Use __future__.annotations in tests/importance_tests/fanova_tests/test_tree.py (#5731, thanks @guisp03!)
• Resolve TODO comments related to dropping Python 3.7 support (#5734)

Continuous Integration

• Fix CI for fastaiv2 (https://github.com/optuna/optuna-integration/pull/164)
• Fix for mypy (https://github.com/optuna/optuna-integration/pull/166)
• Fix mlflow integration for CI (https://github.com/optuna/optuna-integration/pull/168)
• Update CI to support Python 3.12 (https://github.com/optuna/optuna-integration/pull/170, thanks @kAIto47802!)
• Update artifact version to v4 (https://github.com/optuna/optuna-integration/pull/176)
• Add a tri-objective problem to speed benchmarking (#5635)
• Bump actions/download-artifact from 2 to 4.1.7 in /.github/workflows (#5660)
• Update MySQL version in CI from 5.7 to 8 (#5673)
• Remove performance benchmarks (#5675)
• Update PostgreSQL version in CI to latest (#5676)
• Update the version of Python used in the checks from 3.8 to 3.11 (#5684)
• Run tests with Python 3.13 (#5691)
• Fix for mypy (#5706)
• [hotfix] Add a version constraint on fakeredis (#5726)
• Update Python versions used in CI workflow (#5728)
• Update sphinx build's upload-artifact version (#5744)

Other

• Bump up version number to 4.1.0.dev (https://github.com/optuna/optuna-integration/pull/159)
• Update supported Python versions (https://github.com/optuna/optuna-integration/pull/175)
• Bump up to v4.1.0 (https://github.com/optuna/optuna-integration/pull/179)
• Add pre-commit config (#5408)
• Add link to MOEA/D blog post in News (#5719)
• Include all test files in sdist by updating MANIFEST.in (#5720)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@HideakiImamura, @Jonathan43, @Prabhat-Thapa45, @c-bata, @chitvs, @contramundum53, @eukaryo, @gen740, @guisp03, @kAIto47802, @muhlbach, @nabenabe0928, @not522, @nzw0301, @porink0424, @toshihikoyanase, @y0z, @yu9824

Here is the release note of v4.0.0. Please also check out the release blog post.

If you want to update the Optuna version of your existing projects to v4.0, please see the migration guide.

We have also published blog posts about the development items. Please check them out!

• OptunaHub, a Feature-Sharing Platform for Optuna, Now Available in Official Release!
• File Management during LLM (Large Language Model) Trainings by Optuna v4.0.0 Artifact Store
• Significant Speed Up of Multi-Objective TPESampler in Optuna v4.0.0

Highlights

Official Release of Feature-Sharing Platform OptunaHub

We officially released OptunaHub, a feature-sharing platform for Optuna. A large number of optimization and visualization algorithms are available in OptunaHub. Contributors can easily register their methods and deliver them to Optuna users around the world.

Please also read the OptunaHub release blog post.

Enhanced Experiment Management Feature: Official Support of Artifact Store

Artifact Store is a file management feature for files generated during optimization, dubbed artifacts. In Optuna v4.0, we stabilized the existing file upload API and further enhanced the usability of Artifact Store by adding some APIs such as the artifact download API. We also added features to show JSONL and CSV files on Optuna Dashboard in addition to the existing support for images, audio, and video. With this official support, the API backward compatibility will be guaranteed.

For more details, please check the blog post.

JournalStorage: Official Support of Distributed Optimization via Network File System

JournalStorage is a new Optuna storage experimentally introduced in Optuna v3.1 (see the blog post for details). Optuna has JournalFileBackend, a storage backend for various file systems. It can be used on NFS, allowing Optuna to scale to multiple nodes.

In Optuna v4.0, the API for JournalStorage has been reorganized, and JournalStorage is officially supported. This official support guarantees its backward compatibility from v4.0. For details on the API changes, please refer to the Optuna v4.0 Migration Guide.

import optuna
from optuna.storages import JournalStorage
from optuna.storages.journal import JournalFileBackend

def objective(trial: optuna.Trial) -> float:
    ...

storage = JournalStorage(JournalFileBackend("./optuna_journal_storage.log"))
study = optuna.create_study(storage=storage)
study.optimize(objective)

Significant Speedup of Multi-Objective TPESampler

Before v4.0, the multi-objective TPESampler sometimes limits the number of trials during optimization due to the sampler bottleneck after a few hundred trials. Optuna v4.0 drastically improves the sampling speed, e.g., 300 times faster for three objectives with 200 trials, and enables users to handle much more trials. Please check the blog post for details.

Introduction of a New Terminator Algorithm

Optuna Terminator was originally introduced for hyperparameter optimization of machine learning algorithms using cross-validation. To accept broader use cases, Optuna v4.0 introduced the Expected Minimum Model Regret (EMMR) algorithm. Please refer to the EMMREvaluator document for details.

Enhancements of Constrained Optimization

We have gradually expanded the support for constrained optimization. In v4.0, study.best_trial and study.best_trials start to support constraint optimization. They are guaranteed to satisfy the constraints, which was not the case previously.

Breaking Changes

Optuna removes deprecated features in major releases. To prevent users' code from suddenly breaking, we take a long interval between when a feature is deprecated and when it is removed. By default, features are removed when the major version has increased by two since the feature was deprecated. For this reason, the main target features for removal in v4.0 were deprecated at v2.x. Please refer to the migration guide for the removed features list.

• Delete deprecated three integrations, skopt, catalyst, and fastaiv1 (https://github.com/optuna/optuna-integration/pull/114)
• Remove deprecated CmaEsSampler from integration (https://github.com/optuna/optuna-integration/pull/116)
• Remove verbosity of LightGBMTuner (https://github.com/optuna/optuna-integration/pull/136)
• Move positional args of LightGBMTuner (https://github.com/optuna/optuna-integration/pull/138)
• Remove multi_objective (#5390)
• Delete deprecated _ask and _tell (#5398)
• Delete deprecated --direction(s) arguments in the ask command (#5405)
• Delete deprecated three integrations, skopt, catalyst, and fastaiv1 (#5407)
• Remove the default normalization of importance in f-ANOVA (#5411)
• Remove samplers.intersection (#5414)
• Drop implicit create-study in ask command (#5415)
• Remove deprecated study optimize CLI command (#5416)
• Remove deprecated CmaEsSampler from integration (#5417)
• Support constrained optimization in best_trial (#5426)
• Drop --study in cli.py (#5430)
• Deprecate constraints_func in plot_pareto_front function (#5455)
• Rename some class names related to JournalStorage (#5539)
• Remove optuna.samplers.MOTPESampler (#5640)

New Features

• Add Comet ML integration (https://github.com/optuna/optuna-integration/pull/63, thanks @caleb-kaiser!)
• Add Knowledge Gradient candidates functions (https://github.com/optuna/optuna-integration/pull/125, thanks @alxhslm!)
• Add is_exhausted() function in the GridSampler class (#5306, thanks @aaravm!)
• Remove experimental from plot (#5413)
• Implement download_artifact (#5448)
• Add a function to list linked artifact information (#5467)
• Stabilize artifact APIs (#5567)
• Stabilize JournalStorage (#5568)
• Add EMMREvaluator and MedianErrorEvaluator (#5602)

Enhancements

• Pass two arguments to the forward of ConstrainedMCObjective to support botorch=0.10.0 (https://github.com/optuna/optuna-integration/pull/106)
• Speed up non-dominated sort (#5302)
• Make 2d hypervolume computation twice faster (#5303)
• Reduce the time complexity of HSSP 2d from O(NK^2 log K) to O((N - K)K) (#5346)
• Introduce lazy hypervolume calculations in HSSP for speedup (#5355)
• Make plot_contour faster (#5369)
• Speed up to_internal_repr in CategoricalDistribution (#5400)
• Allow users to modify categorical distance more easily (#5404)
• Speed up WFG by NumPy vectorization (#5424)
• Check whether the study is multi-objective in sample_independent of GPSampler (#5428)
• Suppress warnings from numpy in hypervolume computation (#5432)
• Make an option to assume Pareto optimality in WFG (#5433)
• Adapt multi objective to NumPy v2.0.0 (#5493)
• Enhance the error message for integration installation (#5498)
• Reduce journal size of JournalStorage (#5526)
• Simplify a SQL query for getting the trial_id of best_trial (#5537)
• Add import check for artifact store objects (#5565)
• Refactor _is_categorical() in optuna/optuna/visualization (#5587, thanks @kAIto47802!)
• Speed up WFG by using a fact that hypervolume calculation does not need (second or later) duplicated Pareto solutions (#5591)
• Enhance the error message of multi_objective deletion (#5641)

Bug Fixes

• Log None objective trials correctly (https://github.com/optuna/optuna-integration/pull/119, thanks @neel04!)
• Pass two arguments to the forward of ConstrainedMCObjective in qnei_candidates_func (https://github.com/optuna/optuna-integration/pull/124, thanks @alxhslm!)
• Allow single split cv in OptunaSearchCV (https://github.com/optuna/optuna-integration/pull/128, thanks @sgerloff!)
• Update BoTorch samplers to support new constraints interface (https://github.com/optuna/optuna-integration/pull/132, thanks @alxhslm!)
• Fix WilcoxonPruner bug when best_trial has no intermediate value (#5354)
• Debug an error caused by convergence in GPSampler (#5359)
• Fix average_is_best implementation in WilcoxonPruner (#5366)
• Create an unique renaming filename for each release operation in lock systems of JournalStorage (#5389)
• Fix _normalize_value for incomplete trials (#5422)
• Fix heartbeat for race condition (#5431)
• Guarantee weights_below to be finite in MOTPE (#5435)
• Fix _log_complete_trial for constrained optimization (#5462)
• Convert step to int in report (#5488)
• Use a fixed seed value when seed=None in GridSampler (#5490)
• Acquire session lock only for write in heartbeat (#5496)
• Fix a bug in _create_new_trial and refactor it (#5497)
• Add rdb create new trial test (#5525)
• Fix a bug in sample_normalized_param for GPSampler (#5543)
• Fix the error caused in plot_contour() with an impossible pair of variables (#5630, thanks @kAIto47802!)
• Fix inappropriate behavior in plot_rank() when None values exist in trial (#5634, thanks @kAIto47802!)

Installation

• Add an option to install integration dependencies via pip (https://github.com/optuna/optuna-integration/pull/130)
• Add version constraint to numpy (https://github.com/optuna/optuna-integration/pull/131)

Documentation

• Enhance README.md (https://github.com/optuna/optuna-integration/pull/126)
• Add document page and fix docstring and comments (https://github.com/optuna/optuna-integration/pull/134)
• Update the docstring of PyCmaSampler (https://github.com/optuna/optuna-integration/pull/145)
• Add Dashboard and OptunaHub links to the document header (https://github.com/optuna/optuna-integration/pull/155)
• Remove duplicated license definition (https://github.com/optuna/optuna-integration/pull/156)
• Use sphinx rst syntax (#5345)
• Fix typo (#5351)
• Update docs about show_progress_bar (#5393)
• Fix artifact tutorial (#5451)
• Revise docs to specify ArtifactStore methods as non-public (#5474)
• Improve the docstring of JournalFileStorage (#5475)
• Improve make clean in docs (#5487)
• Improve the example of chemical structures in optuna artifact tutorial (#5491)
• Add FAQ for artifact store remove API (#5501)
• Add a documentation for ArtifactMeta (#5511)
• Change docs version from latest to stable (#5518)
• Add a list of supported storage for artifact store (#5532)
• Rename filenames for JournalFileStorage in documents and a tutorial (#5535)
• Add explanation for lock_obj to JournalFileStorage docstring (#5540)
• Fix storages document sections (#5553)
• Add Returns to the docstring of load_study (#5554, thanks @kAIto47802!)
• Fix paths related to storages.journal (#5560)
• Rename filename for JournalFileBackend in examples (#5562)
• Add thumbnail (#5575)
• Add link to optunahub-registry (#5586)
• Fix minor artifacts document issues (#5592)
• Add the OptunaHub link to the document header (#5595)
• Improve a docstring of CmaEsSampler (#5603)
• Fix the expired links caused by the directory name change in optuna-examples (#5623, thanks @kAIto47802!)
• Remove the use of evaluator.evaluate function in the example of PedAnovaImportanceEvaluator (#5632)
• Add the news section to README.md (#5636)
• Add SNS URLs to README.md (#5637)
• Add TPE blog post link to doc (#5638)
• Add artifact store post link (#5639)
• Add installation guide for visualization tutorial (#5644, thanks @kAIto47802!)
• Remove a broken link (#5648)
• Update the News section of README.md (#5649)

Examples

• Support tensorflow 2.16.1 and separate CI run for tensorflow estimator (https://github.com/optuna/optuna-examples/pull/248)
• Replace deprecated jax function (https://github.com/optuna/optuna-examples/pull/250)
• Drop Python 3.7 support for wandb (https://github.com/optuna/optuna-examples/pull/252)
• Support python 3.12 in the CIs (https://github.com/optuna/optuna-examples/pull/253)
• Remove dask version constraint (https://github.com/optuna/optuna-examples/pull/254)
• Update GitHub actions versions to actions/checkout@v4 and actions/setup-python@v5 (https://github.com/optuna/optuna-examples/pull/255)
• Delete an example using deprecated fastaiv1 (https://github.com/optuna/optuna-examples/pull/256)
• Rename fastaiv2 to fastai (https://github.com/optuna/optuna-examples/pull/257)
• Fix CI (https://github.com/optuna/optuna-examples/pull/258)
• Fix path in pruners workflow (https://github.com/optuna/optuna-examples/pull/259)
• Install tensorflow-cpu in CIs (https://github.com/optuna/optuna-examples/pull/260)
• Add python 3.12 and run terminator search cv (https://github.com/optuna/optuna-examples/pull/264)
• Enhance README.md (https://github.com/optuna/optuna-examples/pull/265)
• Organize the directory structure (https://github.com/optuna/optuna-examples/pull/266)
• Fix CI with version constraints of packages (https://github.com/optuna/optuna-examples/pull/267)
• Install numpy<2.0.0 for catboost example (https://github.com/optuna/optuna-examples/pull/268)
• Add Python 3.12 to aim test python versions (https://github.com/optuna/optuna-examples/pull/270)
• Add Python 3.12 to fastai test python versions (https://github.com/optuna/optuna-examples/pull/271)
• Use auc score to unify the intermediate and objective values (https://github.com/optuna/optuna-examples/pull/272)
• Add python 3.12 for mlflow CI python versions (https://github.com/optuna/optuna-examples/pull/274)
• Add python 3.12 to CI python versions for keras and tensorboard examples (https://github.com/optuna/optuna-examples/pull/275)
• Add an example of artifact store (https://github.com/optuna/optuna-examples/pull/276)
• Separate basic and faq directories (https://github.com/optuna/optuna-examples/pull/277, thanks @kAIto47802!)
• Remove chainer CI (https://github.com/optuna/optuna-examples/pull/278)

Tests

• Suppress ExperimentalWarnings (https://github.com/optuna/optuna-integration/pull/108)
• Add a unit test for convergence of acquisition function in GPSampler (#5365)
• Remove unused block (#5368)
• Implement backward compatibility tests for JournalStorage (#5486)
• Add some tests for visualization functions (#5599)

Code Fixes

• Align run name strategy between MLflowCallback and track_in_mlflow method (https://github.com/optuna/optuna-integration/pull/111, thanks @TTRh!)
• Use TYPE_CHECKING for ObjectiveFuncType (https://github.com/optuna/optuna-integration/pull/113)
• Ignore ExperimentalWarning for track_in_wandb (https://github.com/optuna/optuna-integration/pull/122)
• Remove unused module (https://github.com/optuna/optuna-integration/pull/127)
• Apply formatter and follow the Optuna conventions (https://github.com/optuna/optuna-integration/pull/133)
• Refactor an internal process of BoTorchSampler (https://github.com/optuna/optuna-integration/pull/147)
• Add missing experimental decorators to the comet integration (https://github.com/optuna/optuna-integration/pull/148)
• Use __future__.annotations (https://github.com/optuna/optuna-integration/pull/150)
• Fix E721 errors (https://github.com/optuna/optuna-integration/pull/151)
• Fix future annotations in percentile.py (#5322, thanks @aaravm!)
• Delete examples directory (#5329)
• Simplify annotations in optuna/pruners/_hyperband.py (#5338, thanks @keita-sa!)
• Simplify annotations in optuna/pruners/_successive_halving.py and optuna/pruners/_threshold.py (#5343, thanks @keita-sa!)
• Reduce test warnings (#5344)
• Simplify type annotations for storages/_base.py (#5352, thanks @Obliquedbishop!)
• Use TYPE_CHECKING for Study in samplers (#5391)
• Refactor _normalize_objective_values in NSGA-III to supress RuntimeWarning (#5399)
• Make ObjectiveFuncType available externally (#5401)
• Replace numpy with np (#5412)
• Bundle experimental feature warning (#5434)
• Simplify annotations in _brute_force.py (#5441)
• Add __future__.annotations to base sampler (#5442)
• Introduce __future__.annotations to TPE-related modules (#5443)
• Adapt to __future__.annotations in optuna/storages/_rdb/models.py (#5452, thanks @aisha-partha!)
• Debug an unintended assertion error in GPSampler (#5484)
• Make WFG a function (#5504)
• Enhance the comments in create_new_trial (#5510)
• Replace single trailing underscore with double trailing underscore in .rst files (#5514, thanks @47aamir!)
• Fix hyperlinks to use double trailing underscores (#5515, thanks @virendrapatil24!)
• Replace _ with __ in the link in Sphinx (#5517)
• Refactor plot_parallel_coordinate() (#5527, thanks @karthikkurella!)
• Remove an obsolete TODO comment (#5528)
• Unite the argument order of artifact APIs (#5533)
• Make assume_unique_lexsorted in is_pareto_front required (#5534)
• Simplify annotations in terminator module (#5536)
• Simplify the type annotations in samplers/_qmc.py (#5538, thanks @kAIto47802!)
• Rename solution_set to loss_vals in hypervolume computation (#5541)
• Expand the type of callbacks in optimize to Iterable (#5542, thanks @kz4killua!)
• Organize the directory structure of JournalStorage (#5544)
• Add followup of journal storage module structure organization PR (#5546)
• Move RetryFailedTrialCallback to optuna.storages._callbacks (#5551)
• Change removed_version of deprecated JournalStorage classes (#5552)
• Reorganize a journal storage module structure (#5555)
• Remove unused private deprecated class BaseJournalLogSnapshot (#5561)
• Replace relative import to absolute path (#5569, thanks @RektPunk!)
• Simplify annotation in importance (#5578, thanks @RektPunk!)
• Simplify annotation in trial (#5579, thanks @RektPunk!)
• Fix mypy warnings in matplotlib/_contour.py with Python3.11 (#5580)
• Remove redundant loop in _get_rank_subplot_info (#5581, thanks @RektPunk!)
• Simplify annotations in _partial_fixed.py (#5583)
• Simplify annotation in _cmaes.py (#5584, thanks @RektPunk!)
• Simplify annotations related to GridSampler (#5588)
• Reorganize names related to search_space in terminator/improvement/evaluator.py (#5594)
• Refactor _bisect in _truncnorm.py (#5598)
• Simplify annotation storages (#5608, thanks @RektPunk!)
• Add a note about np.unique (#5615)
• Fix ruff errors except for E731 (#5627)

Continuous Integration

• Split ci (https://github.com/optuna/optuna-integration/pull/98)
• Install pipdeptree v2.16.2 to hotfix parse error (https://github.com/optuna/optuna-integration/pull/109)
• Remove dask version constraint (https://github.com/optuna/optuna-integration/pull/112)
• Rename CI name (https://github.com/optuna/optuna-integration/pull/120)
• Add deprecated arg for the comet CI job (https://github.com/optuna/optuna-integration/pull/135)
• Hotfix the CI in LightGBM test (https://github.com/optuna/optuna-integration/pull/142)
• Hotfix the mypy error of test_pytorch_lightning.py (https://github.com/optuna/optuna-integration/pull/144)
• Rename CI jobs (#5353, thanks @Obliquedbishop!)
• Update GitHub actions versions of actions/setup-python and actions/checkout (#5367)
• Add type: ignore for CI hotfix (#5419)
• Use CPU-only PyTorch wheels on GitHub Actions (#5465)
• Add a version constraint to NumPy (#5492)
• Add the note of the Hotfix in the workflow file (#5494)
• Remove unnecessary version constraint from workflow (#5495)
• Remove skipped tests for BoTorch with Python 3.12 (#5519)
• Fix the version conflicts in the workflow for the minimum version constraints (#5523)
• Replace sleep function with spin waiting to stabilize the frequent Mac test failure (#5549)

Other

• Bump up version number to 4.0.0dev (https://github.com/optuna/optuna-integration/pull/102)
• Update PyPI classifiers for Python 3.11 support (https://github.com/optuna/optuna-integration/pull/129)
• Bump up version number to 4.0.0b0 (https://github.com/optuna/optuna-integration/pull/139)
• Bump up version number to 4.0.0 (https://github.com/optuna/optuna-integration/pull/158)
• Bump up to version number v4.0.0.dev (#5319)
• Bump up to version number 4.0.0b0 (#5572)
• Split LICENSE file (#5597)
• Add OptunaHub section in README.md (#5601)
• Remove duplicated definition of license in pyproject.toml (#5645)
• Bump up to version number 4.0.0 (#5653)
• Bump up to version number 4.1.0dev (#5654)

Thanks to All the Contributors and Sponsors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@47aamir, @Alnusjaponica, @HideakiImamura, @Obliquedbishop, @RektPunk, @TTRh, @aaravm, @aisha-partha, @alxhslm, @c-bata, @caleb-kaiser, @contramundum53, @eukaryo, @gen740, @kAIto47802, @karthikkurella, @keisuke-umezawa, @keita-sa, @kz4killua, @nabenabe0928, @neel04, @not522, @nzw0301, @porink0424, @sgerloff, @toshihikoyanase, @virendrapatil24, @y0z

Optuna is sponsored by the following sponsors on GitHub.

@AlphaImpact, @dec1costello, @dubovikmaster, @shu65, @raquelhortab

This is the release note of v4.0.0-b0.

If you want to update your existing projects from Optuna v3.x to Optuna v4, please see the migration guide and try out Optuna v4.

Highlights

OptunaHub Beta Release

The Optuna team released the beta version of OptunaHub, the feature-sharing platform for Optuna. Registered features can be easily implemented on users’ code and contributors can register the features they implement. The beta version of OptunaHub is now ready to accept contributions from all over the world. Visit hub.optuna.org!

The following code shows an example to use a sampler registered on OptunaHub.

% pip install optunahub

import optunahub
import optuna

def objective(trial):
    x = trial.suggest_float("x", 0, 1)
    return x

mod = optunahub.load_module("samplers/simulated_annealing")

sampler = mod.SimulatedAnnealingSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=20)

Stabilization of Artifact

The stable version of the artifact module is available, now equipped with several new APIs. This module introduces capabilities for managing the relatively large-sized data such as model snapshots in hyperparameter tuning, training/validation datasets, and etc. Compared to third-party libraries for experiment tracking, the advantage of using Optuna’s artifact module is a tight integration of Optuna Dashboard. This allows users to see artifacts (files) associated with the Optuna trial or study.

Here is a list of new APIs:

• download_artifact: Download an artifact from the artifact store.
• get_all_artifact_meta: List the associated artifact information of the provided trial or study.

Stabilization of JournalStorage

The stable version of JournalStorage is available. This implies we have decided to maintain backward compatibility of the log format in the future releases.

Please note that this release introduces the following API changes to improve the clarity of class names and the module structure.

Breaking Changes

• Delete deprecated three integrations, skopt, catalyst, and fastaiv1 (https://github.com/optuna/optuna-integration/pull/114)
• Remove deprecated CmaEsSampler from integration (https://github.com/optuna/optuna-integration/pull/116)
• Remove verbosity of LightGBMTuner (https://github.com/optuna/optuna-integration/pull/136)
• Move positional args of LightGBM tuner (https://github.com/optuna/optuna-integration/pull/138)
• Remove multi_objective (#5390)
• Delete deprecated _ask and _tell (#5398)
• Delete deprecated --direction(s) arguments in the ask command (#5405)
• Delete deprecated three integrations, skopt, catalyst, and fastaiv1 (#5407)
• Remove the default normalization of importance in f-ANOVA (#5411)
• Remove samplers.intersection (#5414)
• Drop implicit create-study in ask command (#5415)
• Remove deprecated study optimize CLI command (#5416)
• Remove deprecated CmaEsSampler from integration (#5417)
• Support constrained optimization in best_trial (#5426)
• Drop --study in cli.py (#5430)
• Deprecate constraints_func in plot_pareto_front function (#5455)
• Rename some classnames related to JournalStorage (#5539)

New Features

• Add Comet ML integration (https://github.com/optuna/optuna-integration/pull/63, thanks @caleb-kaiser!)
• Add Knowledge Gradient candidates functions (https://github.com/optuna/optuna-integration/pull/125, thanks @alxhslm!)
• Add is_exhausted() function in the GridSampler class (#5306, thanks @aaravm!)
• Remove experimental from plot (#5413)
• Implement download_artifact (#5448)
• Add a function to list linked artifact information (#5467)
• Stabilize artifact APIs (#5567)
• Stabilize JournalStorage (#5568)

Enhancements

• Pass two arguments to the forward of ConstrainedMCObjective to support botorch=0.10.0 (https://github.com/optuna/optuna-integration/pull/106)
• Speed up non-dominated sort (#5302)
• Make 2d hypervolume computation twice faster (#5303)
• Reduce the time complexity of HSSP 2d from O(NK^2 log K) to O((N - K)K) (#5346)
• Introduce lazy hypervolume calculations in HSSP for speedup (#5355)
• Make plot_contour faster (#5369)
• Speed up to_internal_repr in CategoricalDistribution (#5400)
• Allow users to modify categorical distance more easily (#5404)
• Speed up WFG by NumPy vectorization (#5424)
• Check whether the study is multi-objective in sample_independent of GPSampler (#5428)
• Suppress warnings from numpy in hypervolume computation (#5432)
• Make an option to assume Pareto optimality in WFG (#5433)
• Adapt multi objective to NumPy v2.0.0 (#5493)
• Enhance the error message for integration installation (#5498)
• Reduce journal size of JournalStorage (#5526)
• Simplify a SQL query for getting the trial_id of best_trial (#5537)

Bug Fixes

• Log None objective trials correctly (https://github.com/optuna/optuna-integration/pull/119, thanks @neel04!)
• Pass two arguments to the forward of ConstrainedMCObjective in qnei_candidates_func (https://github.com/optuna/optuna-integration/pull/124, thanks @alxhslm!)
• Allow single split cv in OptunaSearchCV (https://github.com/optuna/optuna-integration/pull/128, thanks @sgerloff!)
• Fix WilcoxonPruner bug when best_trial has no intermediate value (#5354)
• Debug an error caused by convergence in GPSampler (#5359)
• Fix average_is_best implementation in WilcoxonPruner (#5366)
• Create an unique renaming filename for each release operation in lock systems of JournalStorage (#5389)
• Fix _normalize_value for incomplete trials (#5422)
• Fix heartbeat for race condition (#5431)
• Guarantee weights_below to be finite in MOTPE (#5435)
• Fix _log_complete_trial for constrained optimization (#5462)
• Convert step to int in report (#5488)
• Use a fixed seed value when seed=None in GridSampler (#5490)
• Acquire session lock only for write in heartbeat (#5496)
• Fix a bug in _create_new_trial and refactor it (#5497)
• Add rdb create new trial test (#5525)
• Fix a bug in sample_normalized_param for GPSampler (#5543)

Installation

• Add an option to install integration dependencies via pip (https://github.com/optuna/optuna-integration/pull/130)
• Add version constraint to numpy (https://github.com/optuna/optuna-integration/pull/131)

Documentation

• Enhance README.md (https://github.com/optuna/optuna-integration/pull/126)
• Add document page and fix docstring and comments (https://github.com/optuna/optuna-integration/pull/134)
• Use sphinx rst syntax (#5345)
• Fix typo (#5351)
• Update docs about show_progress_bar (#5393)
• Fix artifact tutorial (#5451)
• Revise docs to specify ArtifactStore methods as non-public (#5474)
• Improve the docstring of JournalFileStorage (#5475)
• Improve make clean in docs (#5487)
• Improve the example of chemical structures in optuna artifact tutorial (#5491)
• Add FAQ for artifact store remove API (#5501)
• Add a documentation for ArtifactMeta (#5511)
• Change docs version from latest to stable (#5518)
• Add a list of supported storage for artifact store (#5532)
• Rename filenames for JournalFileStorage in documents and a tutorial (#5535)
• Add explanation for lock_obj to JournalFileStorage docstring (#5540)
• Fix storages document sections (#5553)
• Add Returns to the docstring of load_study (#5554, thanks @kAIto47802!)
• Fixing paths related to storages.journal (#5560)
• Rename filename for JournalFileBackend in examples (#5562)

Examples

• Support tensorflow 2.16.1 and separate CI run for tensorflow estimator (https://github.com/optuna/optuna-examples/pull/248)
• Replace deprecated jax function (https://github.com/optuna/optuna-examples/pull/250)
• Drop Python 3.7 support for wandb (https://github.com/optuna/optuna-examples/pull/252)
• Support python 3.12 in the CIs (https://github.com/optuna/optuna-examples/pull/253)
• Remove dask version constraint (https://github.com/optuna/optuna-examples/pull/254)
• Update GitHub actions versions to actions/checkout@v4 and actions/setup-python@v5 (https://github.com/optuna/optuna-examples/pull/255)
• Delete an example using deprecated fastaiv1 (https://github.com/optuna/optuna-examples/pull/256)
• Rename fastai v2 to fastai (https://github.com/optuna/optuna-examples/pull/257)
• Fix CI (https://github.com/optuna/optuna-examples/pull/258)
• Fix path in pruners workflow (https://github.com/optuna/optuna-examples/pull/259)
• Install tensorflow-cpu in CIs (https://github.com/optuna/optuna-examples/pull/260)
• Add python 3.12 and run terminator search cv (https://github.com/optuna/optuna-examples/pull/264)
• Organize the directory structure (https://github.com/optuna/optuna-examples/pull/266)
• Fix CI with version constraints of packages (https://github.com/optuna/optuna-examples/pull/267)
• Install numpy<2.0.0 for catboot example (https://github.com/optuna/optuna-examples/pull/268)

Tests

• Suppress ExperimentalWarnings (https://github.com/optuna/optuna-integration/pull/108)
• Add a unit test for convergence of acquisition function in GPSampler (#5365)
• Remove unused block (#5368)
• Implement backward compatibility tests for JournalStorage (#5486)

Code Fixes

• Align run name strategy between MLflowCallback and track_in_mlflow method (https://github.com/optuna/optuna-integration/pull/111, thanks @TTRh!)
• Use TYPE_CHECKING for ObjectiveFuncType (https://github.com/optuna/optuna-integration/pull/113)
• Ignore ExperimentalWarning for track_in_wandb (https://github.com/optuna/optuna-integration/pull/122)
• Remove unused module (https://github.com/optuna/optuna-integration/pull/127)
• Apply formatter and follow the Optuna conventions (https://github.com/optuna/optuna-integration/pull/133)
• Fix future annotations in percentile.py (#5322, thanks @aaravm!)
• Delete examples directory (#5329)
• Simplify annotations in optuna/pruners/_hyperband.py (#5338, thanks @keita-sa!)
• Simplify annotations in optuna/pruners/_successive_halving.py and optuna/pruners/_threshold.py (#5343, thanks @keita-sa!)
• Reduce test warnings (#5344)
• Simplify type annotations for storages/_base.py (#5352, thanks @Obliquedbishop!)
• Use TYPE_CHECKING for Study in samplers (#5391)
• Refactor _normalize_objective_values in NSGA-III to supress RuntimeWarning (#5399)
• Make ObjectiveFuncType available externally (#5401)
• Replace numpy with np (#5412)
• Bundle experimental feature warning (#5434)
• Simplify annotations in _brute_force.py (#5441)
• Add __future__.annotations to base sampler (#5442)
• Introduce __future__.annotations to TPE-related modules (#5443)
• Adapt to __future__.annotations in optuna/storages/_rdb/models.py (#5452, thanks @aisha-partha!)
• Debug an unintended assertion error in GPSampler (#5484)
• Make WFG a function (#5504)
• Enhance the comments in create_new_trial (#5510)
• Replace single trailing underscore with double trailing underscore in .rst files (#5514, thanks @47aamir!)
• Fix hyperlinks to use double trailing underscores (#5515, thanks @virendrapatil24!)
• Replace _ with __ in the link in Sphinx (#5517)
• Refactor plot_parallel_coordinate() (#5527, thanks @karthikkurella!)
• Remove an obsolete TODO comment (#5528)
• Unite the argument order of artifact APIs (#5533)
• Make assume_unique_lexsorted in is_pareto_front required (#5534)
• Simplify annotations in terminator module (#5536)
• Simplify the type annotations in samplers/_qmc.py (#5538, thanks @kAIto47802!)
• Rename solution_set to loss_vals in hypervolume computation (#5541)
• Expand the type of callbacks in optimize to Iterable (#5542, thanks @kz4killua!)
• Organize the directory structure of JournalStorage (#5544)
• Add followup of journal storage module structure organization PR (#5546)
• Move RetryFailedTrialCallback to optuna.storages._callbacks (#5551)
• Change removed_version of deprecated JournalStorage classes (#5552)
• Reorganize a journal storage module structure (#5555)
• Remove unused private deprecated class BaseJournalLogSnapshot (#5561)

Continuous Integration

• Split ci (https://github.com/optuna/optuna-integration/pull/98)
• Install pipdeptree v2.16.2 to hotfix parse error (https://github.com/optuna/optuna-integration/pull/109)
• Remove dask version constraint (https://github.com/optuna/optuna-integration/pull/112)
• Rename CI name (https://github.com/optuna/optuna-integration/pull/120)
• Add deprecated arg for the comet CI job (https://github.com/optuna/optuna-integration/pull/135)
• Rename CI jobs (#5353, thanks @Obliquedbishop!)
• Update GitHub actions versions of actions/setup-python and actions/checkout (#5367)
• Add type: ignore for CI hotfix (#5419)
• Use CPU-only PyTorch wheels on GitHub Actions (#5465)
• Add a version constraint to NumPy (#5492)
• Add the note of the Hotfix in the workflow file (#5494)
• Remove unnecessary version constraint from workflow (#5495)
• Remove skipped tests for BoTorch with Python 3.12 (#5519)
• Fix the version conflicts in the workflow for the minimum version constraints (#5523)
• Replace sleep function with spin waiting to stabilize the frequent Mac test failure (#5549)

Other

• Bump up version number to 4.0.0dev (https://github.com/optuna/optuna-integration/pull/102)
• Update PyPI classifiers for Python 3.11 support (https://github.com/optuna/optuna-integration/pull/129)
• Bump up version number to 4.0.0b0 (https://github.com/optuna/optuna-integration/pull/139)
• Bump up to version number v4.0.0.dev (#5319)
• Bump up to version number 4.0.0b0 (#5572)

Thanks to All the Contributors!

This release was made possible by the authors and the people who participated in the reviews and discussions.

@47aamir, @Alnusjaponica, @HideakiImamura, @Obliquedbishop, @TTRh, @aaravm, @aisha-partha, @alxhslm, @c-bata, @caleb-kaiser, @contramundum53, @eukaryo, @gen740, @kAIto47802, @karthikkurella, @keisuke-umezawa, @keita-sa, @kz4killua, @nabenabe0928, @neel04, @not522, @nzw0301, @porink0424, @sgerloff, @toshihikoyanase, @virendrapatil24, @y0z

This is the release note of v3.6.2.

Bug Fixes

• [Backport] Fix the default sampler of load_study function.

Other

• Bump up version number to v3.6.2