Source-linked AI summary
Optuna: A Next-generation Hyperparameter Optimization Framework
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, Masanori Koyama
TL;DR
Hyperparameter search is cumbersome, while existing frameworks require static search spaces that are difficult to describe for complex model families. This paper introduces Optuna’s define-by-run design, efficient sampling and pruning, and versatile architecture, reporting stronger performance than major black-box frameworks across experiments and applications.
Problem
Existing hyperparameter optimization frameworks require statically defined search spaces, limiting their suitability for complex experiments with heterogeneous models and conditional variables.
Method
Optuna combines a define-by-run API with customizable sampling and pruning algorithms and an architecture supporting interactive experiments through distributed computation.
Results
Optuna outperforms many major black-box optimization frameworks while supporting experiments on benchmark datasets and real-world applications.
Takeaways & Limitations
The results support adopting define-by-run, efficient search and pruning, and scalable versatile architecture in next-generation hyperparameter optimization frameworks.
Takeaways & Limitations
Optimization efficiency depends on the relationship between the number of workers and optimization efficiency, which is not intuitive.
Abstract
from arXiv · showhide
The purpose of this study is to introduce new design-criteria for next-generation hyperparameter optimization software. The criteria we propose include (1) define-by-run API that allows users to construct the parameter search space dynamically, (2) efficient implementation of both searching and pruning strategies, and (3) easy-to-setup, versatile architecture that can be deployed for various purposes, ranging from scalable distributed computing to light-weight experiment conducted via interactive interface. In order to prove our point, we will introduce Optuna, an optimization software which is a culmination of our effort in the development of a next generation optimization software. As an optimization software designed with define-by-run principle, Optuna is particularly the first of its kind. We will present the design-techniques that became necessary in the development of the software that meets the above criteria, and demonstrate the power of our new design through experimental results and real world applications. Our software is available under the MIT license (https://github.com/pfnet/optuna/).
1 Introduction
The introduction identifies static search-space construction, inefficient pruning, and setup limitations as shortcomings of existing hyperparameter optimization frameworks. It proposes Optuna and three design criteria—define-by-run search spaces, efficient customizable sampling and pruning, and easy-to-set-up versatility—to address them.
- Motivation: Hyperparameter search is cumbersome, increasing demand for efficient automatic tuning as deep learning methods grow more complex.
- Related work: Existing frameworks use varied sampling methods, including Gaussian processes, TPE, random forests, pruning, and distributed parallel optimization.
- Limitations: Many frameworks require users to statically describe each model’s search space, which can be difficult for large-scale experiments and undermine advanced optimization.
- Design criteria: The proposed criteria are dynamic define-by-run search spaces, efficient customizable sampling and pruning, and versatile minimal-setup deployment from interactive experiments to distributed computation.
- Optuna: Optuna demonstrates these criteria as open-source software, presenting new design techniques and optimization algorithms intended to improve performance and ease of use across environments.
2 Define-by-run API
Optuna’s define-by-run API lets users dynamically construct hyperparameter search spaces while an objective function is evaluated. This design supports expressive Python-defined spaces and modular optimization code.
- Define-by-run principle: Define-by-run means users dynamically construct the search space during optimization rather than specifying it beforehand.Optuna uses the term for an optimization-framework design that allows dynamic search-space construction.
- Runtime construction: Optuna builds search spaces through trial-object methods invoked inside the objective function at runtime.Users invoke suggest API methods to generate hyperparameters for each trial.
- Runtime construction: Suggest API calls sample hyperparameters statistically from the history of previously evaluated trials.The sampled values can define neural-network architecture parameters such as layer counts and hidden-unit counts.
- Expressive parameter spaces: Ordinary Python loops and conditional statements let users represent a wide variety of parameter spaces, including heterogeneous model spaces.Figure 3 shows simultaneous exploration of random-forest and MLP parameter spaces.
- Modularity: The define-by-run design makes optimization code highly modular, allowing model topology and optimizer hyperparameters to be optimized through independent methods.Independent methods can be changed separately and augmented with additional conditional variables, methods, and model choices.
3 Efficient Sampling and Pruning Mechanism
Optuna addresses cost-effective hyperparameter optimization through efficient sampling and pruning mechanisms. It supports relational and independent sampling and uses API-driven, asynchronous pruning based on trial performance.
- Design goals: The section frames optimization cost-effectiveness around the efficiency of searching strategies and performance estimation strategies that identify trials to investigate or discard.The authors present Optuna’s designs for both sampling and pruning because existing frameworks often lack efficient pruning strategies.
- Sampling: Optuna provides both relational sampling, which exploits parameter correlations, and independent sampling, whose cost-effectiveness depends on the environment and task.The paper notes that independent sampling can perform well without parameter correlations, as with TPE.
- Pruning: Pruning monitors intermediate objective values periodically and terminates trials that fail a predefined condition through Optuna’s report and should prune APIs.The report API handles monitoring, while should prune API handles premature termination.
- Pruning: Asynchronous Successive Halving lets workers aggressively stop trials using provisional rankings without waiting for other workers’ results at each pruning round.This enables multiple trials to run in parallel without delay and suits distributed environments.
- Pruning: Optuna implements pruning based on Successive Halving, allowing a trial to advance when its provisional ranking is within the top 1/η.The algorithm uses the current rung, a reduction factor η, minimum resource, and minimum early-stopping rate.
4 Scalable and versatile System that is Easy to setup
Optuna is designed as a scalable, versatile system supporting both lightweight interactive experiments and distributed computation. Its storage design reduces setup effort, supports container orchestration, and scales linearly with worker count.
- Scalability and versatility: Optuna supports tasks ranging from massive-worker experiments to lightweight, trial-level computations through interfaces such as Jupyter Notebook.Trial objects share objective-function evaluation histories through storage.
- Lightweight setup: When no storage is specified, Optuna automatically uses a built-in in-memory data structure, simplifying lightweight local experiments.This avoids requiring users to access an organizational multi-tenant system or deploy a database themselves.
- Distributed computation: Distributed computation can use a relational database or SQLite backend by running run.py multiple times with the same study identifier and storage URL.This design enables multiple workers to share the same optimization study.
- Deployment and scaling: Distributed computation scales linearly with the number of workers, while Optuna reduces storage-deployment effort, supports Kubernetes, and installs with one command.The system’s flexible design enables incorporation into container-orchestration systems.
5 Experimental Evaluation
The experiments evaluate Optuna’s optimization algorithms, pruning procedure, and distributed scalability. Results show competitive sampling performance, substantial trial-throughput gains from pruning, and linear scaling with workers through parallelized trials.
- Sampling performance: TPE+CMA-ES was worse than random search in 1/56 cases, Hyperopt in 1/56, and SMAC3 in 3/56, while GPyOpt was better in 34/56 cases.TPE+CMA-ES also took an order-of-magnitude less time per trial than GPyOpt.
- Sampling performance: GPyOpt required twenty times longer to complete a study, whereas TPE+CMA-ES, Hyperopt, SMAC3, and random search finished studies within a few seconds.The comparison included test cases with more than ten design variables.
- Pruning performance: 1278.6 trials per study were explored by TPE with pruning, compared with 35.8 without pruning; 1271.5 of the pruned trials were pruned during the study.Each study used one NVIDIA Tesla P100 card, ran for four hours, and was repeated 40 times.
- Pruning performance: Pruning significantly accelerated optimization for both TPE and random search, and ASHA significantly outperformed Median pruning.The experiments used a simplified AlexNet with eight hyperparameters on the SVHN dataset.
- Distributed scalability: Optimization convergence speed increased with 1, 2, 4, and 8 distributed workers, while optimization scores per trial barely changed with worker count.This indicates linear scaling with the number of trials and, consequently, with the number of workers.
6 Real World Applications
Optuna has been successfully applied in production and real-world projects, including machine-learning competitions and non-machine-learning optimization tasks. Applications at Preferred Networks demonstrate its use for object detection, high-performance computing, storage configuration, and multimedia encoding.
- Machine-learning applications: Optuna supported Preferred Networks’ Faster-RCNN development for the Google AI Open Images Object Detection Track 2018, where PFDet won second place.
- Non-machine-learning applications: Optuna was applied to High Performance Linpack, a dense-matrix LU-factorization benchmark used to measure computing power and rank TOP500 supercomputers.HPL involves many hyperparameters.
- Non-machine-learning applications: 30 seconds: with pruning, Optuna found a RocksDB parameter set reducing computation time from 372 seconds under default settings.The search explored 34 of RocksDB’s over hundred user-customizable parameters on 500,000 10KB files.
- Non-machine-learning applications: Optuna found an FFmpeg encoding parameter set performing on par with the developers’ second-best preset for the Blender Open Movie Project’s “Big Buck Bunny.”
7 Conclusions
Optuna’s efficacy supports adopting its design criteria for next-generation optimization frameworks. These criteria include dynamic search-space construction through define-by-run and improved cost effectiveness through efficient searching and pruning.
- Optuna’s efficacy supports adopting its new design criteria for next-generation optimization frameworks.
- The define-by-run principle enables users to dynamically construct search spaces in ways previous hyperparameter tuning frameworks could not.
- Combining efficient searching and pruning algorithms greatly improves the cost effectiveness of optimization.