mlrMBO: A Modular Framework for Model-Based Optimization of Expensive Black-Box Functions

1. Introduction

• Black-box functions are often costly to evaluate and lack known internal workings; SMBO fits a regression model, proposes a point via an infill criterion, evaluates it, and repeats.The approach aims to find optimal settings with a minimal number of tries.Source: §1 · pp. 1–2

1.1. Related Software

• DiceOptim, rBayesianOptimization, and Spearmint [14] offer EGO implementations but support only numerical, non-conditional parameters; only Spearmint supports multi-criteria optimization and parallelization.A multi-criteria version of Spearmint is introduced in [15].Source: §1.1 · pp. 2–3
• BayesOpt [16] supports mixed and conditional parameters, while SMAC handles mixed spaces with random forests but is limited to single-criteria optimization without parallelization.BayesOpt includes Student-t processes and meta-criteria algorithms for selecting infill criteria.Source: §1.1 · p. 3
• Hyperopt supports numerical, categorical, and conditional parameters with distributed asynchronous execution, while SPOT offers noisy-function handling, constraints, and multi-objective optimization.Hyperopt uses tree-of-Parzen-estimators suggestions; SPOT offers a variety of modeling techniques and statistical methods.Source: §1.1 · p. 3

1.2. Main Contributions and Prior Applications

• mlrMBO is a modular R SMBO framework with plug-and-play components, over 60 regression learners via mlr, and support for mixed spaces, multi-point proposals, and multi-objective optimization.The paper also describes interfaces for user-specific extensions.Source: §1.2 · p. 3
• Benchmarks show mlrMBO achieves state-of-the-art performance across mixed-space, multi-point, and multi-objective optimization domains.The framework supports combinations of these domains, which the authors say no other software then supported.Source: §1.2 · p. 3
• Prior applications of mlrMBO include machine-learning pipeline tuning, surrogate-model selection, runtime-accuracy trade-offs, and studies of multi-objective tasks and initial-design sampling.These applications are described in,,,, and, respectively.Source: §1.2 · pp. 3–4

2. Sequential Model-Based Optimization

• SMBO’s modular setup allows its components to be customized for different optimization tasks.The section presents the individual building blocks in Subsection 2.1.Source: §2 · p. 4
• The section highlights literature combinations including EGO and SMAC-like optimizers.These are covered in Subsections 2.2 and 2.3, respectively.Source: §2 · p. 4
• The section introduces multi-point proposal for parallelization and multi-objective optimization.These topics are covered in Subsections 2.4 and 2.6.Source: §2 · p. 4

2.1. Sequential model-based optimization

• SMBO fits a surrogate to evaluated points, proposes promising new points, evaluates them, and repeats until the budget or another stopping criterion is met.Infill proposals seek either good expected objective values or potential to improve the surrogate.Source: §2.1 · pp. 4–5

2.1.1. Initial Design

• A poor or oversized initial design can respectively impair the surrogate or consume too much budget, so mlrMBO supports manual, random, coarse-grid, and space-filling Latin hypercube designs.Initial points are evaluated to provide the data for the first surrogate.Source: §2.1 · p. 4, §2.1.1 · p. 5

2.1.2. Surrogate Models

• For numeric spaces, Kriging is recommended; random forests handle categorical inputs directly, and mlrMBO also supports other mlr regression learners and bagging-based uncertainty estimates.Kriging and random forests provide uncertainty estimates natively.Source: §2.1.2 · p. 5

2.1.3. Infill Criteria

• Infill criteria balance predicted mean and uncertainty to trade off exploitation and exploration; mlrMBO also supports pure mean minimization, pure uncertainty maximization, and criteria for batch, noisy, and multi-objective optimization.The mean and uncertainty are estimated by the surrogate model.Source: §2.1.3 · p. 6

2.1.4. Infill Optimization

• The default focus search optimizes the infill criterion with random designs and iterative search-space shrinking, handling numeric, categorical, mixed, and hierarchical spaces; CMA-ES or custom optimizers are alternatives.The search can be restarted to reduce the risk of local optima.Source: §2.1.4 · pp. 6–7

2.1.5. Termination

• Termination can be based on evaluation or iteration limits, elapsed time, a time budget, a target objective value, or user-defined rules.Source: §2.1.5 · p. 7

2.1.6. Final Point

• The final solution is usually the best observed point, though a last surrogate fit can select the best predicted point, especially when the objective is noisy.Source: §2.1.6 · p. 7

2.2. Efficient Global Optimization (EGO)

• Kriging models are a popular surrogate choice because they are flexible and provide a local uncertainty estimator.Source: §2.2 · p. 8
• Jones et al. introduced EGO for sequential optimization of box-constrained functions with real-valued inputs, using Kriging models and expected improvement (EI).The section considers a numeric-only input domain.Source: §2.2 · p. 8
• Maximizing EI balances exploitation of the model structure with exploration of highly uncertain regions and can ensure global convergence, though that is often unrealistic under tight budgets.The passage describes EI as highly effective for expensive black-box optimization.Source: §2.2 · p. 8
• Figure 2 illustrates how high uncertainty and low predicted values contribute to EI and guide EGO’s next-point selection on a multimodal cosine mixture function.The figure shows the third and fourth iterations and the corresponding EI optima.Source: §2.2 · p. 8, Figure 2 · p. 8

2.3. Mixed Space Optimization

• In an SVM search space, γ is active only for the radial kernel while C is always active; Kriging is poorly suited to such domains because suitable covariance kernels remain under research.Source: §2.3 · pp. 8–9
• All initial-design options support categorical parameters and hierarchical dependencies, where feasible parameter values depend on other parameter settings.Source: §2.3 · p. 9
• A modified random forest surrogate handles categorical features and inactive conditional parameters by coding missing categorical values as a new level and numerical values outside the parameter bounds.This separate-class imputation was reported to perform best for decision trees when missingness is related to the outcome.Source: §2.3 · p. 9
• The jackknife is mlrMBO’s default random-forest uncertainty estimator for LCB and EI, although its properties are less intuitive than those of Kriging models.Other options include an extra bootstrap, the infinitesimal jackknife, or a standard-error estimator from the forest’s internal bootstrap.Source: §2.3 · p. 9

2.4. Multi-Point Proposal

• qLCB draws m values of λ from an exponential distribution, optimizes the criterion separately for each, and balances exploitation with exploration across proposals.Low λ values yield exploitative proposals near the best observed outcome, while high values encourage exploration.Source: §2.4 · p. 10
• Constant liar proposes points sequentially by updating the surrogate with a made-up target for each pending evaluation, such as the minimum, maximum, mean, or predicted posterior mean.Using the predicted posterior mean is also called kriging believer.Source: §2.4 · p. 10
• MOIMBO jointly optimizes posterior mean and variance as multiple objectives to propose a set of points, optionally using distance as a third objective to promote diversity.Source: §2.4 · p. 10

2.5. Noisy Optimization

• Noisy optimization targets E[f(x)]; mlrMBO offers adapted infill criteria rather than repetition strategies, though users can average a fixed number of repetitions in the objective.Source: §2.5 · p. 10
• Expected quantile improvement (EQI) replaces EI’s best observed value with qmin, the lowest estimated β-quantile among evaluated points, enabling re-evaluations of promising designs.The estimated quantile is q(x) = μ̂(x) + Φ^-1(β)ŝ(x), with β user-controlled; EQI can be non-zero at evaluated points.Source: §2.5 · pp. 10–11
• mlrMBO also offers augmented expected improvement, and its modular design makes extending the criterion set straightforward; further discussion and benchmarks are referenced in.Source: §2.5 · p. 11

2.6. Model-Based Multi-Objective (MBMO) Optimization

• The goal is to approximate the Pareto set of non-dominated solutions, which represent incomparable trade-offs across objectives.The Pareto front is the image of the Pareto set under the objective functions.Source: §2.6 · p. 11
• MBMO algorithms use scalarization, Pareto-based optimization of individual-model infill criteria, or direct indicator-based aggregation of those models.Scalarization-based approaches apply EGO to randomly weighted scalarized objectives at each iteration.Source: §2.6 · p. 11
• mlrMBO supports ParEGO, MSPOT, SMS-EGO, and ε-EGO, covering all three MBMO algorithm classes.ParEGO is scalarization-based, MSPOT is Pareto-based, and SMS-EGO and ε-EGO are direct indicator-based.Source: §2.6 · p. 11

3. mlrMBO R Package

• mlrMBO lets users combine model-based optimization components, including infill criteria and stopping conditions, in a plug-and-play framework.Its components can be adapted to the characteristics of the optimization problem.Source: §3 · p. 11

Definition of the black-box function

• Black-box functions can use built-in benchmarks or custom constructors, with parameter sets supporting box constraints, transformations, and dependencies.The package uses smoof for a unified interface to black-box functions.Source: §3 · p. 12

Definition of the Initial Design

• If users provide no initial design, mlrMBO generates a maximin Latin Hypercube Design with 4 times the number of function parameters.A Latin Hypercube Design is recommended for specifying initial evaluations.Source: §3 · p. 12

Definition of the surrogate regression model

• Users can select regression learners; EI or LCB requires uncertainty estimates or bagging, with Kriging or random-forest fallbacks depending on parameter types.For fully numeric settings, the fallback is Kriging with a Matern-3/2 kernel; otherwise, it is a random forest with jackknife variance estimation.Source: §3 · p. 12

Definition of the control flow

• Control objects set proposal counts, error handling, infill, and termination; default LCB uses λ = 1 for numeric settings and λ = 2 when a discrete parameter is present.Multi-point and multi-objective controls are available, and parallel evaluation supports multicore, sockets, and MPI.Source: §3 · pp. 12–13

Putting it all together

• The optimization object stores the full optimization path, evaluation runtimes, final state, potential errors, and optionally fitted surrogate models.Diagnostic plots and step-by-step visualizations are available for one- and two-dimensional domains with single- or multi-objective targets.Source: §3 · p. 13

4.1. Model-Based Single-Objective Optimization

• For synthetic functions evaluated in microseconds, mlrMBO was slower than DiceOptim but more than twice as fast as SPOT and orders of magnitude faster than rBayesianOptimization.The authors frame this overhead as a reasonable cost for expensive objectives: at one minute per evaluation, random search would take 200 minutes versus 212 minutes for mlrMBO.Source: §4.1 · p. 15, Table 1 · p. 16
• Across six test functions, mlrMBO ranked best overall, beating random search everywhere, outperforming other EGO methods on four and matching two; SPOT ranked second.CMA-ES was comparable on Alpine01 and slightly better on Rosenbrock, but considerably worse on the other four functions.Source: §4.1 · p. 14, Figure 4 · p. 15, Table 1 · p. 16

4.2. Model-Based Single-Objective Optimization in Mixed Spaces

• On HPOlib, mlrMBO ranked first for numeric and mixed spaces, outperforming SMAC and TPE on all four synthetic tasks and performing exceptionally well on all three grid problems.On neural-network problems, it matched SMAC and slightly beat Spearmint on hpnnet and hpdbnet; it was worse than competitors only on hpdbnet/cv_convex. Spearmint performed clearly worse on three of six mixed-space problems.Source: §4.2 · pp. 16–17, Figure 5 · p. 17, Table 2 · p. 18

4.3. Model-Based Multi-Objective Optimization

• In the multi-objective benchmark, SMS-EGO, ParEGO, and MSPOT beat both baselines on nearly all test functions, with MSPOT superior; GPareto excelled only on separable functions.The study used 55 bi-objective BBOB problems at d = 5, with 44d evaluations including a 4d initial design; hypervolume was the principal performance measure, and mean ranks were computed over 20 replications per test function.Source: §4.3 · p. 18, Table 3 · p. 19

5. Conclusion

• In single-objective benchmarks, mlrMBO achieved state-of-the-art solution quality versus CMA-ES, random search, and alternative SMBO implementations while remaining reasonably fast, and was on par with SMAC, Spearmint, and TPE on HPOlib.The passage describes these findings as benchmark results across different scenarios.Source: §5 · p. 19
• In expensive multi-objective benchmarks, SMBO methods—particularly SMS-EGO—performed excellently, while NSGA-II and random search were outperformed on all nine test functions; ParEGO occasionally failed.The passage identifies random search as the baseline and NSGA-II as a state-of-the-art evolutionary algorithm.Source: §5 · p. 19
• The introduced modular R toolbox is intended for model-based optimization and supports expensive single- and multi-objective tasks with continuous or mixed parameter spaces.The paper also reports an introduction to software-specific aspects and features.Source: §5 · p. 19

• Figures 6 and 7 plot final Pareto-front hypervolume values for the respective algorithms on each test function.Source: Figure 6 · p. 25, Figure 7 · p. 26