Source-linked AI summary
SMT 2.0: A Surrogate Modeling Toolbox with a focus on Hierarchical and Mixed Variables Gaussian Processes
Paul Saves, Remi Lafage, Nathalie Bartoli, Youssef Diouane, Jasper Bussemaker, Thierry Lefebvre, John T. Hwang, Joseph Morlier, Joaquim R. R. A. Martins
TL;DR
SMT 2.0 addresses the need for surrogate models that support mixed and hierarchical variables in complex engineering design settings. It extends Kriging, sampling, surrogate models, and related applications, with examples showing differentiated performance among mixed kernels and hierarchical methods. The release is positioned as an open-source toolbox for these variable types and associated surrogate-modeling workflows.
Problem
Complex engineering design tasks increasingly involve mixed variables and hierarchical choices, which require specialized surrogate-modeling techniques.
Method
SMT 2.0 uses a unified framework for mixed-variable Kriging kernels and adds hierarchical-variable Kriging, design-space support, sampling extensions, and other surrogate-modeling capabilities.
Results
SMT 2.0 provides mixed and hierarchical surrogate-modeling capabilities, with EHH and HH yielding lower displacement error and higher likelihood than GD and CR in one test case, at greater computational cost.
Takeaways & Limitations
SMT 2.0 offers an open-source Python toolbox for building surrogate models with continuous, mixed, and hierarchical variables.
Abstract
from arXiv · showhide
The Surrogate Modeling Toolbox (SMT) is an open-source Python package that offers a collection of surrogate modeling methods, sampling techniques, and a set of sample problems. This paper presents SMT 2.0, a major new release of SMT that introduces significant upgrades and new features to the toolbox. This release adds the capability to handle mixed-variable surrogate models and hierarchical variables. These types of variables are becoming increasingly important in several surrogate modeling applications. SMT 2.0 also improves SMT by extending sampling methods, adding new surrogate models, and computing variance and kernel derivatives for Kriging. This release also includes new functions to handle noisy and use multifidelity data. To the best of our knowledge, SMT 2.0 is the first open-source surrogate library to propose surrogate models for hierarchical and mixed inputs. This open-source software is distributed under the New BSD license.
1. Motivation and significance
SMT 2.0 extends an open-source surrogate-modeling toolbox to address mixed and hierarchical variables, while adding broader modeling, derivative, sampling, and software capabilities.
- 1. Motivation and significance: Mixed and hierarchical variables represent design settings in which discrete choices coexist with continuous variables or activate different variable sets.Architectural choices can involve technologies, materials, connections, or component counts, and different choices can produce different design-variable sets.
- 1. Motivation and significance: SMT 2.0 adds mixed-variable surrogate models and hierarchical-variable support within Kriging models.The release implements mixed-variable kernels through a unified framework and introduces Kriging methods for hierarchical variables.
- 1. Motivation and significance: SMT 2.0 implements CR, GD, HH, and EHH mixed-variable kernels, including more general kernels that require more hyperparameters but can yield more accurate Kriging surrogates.The release uses a unified implementation for these kernels; EHH and HH can reduce displacement error and raise likelihood at higher computational cost than GD and CR.
- 1. Motivation and significance: The release preserves SMT’s emphasis on derivatives while adding sampling procedures, surrogate models, Kriging kernel and variance derivatives, and an adaptive criterion for high-dimensional problems.SMT’s derivative support includes prediction, training, and output derivatives.
- 1. Motivation and significance: SMT 2.0 retains the toolbox’s modular open-source architecture and expands it with model applications, interactive notebooks, improved documentation, and automated testing.The toolbox remains organized around sampling methods, problems, and surrogate models, while the updated architecture adds new sections and development infrastructure.
Hierarchical and mixed design space.
SMT 2.0 extends design-space sampling and surrogate-model options for mixed and hierarchical problems, and demonstrates applications including efficient global optimization.
- Hierarchical and mixed design space.: SMT adapts Latin Hypercube Sampling to multifidelity, mixed, and hierarchical variables and adds a criterion for managing sampling randomness.Random and Full Factorial sampling remain available alongside LHS.
- Hierarchical and mixed design space.: The toolbox includes mixed cantilever-beam and hierarchical neural-network engineering problems as examples.These problems demonstrate the capabilities of the updated surrogate-modeling toolbox.
- Hierarchical and mixed design space.: SMT 2.0 adds gradient-enhanced neural networks and marginal Gaussian process models to its available surrogate methods.These models extend the surrogate options relative to the original publication.
- Hierarchical and mixed design space.: SMT 2.0 applies efficient global optimization to expensive black-box problems using selected surrogate models and optimization criteria, including hierarchical and mixed variables.The paper describes EGO applications for these variable types and provides a dedicated notebook for reproducing hierarchical and mixed-variable results.
Interactive notebooks.
SMT 2.0 provides unified Kriging support for mixed variables, including categorical kernels and a beam-design validation study comparing their accuracy and computational costs.
- Mixed Gaussian processes: The categorical correlation matrix is estimated using kernel-based methods or direct correlation modeling, depending on the selected approach.The reviewed approaches include continuous relaxation, latent variables, Gower distance, and models that directly parameterize correlation entries.
- Mixed Gaussian processes: Four categorical kernels—GD, CR, EHH, and HH—are available in SMT 2.0 through a unified formulation.Continuous and ordinal variables use continuous kernels, while categorical variables select among four kernel models through the API.
- An engineering design test-case: The beam study trains the models with 98 Latin-hypercube points and validates them on a 10,800-point grid using error, likelihood, hyperparameter, and timing metrics.The test problem uses 12 possible cross-sections and continuous beam-design variables.
- An engineering design test-case: In the cantilever-beam validation, EHH and HH achieve lower displacement error and higher likelihood than GD and CR, but require more hyperparameters and computational cost.EHH is easier to optimize than HH in this test case, while their performance is generally similar.
4. Surrogate models with hierarchical variables in SMT 2.0
SMT 2.0 introduces Kriging models for hierarchical and mixed variables, using a framework that separates neutral, meta, and decreed variables and a new Alg-Kernel for hierarchical inputs.
- 4.1. The hierarchical variables framework: SMT 2.0 implements Kriging models for hierarchical variables through a framework that classifies variables as neutral, meta, or decreed and supports mixed variable types.The framework also distinguishes categorical, ordinal, and continuous variables.
- 4.1. The hierarchical variables framework: In the MLP example, the number of hidden layers is a meta variable, while per-layer neuron counts are decreed because their inclusion depends on that choice.Learning rate, momentum, activation function, and batch size are neutral variables unaffected by the hierarchy.
- 4.1. The hierarchical variables framework: The hierarchical input space is decomposed into neutral variables, meta variables controlling subspace inclusion, and decreed variables activated or excluded by the meta values.For the MLP, meta variables encode the number of layers, while neutral variables include learning rate, momentum, activation function, and batch size.
- 4.2. The hierarchical Kriging model: SMT 2.0 introduces Alg-Kernel, an algebraic correlation kernel for hierarchical variables that avoids extra hyperparameters and natural normalization while allowing distinct meta and decreed kernels.The kernel uses a non-stationary algebraic distance that does not require rescaling by variable bounds.
- 4.2.4. Illustration on the MLP problem: For the MLP problem, Alg-Kernel uses 10 hyperparameters, compared with 12 for Arc-Kernel and 15 for Wedge-Kernel.With up to 10 hidden layers, the corresponding counts are 17 for SMT 2.0, 26 for Arc-Kernel, and 36 for Wedge-Kernel.
- 4.3. A neural network test-case using: On a hierarchical-variable test problem, SMT Alg-Kernel achieved better RMSE performance than the imputation method and SMT Arc-Kernel, with similar computational times because all methods used the same number of hyperparameters.The comparison used 99 training points and a 3,000-point validation set.
5. Bayesian optimization within SMT 2.0
SMT 2.0 uses Kriging models that support mixed and hierarchical variables within EGO, avoiding subproblem-based approaches for these designs. On the mixed toy problem, the more complex methods converged faster and approached the true optimum more closely.
- Bayesian optimization within SMT 2.0: SMT 2.0 enables EGO to optimize problems with mixed and hierarchical variables directly through compatible Kriging models.Other software often handles these cases by solving continuous or non-hierarchical subproblems, an approach described as less efficient and poorly scaling.
- A mixed optimization problem: Both SMT HH and SMT EHH converged in around 18 evaluations, compared with around 26 iterations for SMT CR and SMT GD.The comparison used 20 runs, five initial LHS points per run, and a budget of 55 infill points on a toy problem with one continuous and one 10-level categorical variable.
- A mixed optimization problem: The more complex EGO models produced solutions closer to the real optimum than the simpler models on the mixed test case.Figure 6a shows convergence curves and Figure 6b shows boxplots of the 20 best solutions after 20 evaluations.
- Optimization examples: SMT 2.0 illustrates mixed-variable EGO on a mixed Branin problem and hierarchical-variable EGO on a modified Goldstein function.The hierarchical Goldstein experiment used 11 variables spanning continuous, integer, and categorical types, with four subproblems determined by a meta variable.
- A hierarchical optimization problem: The hierarchical Goldstein study compares SMT's method with random search and an imputation-based Kriging baseline across 20 runs.Each run used 12 initial LHS points and 55 infill evaluations.
6. Other relevant contributions in SMT 2.0
Beyond hierarchical and mixed-variable Kriging, SMT 2.0 expands sampling and surrogate-model capabilities. Its sampling additions include nested, expandable, mixed-variable, and hierarchical-variable designs, while Kriging gains new kernels and derivatives.
- Release organization: These improvements are organized across SMT's Sampling, Problems, Surrogate Models, Applications, and Notebooks sub-modules.The release section presents the changes as contributions beyond hierarchical and mixed-variable Kriging.
- Sampling: SMT 2.0 adds NestedLHS for multi-fidelity surrogates, expand lhs for enlarging designs, and sampling procedures for mixed and hierarchical variables.The expand lhs method increases design size while maintaining the ese property.
- Surrogate models: SMT 2.0 implements four Kriging correlation kernels and their gradients and Hessians.The kernels are absolute exponential, Gaussian, Matern 5/2, and Matern 3/2.
- Surrogate models: Variance derivatives extend SMT's derivative support for uncertainty quantification and system analysis.The extension covers Kriging variances and kernels.
Noisy Kriging.
SMT 2.0 extends surrogate modeling to noisy data, including noisy Kriging and noisy multi-fidelity Kriging for data-fusion settings. It also supports heteroscedastic noise and source-specific noise levels.
- Noisy Kriging: SMT 2.0 accommodates noisy Kriging and noisy MultiFidelity Kriging for engineering and data-fusion applications.The toolbox is designed to handle different noise levels for each data source.
- Noisy Kriging: Heteroscedastic noise can be incorporated through the use het noise option.This capability is presented as part of SMT 2.0's support for noisy data.
- High-dimensional surrogate models: KPLS and KPLSK reduce the effective input dimension by projecting data into a smaller space before incorporating that information into the Kriging correlation matrix.This reduces the number of hyperparameters required for high-dimensional problems.
Kriging with partial least squares.
SMT 2.0 implements Marginal Gaussian Process models for high-dimensional problems by representing the target function in a lower-dimensional linear embedding.
- Kriging with partial least squares: MGP models assume the function lies in a linear embedding of dimension d_e much smaller than the original dimension d.The kernel is defined through the projected coordinates Ax, using k(x, x') = k_A(Ax, Ax').
- Kriging with partial least squares: MGP accounts for uncertainty in hyperparameters through a probability-density representation.The model is formulated for functions on a d-dimensional design domain.
Marginal Gaussian process.
SMT 2.0 extends the toolbox with new surrogate-modeling capabilities, sampling criteria, and support for reproducible collaboration. The release includes gradient-enhanced neural networks and parallel optimization criteria alongside its broader Kriging-focused upgrades.
- Gradient-Enhanced Neural Networks train on both response and partial-derivative errors to improve accuracy with fewer training points.
- SMT 2.0 includes methods for sampling conditioned Gaussian processes, addressing computational and numerical issues associated with large covariance matrices.
- Five parallel optimization criteria are implemented in SMT 2.0 for batch Efficient Global Optimization.Parallel optimization extracts multiple new sampling points for evaluation in a computing environment.
- SMT 2.0 adds hierarchical and mixed-variable support while improving Kriging models and providing an open-source, documented platform for reproducible collaboration.The release is freely available and supports exchange of methods between users and developers.
Appendix A. Toy test function
Appendix A provides the detailed toy test function used in Section 5.1 and introduces its defining expression.
- Appendix A details the toy function used in Section 5.1 and begins by recalling the associated optimization problem.
- The toy function f is defined in the appendix as the test function for this optimization setup.
Appendix B. Hierarchical Goldstein test function
Appendix B specifies the hierarchical Goldstein test function used in Section 5.2. The function combines hierarchical components, mixed variants, and a continuous Goldstein component.
- The hierarchical and mixed function f depends on f0, f1, f2, and Goldcont.
- The component functions f0, f1, and f2 are defined as mixed variants of Goldcont.
- Goldcont is defined as a continuous component using x1, x2, x3, x4, z3, z4, x5, and w2.