Source-linked AI summary
Adaptive design and analysis of supercomputer experiments
Robert B. Gramacy, Herbert K. H. Lee
TL;DR
Expensive computer experiments need designs that avoid both prohibitive dense sampling and inadequate fixed sparse sampling in nonstationary response spaces. The paper uses a Bayesian treed Gaussian process with a hybrid adaptive sequential-design framework for asynchronous parallel supercomputing, and demonstrates the approach in synthetic and rocket-booster CFD examples.
Problem
Dense designs are prohibitively expensive, while fixed sparse designs can provide insufficient information when the response surface is nonstationary.
Method
The approach combines a Bayesian treed Gaussian-process surrogate with predictive-uncertainty-guided adaptive sampling and hybrid optimal-design and active-learning strategies.
Results
The approach is demonstrated through synthetic examples and a computational-fluid-dynamics experiment involving a reusable launch vehicle.
Takeaways & Limitations
The paper presents adaptive sampling as a highly efficient strategy for sequential design of supercomputer experiments in nonstationary settings.
Takeaways & Limitations
Standard GP models use one covariance structure throughout the input space, which is unsuitable when physical regimes differ substantially.
Abstract
from arXiv · showhide
Computer experiments are often performed to allow modeling of a response surface of a physical experiment that can be too costly or difficult to run except using a simulator. Running the experiment over a dense grid can be prohibitively expensive, yet running over a sparse design chosen in advance can result in obtaining insufficient information in parts of the space, particularly when the surface calls for a nonstationary model. We propose an approach that automatically explores the space while simultaneously fitting the response surface, using predictive uncertainty to guide subsequent experimental runs. The newly developed Bayesian treed Gaussian process is used as the surrogate model, and a fully Bayesian approach allows explicit measures of uncertainty. We develop an adaptive sequential design framework to cope with an asynchronous, random, agent--based supercomputing environment, by using a hybrid approach that melds optimal strategies from the statistics literature with flexible strategies from the active learning literature. The merits of this approach are borne out in several examples, including the motivating computational fluid dynamics simulation of a rocket booster.
1 Introduction
The paper addresses the cost and inflexibility of pre-specified designs for complex, nonstationary computer experiments by combining adaptive exploration with response-surface modeling. It develops a Bayesian treed GP framework and hybrid sequential design for asynchronous parallel supercomputing environments.
- Contributions: The proposed framework combines treed modeling, Bayesian GP surrogates, classic sequential design, and active learning into a fast, flexible interface.Its hybrid strategy balances optimality and flexibility while supporting sequential experimentation in asynchronous, parallel computing environments.
- Motivation: Dense grids are often computationally prohibitive, while sparse stationary designs can miss informative regions when responses require nonstationary models.Traditional space-filling approaches use the same metric throughout the input space and may fail to concentrate exploration where behavior is more complex.
- Motivation: The motivating CFD application models lift, drag, pitch, side-force, yaw, and roll from Mach number, angle of attack, and side-slip angle.Each Euler-solver run takes approximately 5–20 hours on a high-end workstation, making adaptive allocation valuable.
- Motivation: The lift surface changes sharply at Mach 1, where a ridge separates subsonic and supersonic regimes and the transition may be nonlinear, nondifferentiable, or discontinuous.These differing physical regimes motivate a surrogate capable of nonstationary behavior.
- Limitations of standard models: Standard Gaussian processes are computationally costly at scale and usually impose one covariance structure across the entire input space.Their predictive error also does not locally depend on previously observed output responses, limiting suitability for this setting.
- Contributions: The paper's primary contributions are an integrated sequential-design strategy for a nonstationary model and methods for asynchronous parallel experiment design.Examples include synthetic data and a computational-fluid-dynamics experiment for a reusable launch vehicle.
2 Review
The review develops treed Gaussian-process surrogate models for nonstationary computer experiments and combines them with Bayesian inference and adaptive sequential-design strategies.
- The approach combines classic design strategies and active learning with modern nonstationary spatial modeling.
- Gaussian-process surrogate modeling: The model includes a positive nugget to represent noise and improve covariance-matrix numerical stability, despite many computer experiments being treated as deterministic.The authors report that the LGBB simulator was theoretically deterministic but not necessarily so in practice, partly because of random solver behavior.
- Treed Gaussian process model: The treed Gaussian process partitions the input space, fitting independent Gaussian processes within regions to represent a globally nonstationary response.The tree makes predictive uncertainty region-specific and can permit discontinuities across partition boundaries.
- Bayesian inference: The framework uses posterior sampling through MCMC to obtain predictions and uncertainty measures from the treed model.Covariance and related parameters are sampled using Gibbs steps except for covariance structure parameters, which use Metropolis–Hastings.
- Treed Gaussian process model: The treed GP retains the capability to model discontinuities when the data indicate a non-smooth process.
- Treed Gaussian process model: Bayesian model averaging yields an adaptively semiparametric model that can be faster, more parsimonious, and numerically stable.Indicator parameters allow the GP or a treed linear model to govern each input dimension within a region.
- Treed Gaussian process model: The treed GP LLM provides a fast divide-and-conquer implementation for nonstationary Gaussian-process modeling.It is especially suited to axis-aligned nonstationarity, although it has also compared favorably in more general situations.
3 Adaptive sequential design
The framework automates sequential exploration and surrogate updating for expensive, asynchronous computer experiments. It combines space-filling candidate generation with treed-GP uncertainty criteria to prioritize informative simulations as computing agents become available.
- 3.1 Asynchronous distributed supercomputing: High-fidelity simulations run asynchronously across independent agents, with an emcee controller collecting completed responses and supplying free agents with new configurations.Simulations can start and finish at different, potentially random times.
- 3.1 Asynchronous distributed supercomputing: The sampler maintains a queue of well-chosen candidates, updates the surrogate with finished responses, and replenishes the queue for subsequent runs.Figure 2 depicts the exchange among the emcee program, sampler, and queue.
- 3.2 Hybrid candidate generation and ranking: The hybrid design first generates well-spaced candidates using maximum entropy, maximin, or Latin hypercube designs, then ranks them using treed-GP uncertainty.The first stage encourages exploration relative to both candidate points and previously sampled locations.
- 3.3 ALM and ALC algorithms: ALM selects candidates with greatest predictive standard deviation, whereas ALC selects those maximizing expected reduction in squared prediction error over the input space.ALM costs O(|X̃|), while ALC costs O(|X̃||Y|).
- 3.4 Choosing candidates: The approach avoids overly dense candidate sets because model refitting may be too slow for asynchronous agents, while still placing informative candidates first in the queue.This supports simulation scheduling when agents become available at different times.
- 3.4 Choosing candidates: A sequential treed maximum entropy design addresses computational and nonstationary-modeling issues by adapting candidate density to the treed partition.In the example, the first quadrant receives nearly twice the sample density of the other regions, unlike a classical maximum entropy design.
4 Illustrative examples
Across synthetic examples, BAS adaptively concentrates samples where the treed GP remains uncertain or structurally complex, while predictive accuracy improves and uncertainty decreases as samples accumulate.
- 4.1 1-d Synthetic Sinusoidal data: After 45 samples, BAS allocated roughly twice as much effort to the sinusoidal region and learned substantially more.
- 4.1 1-d Synthetic Sinusoidal data: After 97 samples, BAS devoted almost three times more sampling effort to the left region and captured its secondary cosine structure.
- 4.1 1-d Synthetic Sinusoidal data: As samples accumulated, the treed GP improved mean prediction and its posterior predictive variance decreased; ALC and ALM differed negligibly overall.
- 4.2 2-d Synthetic Exponential data: Across comparisons, treed and adaptive approaches generally outperformed stationary or non-adaptive alternatives, with BAS at least two times more efficient than LHS in a reported serial study.
- 4.2 2-d Synthetic Exponential data: On exponential data, BAS concentrated 54% of samples in the first quadrant, occupying 25% of the input space, and achieved a very good final approximation.
5 LGBB CFD experiment
BAS was deployed for a three-parameter NASA LGBB CFD experiment with six simultaneous responses, concentrating samples near the most complex near-Mach-one, high-angle-of-attack region.
- Experiment setup: The motivating experiment used a proposed NASA Langley Glide–Back Booster CFD simulator with three varied inputs and six simultaneously computed outputs.
- Experiment setup: The study developed a live sequential design for a fully asynchronous NASA environment with highly variable supercomputer workload.
- Adaptive sampling: BAS sampled 780 configurations, initialized with 30 maximum-entropy candidates and proposing 100 treed maximum-entropy candidates per adaptive round.
- Adaptive sampling: Most selected configurations were near Mach one, with the highest density at large alpha, while samples were scarce above Mach two.
- Posterior predictive surfaces: The MAP partition separated the near-Mach-one region, where samples were densely concentrated, especially for large alpha.
- Posterior predictive surfaces: All six responses were relatively tame in supersonic cases, with the most interesting behavior near Mach 1 and large alpha; BAS reduced simulation burden by more than 75%.
6 Conclusion
The paper develops Bayesian adaptive sampling with a treed GP surrogate for sequential design in asynchronous agent-based supercomputing. It combines classical optimal design and active learning, with ALM and ALC selecting candidate simulation runs using predictive uncertainty.
- The treed GP LLM serves as a surrogate model for sequential design by adapting classic optimal design methodology to Bayesian partition models.
- The framework combines active learning and classical design for a flexible system suited to highly variable asynchronous agent-based supercomputing.
- Two sampling algorithms were proposed: ALM selects configurations with high posterior predictive variance, while ALC targets average global uncertainty reduction.
- Model uncertainty statistics rank optimally spaced candidate locations to determine which configurations should be simulated next.
- The resulting Bayesian adaptive sampling strategy is described as highly efficient and a significant improvement over the state of the art in NASA computer-experiment methodology.
- The methods are implemented in the tgp package for R, with asynchronous supercomputer interface code available upon request.
- The authors identify extensions toward sampling extrema, finding contours and boundaries, handling multi-fidelity codes, and pairing computer with physical experiments.
A Active Learning – Cohn (ALC)
This section derives the ALC criterion for the hierarchical Gaussian process. The derivation supports uncertainty-reduction-based adaptive sampling.
- Section A.1 derives the ALC criterion for the hierarchical Gaussian process.
- The derivation forms part of the paper's methodology for adaptive sampling with hierarchical Gaussian-process models.
- The corresponding derivation for the limiting linear model is deferred to Section A.2.
A.1 For hierarchical Gaussian processes
The appendix derives computational formulas for variance reduction and predictive variance when adding a point to a hierarchical Gaussian-process model. Partitioned inverse identities reduce the update cost.
- Partition inverse equations express the covariance update after adding a new point to the data.
- The resulting calculations can be performed in O(n^2) rather than the usual O(n^3).
- For the hierarchical GP, the derivation applies partitioned inverse equations and covariance structure to simplify the required computations.
- The formulas calculate variance reduction at y when x is added and support predictive-variance evaluation for ALC sampling.
A.2 For hierarchical (limiting) linear models
This section derives ALC computation for hierarchical limiting linear models. The calculation is simpler under the limiting linear-model assumption and uses updated parameter covariance.
- Under the limiting linear model, computing the ALC statistic is more straightforward.
- The parameter-covariance term Ṽβ_N+1 includes the candidate point x, whereas Ṽβ_N does not.
- The Sherman-Morrison-Woodbury formula is used in the limiting linear-model calculation.