Source-linked AI summary
Isotonic surrogate modeling for computer experiments with many input variables
Jaehoan Kim, Simon Mak
TL;DR
High-dimensional computer experiments are costly, and existing Bayesian isotonic surrogates can face computational and statistical challenges. TAIM combines a data-estimated monotone link with additive monotone components and spike-and-slab basis-weight priors. Under transformed-additive truth with mild smoothness, its posterior contraction rate is O(n^-1/3) up to log factors, with terms involving n independent of input dimension.
Problem
High-dimensional computer experiments are costly, while existing Bayesian isotonic models can face computational and predictive limitations with limited data.
Source: §1 · pp. 2, 4Method
TAIM uses a data-estimated monotone link over an additive form with monotone basis functions and spike-and-slab priors on basis weights.
Source: §1 · pp. 4–5Results
O(n^-1/3) up to log factors is TAIM's posterior contraction rate under transformed-additive truth with mild smoothness, with terms involving n independent of d.
Source: §1 · p. 5, §3.2.2 · p. 16Takeaways & Limitations
TAIM's Gibbs sampler requires work linear in d per iteration, and an extension models potential deviations from transformed additivity.
Source: §1 · p. 5Existing projection-based and derivative-constrained isotonic alternatives are computationally costly in high dimensions, with derivative-point counts growing exponentially in d.
Source: §2.1 · pp. 6–7
Abstract
from arXiv · showhide
Virtual simulators are widely used for studying complex physical phenomena, from particle collisions to rocket propulsion. Such "computer experiments" can be highly time-intensive, and a Bayesian surrogate model can be used for efficient emulation with reliable uncertainty quantification. To train accurate surrogates with a limited sample size $n$, recent work has explored the incorporation of monotonicity (or isotonicity) information, which can often be elicited from physical systems. In practical applications with many input variables, however, existing Bayesian isotonic models can face statistical and computational limitations, which may result in worse performance compared to models that do not incorporate isotonicity. We propose a new transformed additive isotonic model (TAIM), which aims to tame this "curse-of-dimensionality". TAIM makes use of a flexible transformed additive isotonic modeling framework, which leverages a data-estimated link transformation and a monotone basis model with spike-and-slab priors on basis weights. Prediction-wise, TAIM achieves (up to log factors) a posterior contraction rate of $O(n^{-1/3})$ when the true black-box function is in a transformed additive isotonic form with mild smoothness conditions. Such a rate does not depend on the input dimension $d$ for terms involving $n$, which softens the effect of dimensionality on posterior predictions. Computation-wise, TAIM allows for efficient posterior inference via a carefully designed Gibbs sampler, where each sampling iteration requires only linear work in $d$. We further present an extension of TAIM that can model potential deviations from transformed additivity. Numerical experiments and two applications show the effectiveness of TAIM for isotonic surrogate modeling with many input variables.
1 Introduction
High-dimensional computer experiments are costly, while existing Bayesian isotonic surrogates can face computational and predictive limitations. TAIM combines a data-estimated monotone link with additive monotone components to address these challenges.
1 Introduction
- Physical intuition can supply isotonicity information: for a dropping ball, height decreases as time increases, and isotonicity means monotonic increase on designated inputs.Prior surrogate approaches include conditioning on nonnegative derivatives at discrete points and basis expansions with constrained weights.Source: §1 · p. 3
- High-dimensional simulations can require thousands of CPU hours, while Gaussian-process surrogates can predict poorly as input dimension grows and isotonic models face computational and statistical limits.The limitations may leave isotonic models performing worse than models without isotonicity in realistic applications.Source: §1 · pp. 2, 4
- TAIM models a latent additive sum through a data-estimated monotone link, using monotone basis functions and spike-and-slab priors on basis weights.Monotonicity over designated inputs is ensured by monotone component functions and the monotone link.Source: §1 · pp. 4–5
- TAIM achieves posterior contraction at O(n^-1/3) up to log factors under a transformed additive truth with mild smoothness, and its Gibbs sampler requires work linear in d per iteration.The rate's terms involving n do not depend on d, unlike rates for existing Bayesian isotonic models that can deteriorate with dimension.Source: §1 · p. 5
- TAIM also extends to potential deviations from transformed additivity and is evaluated in numerical experiments and two surrogate-modeling applications.Existing additive isotonic structure can be too restrictive in some applications.Source: §1 · pp. 4–5
2 Background and Motivation
The section sets up Bayesian isotonic surrogate modeling and reviews why existing approaches can become costly or statistically challenging as input dimension grows. In the motivating aircraft-wing experiment, the existing isotonic models have higher median error than a standard GP, while the proposed models achieve better predictions.
2 Background and Motivation
- The setup treats simulator outputs as noisy observations on the unit hypercube and focuses on Bayesian isotonic surrogates that provide uncertainty quantification.The paper adopts the noisy setting to match its applications and existing isotonic-model literature.Source: §2 · pp. 5–6
2.1 Projection-Based Approaches
- Projection-based methods require posterior paths over a grid, making sampling costly in high dimensions and potentially forcing coarse discretization that degrades predictions.For projection-based sampling, the passage gives a cost of O(N^3d) per projected path; the model's reported contraction rate is O(n^-1/(2+d)).Source: §2.1 · pp. 6–7
- Monotone-constrained GPs condition on non-negative derivatives at finite points, but ensuring monotonicity across the domain requires exponentially more points as dimension grows.The passage notes promise in low-dimensional surrogate modeling applications.Source: §2.1 · p. 7
2.2 Basis Expansion Approaches
- Basis-expansion isotonic models use knot-controlled basis functions and constrained weights, but their basis count grows multiplicatively across dimensions, making high-dimensional MCMC intractable at fine resolution.With 10 knots per dimension in d = 10, the passage gives 10^10 basis functions; smaller knot counts may worsen predictions.Source: §2.2 · pp. 7–8
2.3 Motivating Experiment
- In the aircraft-wing experiment, existing isotonic models had worse median error than the standard GP on both metrics, while the proposed models achieved better predictions.The authors attribute the existing models' performance likely to coarse discretization under the MCMC time constraint.Source: §2.3 · p. 9, Figure 1 · p. 9
3 Transformed Additive Isotonic Model
TAIM models an isotonic response as a monotone transformation of additive component functions, with the link estimated from data. Under mild smoothness conditions, its posterior contraction rate is O(n^-1/3) up to logarithmic factors in both the one-dimensional and transformed-additive multidimensional settings.
3.1 Modeling Framework
- TAIM applies a data-estimated monotone link to additive component functions, extending transformed-additive modeling to high-dimensional isotonic modeling.Unlike generalized additive shape-constrained models described here, TAIM does not presume a fixed link.Source: §3.1 · p. 10
- Monotone piecewise-linear basis functions and nonnegative basis weights enforce monotonicity, while spike-and-slab priors are assigned to the weights.For inputs outside the monotone set, the component functions need not be monotone.Source: §3.1 · pp. 11–12
3.2.1 1-d Setting
- O(n^-1/3(log n)^1/3) is the one-dimensional contraction rate for the monotone basis model when the true function is Hölder-smooth.This matches the one-dimensional isotonic minimax rate up to logarithmic factors.Source: §3.2.1 · p. 14
3.2.2 Multi-dimensional Setting
- The multidimensional rate result requires discretization levels meeting smoothness-dependent lower bounds; conservative smoothness bounds can retain the rate but increase computation.In practice, the authors also choose discretization levels to meet an MCMC time constraint.Source: §3.2.2 · pp. 15, 17
- O(n^-1/3(log n)^1/3) is the multidimensional contraction rate under transformed-additive isotonic structure and mild smoothness, with no d-dependence in terms involving n.The theorem assumes a Lipschitz link and Hölder-smooth component functions; the prescribed discretization levels depend on their smoothness.Source: §3.2.2 · pp. 15–16
4 Posterior Computation
The section extends TAIM with a GP discrepancy for departures from transformed additivity and develops Gibbs sampling for posterior inference. Its sampler has closed-form updates for most parameters and work linear in input dimension when other quantities are fixed.
4.1 Transformed Approximate Additive Isotonic Model
- TAAIM adds a GP discrepancy δ to represent departures from transformed additivity, but δ means isotonicity is no longer guaranteed almost surely.The discrepancy can be checked post hoc; if isotonicity is violated, the model can be refit with a smaller prior distribution on its variance.Source: §4.1 · p. 18
4.2 Gibbs Sampling Steps
- Most parameter full conditionals have closed forms for Gibbs sampling; the GP length-scale ξ is instead sampled with Metropolis–Hastings.The sampler updates link and component basis indicators and weights, noise variance, and—when using TAAIM—the discrepancy and its variance.Source: §4.2 · pp. 19, 21, 22
4.3 Sampling Algorithm and Computational Complexity
- With n, Ng, and common discretization level N fixed, each Gibbs iteration scales linearly in d, unlike existing isotonic models whose posterior sampling can grow exponentially in d.The stated work is O{nNgNd log n + n^3} when every input uses the same discretization level.Source: §4.3 · p. 23
- Posterior predictions at a new input combine a GP predictive draw for δ(xnew) with the current link and additive components, repeated across posterior parameter draws.TAIM uses the same sampler by omitting steps involving δ.Source: §4.1 · p. 18, §4.3 · pp. 23–24
5 Numerical Experiments
Across synthetic tests and two established engineering functions, TAIM and TAAIM perform especially well in higher dimensions, where existing isotonic benchmarks can lose ground to standard GP. The experiments compare log-MSE and CRPS under a common MCMC time constraint.
Experimental setup
- All models use the same LHD points and 20 replications, with at least 2,000 MCMC iterations and a 30-minute sampling limit.The sample size is n = 10d, and the benchmark comparison uses a time constraint that limits discretization for existing isotonic models.Source: §5 · pp. 25–26
Synthetic functions
- In 5-D and 9-D synthetic tests, TAIM and TAAIM perform considerably better than benchmarks on MSE and CRPS, while existing isotonic models fall behind standard GP.At 2-D, existing isotonic models outperform standard GP and TAIM/TAAIM perform comparably to those benchmarks.Source: Figure 4 · p. 26, §5 · p. 26
Borehole and wing-weight functions
- On borehole and wing-weight functions, TAIM and TAAIM outperform all benchmarks on both metrics, with TAAIM slightly ahead of TAIM.Basis-GP gives only slight improvements over standard GP, while Proj-GP performs worse; the authors link TAAIM’s slight edge to likely departures from transformed additivity.Source: Figure 5 · p. 27, §5 · p. 27
6 Surrogate Modeling Applications
In two simulator applications, TAIM and TAAIM perform better than benchmark models on predictive metrics, while existing isotonic models perform worse than standard GP. The applications use physically motivated monotonicity information in six- and eleven-input settings.
6.1 Jet Turbine Deformation
- For the six-input turbine blade simulator, TAIM and TAAIM outperform benchmarks on log-MSE and CRPS, while Proj-GP and Basis-GP perform worse than standard GP.The inputs cover four material properties and two operating conditions; physical analysis suggests deformation increases with pressure drop, temperature drop, and thermal expansion, and as Young’s modulus decreases.Source: §6.1 · pp. 28–29, Figure 6 · p. 28
6.2 Aircraft Wing Deflection
- For the eleven-input wing simulator, TAIM and TAAIM outperform all benchmarks on both metrics and perform similarly; existing isotonic models fall behind standard GP.Dynamic pressure and the ten lift-load multipliers are treated as monotone inputs, with n = 110 training points under the shared setup.Source: §6.2 · pp. 30–31
7 Conclusion
TAIM addresses high-dimensional isotonic surrogate modeling with a transformed additive framework, offering a dimension-independent-in-n contraction rate under stated conditions and Gibbs sampling with work linear in d.
7 Conclusion
- TAIM combines a data-estimated link function with a monotone basis model using spike-and-slab priors on basis weights.The paper also presents an extension for potential deviations from transformed additivity.Source: §7 · p. 31
- Under mild smoothness conditions and a transformed additive isotonic truth, TAIM achieves posterior contraction at O(n^-1/3) up to log factors.The terms involving n in this rate do not depend on input dimension d.Source: §7 · p. 31
- Each Gibbs sampling iteration requires work linear in d, and numerical experiments and two applications show TAIM's effectiveness with many input variables.Source: §7 · p. 31
S1 Proofs of Theorems 1 and 2
This section establishes the one-dimensional and TAIM posterior contraction results through small-ball and sieve-based arguments, verifying approximation, prior-mass, sieve-tail, and entropy conditions. It also notes that rescaling to general prior parameters changes only rate constants.
S1.1 Small ball probability bound
- For monotone g0 in Cα[0, 1], Lemma S1 gives a small-ball probability lower bound when 0 < ϵ ≤ 1 and N ≥ C1ϵ^-1/(α∧1), with G2 at least the endpoint range.The bound supports the subsequent one-dimensional contraction proof.Source: S1.1 Small ball probability bound · pp. 37–38
S1.2 Proof of Theorem 1
- For fixed-design Gaussian regression, KL-ball, sieve-mass, and metric-entropy conditions on the prior suffice for posterior contraction at rate ϵ_n.The proof applies this criterion to the sparse basis model and bounds the metric entropy of its coefficient-constrained sieve.Source: S1.2 Proof of Theorem 1 · pp. 41, 43
S1.3 Proof of Theorem 2
- The TAIM proof constructs a prior-adapted sieve Fk,M and uses lemmas to control its metric entropy and the prior mass of its complement.The proof strategy parallels the one-dimensional argument.Source: S1.3 Proof of Theorem 2 · pp. 45–46
- The proof verifies the posterior-contraction conditions using approximation and prior-mass bounds for the transformed additive components.It applies Lemma S1 to the link and component functions, then bounds the prior mass around the truth using their independence.Source: S1.3 Proof of Theorem 2 · pp. 48–49
- Choosing A > 4λ^2D lets the sieve-tail bound satisfy the required exponential control for sufficiently large n.The proof uses Lemma S6 and the bound log(kn/eζmax) ≥ (log n)/4.Source: S1.3 Proof of Theorem 2 · p. 49
- A covering-number bound for Fk,M verifies the third condition using η = ˜ϵn/8 and the stated growth bounds on N̄ and kn.The proof bounds the log covering number by A′n˜ϵ2, with A′ = 4(d+1)A(c0+1).Source: S1.3 Proof of Theorem 2 · p. 49
S2 Details on Posterior Computation
The supplement details the Gibbs sampler's full conditional distributions, an on-the-fly normalization adjustment, computational complexity, and prior hyperparameter specification.
S2.1 Derivation of the full conditional distributions
- For the magnitude parameter, breakpoints divide its domain into intervals where fitted values are linear, yielding a finite mixture of truncated normals for its update.The link-function coefficients reduce to a Gaussian linear model conditional on the link arguments.Source: S2.1 Derivation of the full… · pp. 50–52
S2.2 On-the-fly normalization
- On-the-fly normalization improves parameter identifiability and can improve MCMC mixing, while potentially changing the posterior distribution under the constraint.The adjustment preserves the sampled function at the link-function knots; the authors report that the distributional difference is typically minimal in practice.Source: S2.2 On-the-fly normalization · pp. 53–54
S2.3 Computational complexity
- The stated total work is O(nNg(sum_l N_l) log n), while updates for parameters in Θδ require O(n^3) work.The Θδ updates' cubic cost is due to a fresh covariance factorization when a proposal changes K.Source: S2.3 Computational complexity · p. 55
S2.4 Specification of prior hyperparameters
- For limited-data applications, the prior hyperparameters are data-dependent; the prior variance for the link intercept α0 is set to Var(y)/n, focusing it on the data minimum.The passage states that this prior concentrates as n grows.Source: S2.4 Specification of prior… · p. 55