PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoQ: Geometry-Aware Conditional Quantile Error Estimation for Scientific Surrogate Models

Khoa Nguyen, Daniel Serino, Aviral Prakash, Marc Klasky

arXiv:2608.21652v1cs.LGstat.ML

TL;DR

Scientific surrogate deployment needs query-specific error estimates beyond distribution-level validation, particularly for extrapolative and autoregressive use. GeoQ combines anchor-based calibration, geometry-aware conditional quantiles, cross-fitting, and support scoring; across four benchmark settings, it achieved competitive or superior calibration-oriented performance, with limitations in calibration cost and fine-scale chaotic localization.

  • Problem

    Standard validation and generalization tools summarize average performance over chosen distributions but do not directly provide input-dependent surrogate-error estimates near or beyond training support.

  • Method

    GeoQ combines an anchor-averaged calibration error with a learned nonnegative upper conditional-quantile correction using geometric features, cross-fitted calibration tuples, and a feature-space support score.

  • Results

    Across Forrester, Kuramoto–Sivashinsky, WeatherBench, and Richtmyer–Meshkov instability, GeoQ achieved competitive or in several cases superior calibration-oriented performance, particularly for pinball loss.

  • Takeaways & Limitations

    GeoQ provides a practical, non-intrusive approach to validity-aware, spatially and temporally resolved error estimation for scientific surrogate models.

  • Takeaways & Limitations

    Calibration can be dominated by training an auxiliary surrogate for each cross-fitting fold, while chaotic rollouts may retain moderate correlation and localized undercoverage.

Abstract

from arXiv · show

Neural-network surrogate models are increasingly used to accelerate scientific simulations, but their deployment in extrapolative and autoregressive settings requires input-dependent estimates of prediction error. In this work, we introduce GeoQ (Geometry-Aware Conditional Quantile Error Estimation), a non-intrusive calibration framework for estimating surrogate error at individual query points. GeoQ represents the error at a query point as an anchor-averaged calibration error plus a learned nonnegative correction. This correction is modeled as an upper conditional quantile of the anchor-relative error increment, using geometry-based features that encode representation-space displacement and local support density. A cross-fitting procedure generates approximately out-of-sample calibration tuples, while a feature-space k-nearest-neighbor support score identifies regions \textcolor{black}{where the learned error model is supported by calibration data}. We evaluate GeoQ on scalar regression, chaotic dynamics, medium-range weather forecasting, and Richtmyer-Meshkov instability prediction. The results demonstrate that geometry-aware conditional quantile modeling provides a practical and non-intrusive approach for validity-aware error estimation in scientific surrogate models.

1. Introduction

GeoQ addresses the lack of input-dependent surrogate-error estimates by combining local anchor errors with geometry-aware conditional quantile modeling. Cross-fitting and support scoring extend calibration toward interpolation and extrapolation queries.

  • Standard validation and generalization tools summarize performance over distributions but do not directly estimate error at individual queries, especially near or beyond training support.
  • GeoQ is a non-intrusive framework that estimates surrogate error at individual query points using calibration data with known reference outputs.
  • GeoQ combines an anchor-averaged observed error baseline with a learned nonnegative correction representing an upper conditional quantile of anchor-relative error increments.
  • Geometry-based features encode query displacement from anchors and local support in a physical or learned representation space.
  • Structured cross-fitting creates approximately out-of-sample calibration errors and query–anchor configurations that mimic interpolation and extrapolation.
  • GeoQ includes an explicit support score and is demonstrated across scalar regression, chaotic dynamics, weather forecasting, and Richtmyer–Meshkov instability prediction.

2. Related work

Related work spans distribution-level validation, uncertainty methods, and geometry- or applicability-domain-aware approaches. GeoQ targets query-specific, support-aware error estimation while remaining non-intrusive and avoiding full Gaussian-process covariance inference.

  • Validation loss, test error, and cross-validation estimate average predictive performance on held-out distributions and help control overfitting.
  • Applicability-domain awareness uses distance or support information relative to training or calibration data, while OOD-risk detection flags unsupported queries.
  • Geometry-, distance-, density-, and novelty-based methods assess whether new queries resemble data used for model construction.
  • Scientific-surrogate uncertainty methods include Bayesian neural networks, ensembles, generative models, conformal prediction, and Gaussian-process or kernel-based approaches.
  • GeoQ preserves geometry-aware similarity to calibration data while avoiding full Gaussian-process covariance inference and modifications to the deployed surrogate architecture or training objective.

3. Method

The method formulates surrogate error through a reference map and error functional, then motivates an anchor-averaged bound using regularity assumptions. GeoQ treats that structure as motivation and learns the correction empirically from calibration data.

  • The formulation assumes a deterministic reference map F with observed outputs yi = F(xi) and a surrogate approximation bF that may not preserve the reference output structure exactly.
  • Prediction error is defined by an error functional ρ, allowing scalar, componentwise, or pointwise error fields.
  • Observed dataset errors are used to estimate E(x) at previously unseen queries, including inputs near or outside available-data support.
  • The method assumes calibration and test data come from the same reference map or data source, while usefulness depends on represented query geometry and safe-region support.
  • Regularity arguments relate query error to anchor error plus a distance-scaled term, with analogous componentwise or pointwise interpretations for non-scalar errors.
  • GeoQ uses the regularity structure only as motivation and learns its nonnegative correction empirically rather than estimating an input-space Lipschitz term directly.

3.3. Anchor-based error decomposition

GeoQ decomposes query error into an anchor-averaged local baseline and a learned geometry-dependent correction. Nearest-neighbor features summarize representation-space displacement and normalized input-space support at configurable geometric scales.

  • The anchor set is selected by geometric nearness, and observed anchor errors are averaged to form the query’s local baseline.
  • A representation map ϕ may be identity, learned, or otherwise chosen to encode geometry relevant to surrogate error.
  • GeoQ estimates query error by adding a learned nonnegative correction to the anchor-averaged baseline, with calibration targets based on anchor-relative error increments.
  • Anchor-relative displacement and local support features are constructed from nearest neighbors in specified embeddings.
  • The parameters ka and k control neighborhood and support scales: smaller values emphasize localized structure, while larger values produce smoother summaries.
  • Additional gradient-based features are possible, but the method restricts attention to two geometry-based features because gradient reliability and high-dimensional sparsity may limit their utility.

3.5. Conditional quantile increment model

GeoQ learns a nonnegative upper conditional-quantile correction to an anchor-averaged error, using geometry-aware features and cross-fitted calibration data.

  • Conditional-quantile correction: GeoQ estimates an upper conditional quantile of the anchor-relative error increment rather than relying on generally unavailable deterministic worst-case regularity factors.The correction targets the error increment relative to nearby anchors.
  • Conditional-quantile correction: The increment model uses a fixed standardized feature map and a nonnegative model multiplied by feature magnitude, ensuring zero correction at zero geometric separation.This parameterization also makes the correction nonnegative.
  • Cross-fitted calibration: Cross-fitting trains auxiliary surrogates on complementary folds and evaluates held-out points to produce approximately out-of-sample errors and query–anchor configurations.The representation and normalization maps remain fixed and shared across folds.
  • Cross-fitted calibration: Cross-fitted calibration increments combine held-out query residuals with anchor-averaged errors computed from fold-training inputs.Anchor residuals are intentionally in-sample because deployment provides observed anchor errors while query errors are unknown.
  • Cross-fitted calibration: For extrapolative or shifted settings, structured folds can expose larger separations, while the fold count trades geometric diversity against fold-surrogate accuracy.Smaller Kf increases diversity but may reduce surrogate accuracy; larger Kf improves fidelity but reduces displacement-space coverage.
  • Conditional-quantile correction: A high-level pinball-loss level encourages the learned correction to approximate an upper conditional quantile, with function-valued errors handled pointwise.Underestimation is penalized by τ and overestimation by 1 − τ.

3.7. Safe-region score

GeoQ uses a density-based feature-space support score to identify whether query geometries lie in calibration-supported regions.

  • Support-score construction: The support score is computed from cross-fitted calibration features in a standardized representation space using nearest calibration-feature neighbors.The standardization map is fit on the calibration set and shared with the increment model.
  • Support-score interpretation: Small support scores indicate well-represented query geometries, whereas large scores indicate sparse or previously unseen feature configurations.The score is designed to assess whether the learned conditional quantile model is being applied where calibration data provide support.
  • Safe-region threshold: Leave-one-out scores exclude each calibration feature from its own neighbor set before selecting the safe-region threshold as an empirical high quantile.This prevents zero self-distance from artificially lowering calibration scores.
  • Validity boundary: Within the safe region, coverage still depends on conditional-quantile quality and on the representativeness of cross-fitted calibration features.The criterion distinguishes calibration support from insufficient representation of geometric configurations.

3.8. Algorithm summary and implementation choices

Algorithm 1 summarizes GeoQ calibration and inference, allowing either a fixed pretrained surrogate or joint surrogate training before calibration.

  • Workflow: GeoQ can calibrate an error estimator around a fixed pretrained surrogate or train the deployed surrogate first when none is supplied.The workflow then constructs the GeoQ calibration model around that deployed surrogate.
  • Design choices: The implementation accepts choices for the representation map, anchor and support-neighbor sizes, quantile level, safe-score neighbors, and fold count.These parameters define the geometry, quantile calibration, support assessment, and cross-fitting configuration.
  • Calibration: The algorithm defines inputs and observed anchor errors, partitions data into folds, and fits the correction model using standardized features and componentwise pinball loss.Cross-fitted calibration precedes correction-model training.
  • Inference: At inference, GeoQ returns the anchor-averaged error plus learned correction together with a safe indicator based on the standardized support score.The safe indicator marks whether the query geometry satisfies the safe-region threshold.

4. Numerical results

GeoQ estimates surrogate error across scalar, chaotic, weather, and instability benchmarks, including extrapolative and long-horizon settings. Its estimates generally track error growth and spatial structure, while fine-scale localization and configuration selection remain limitations.

  • Evaluation setup: GeoQ uses a fixed default configuration with one nearest anchor and a 20-neighbor local input-space support feature across test cases.The feature vector combines geometry and support information, although several design choices remain problem-dependent.
  • Forrester function: On the Forrester benchmark, GeoQ captures local error variations and follows the rapid error increase near training-support boundaries.The surrogate is trained on [0.25, 0.75] and tested over [0, 1], creating extrapolative regions outside the training interval.
  • Kuramoto–Sivashinsky equation: Over a 600-step KS rollout calibrated to 300 steps, GeoQ captures rapid early error growth and subsequent saturation beyond the calibration horizon.The estimate remains close to or above the true mean error, but is smoother than the true field and can under-cover localized structures from chaotic phase drift.
  • WeatherBench dataset: On WeatherBench day 10, GeoQ recovers dominant error locations and relative intensities, closely tracking z500 and T850 while producing broader, more conservative T2m estimates.The results support spatially resolved error estimation under temporally shifted forecasting conditions.
  • Richtmyer–Meshkov instability: For RMI, GeoQ identifies localized errors near interface deformation and vortex formation at t = 6 and tracks increased error concentration at t = 12.The estimated field remains strongly correlated with the true error distribution as the dynamics become more nonlinear.
  • Quantitative comparison: Global comparisons include near-support and out-of-distribution samples, while GeoQ directly predicts upper error estimates through conditional quantile modeling.Forrester’s GP achieves the best pinball loss and perfect coverage, whereas GeoQ achieves the highest correlation with competitive coverage; probabilistic baselines use approximate bounds 1.96σ.

5. Conclusion and Future work

GeoQ combines anchor-averaged calibration errors with a geometry-based conditional-quantile correction and support criterion for query-specific surrogate-error estimation. Across four scientific benchmarks, it achieved competitive or superior calibration, while leaving several computational and methodological questions for future work.

  • Conclusion: GeoQ estimates query-point surrogate error by adding a learned nonnegative conditional-quantile correction to anchor-averaged calibration error.The correction uses geometric displacement and local support information, while the feature-space kNN criterion identifies calibration-supported regions.
  • Conclusion: Across Forrester, Kuramoto–Sivashinsky, WeatherBench, and Richtmyer–Meshkov instability, GeoQ was competitive or superior to Gaussian processes, MC dropout, deep ensembles, and split conformal prediction.It was particularly effective for pinball loss; for WeatherBench and RMI it produced spatially coherent error fields and reliable coverage.
  • Conclusion: For KS autoregressive rollouts, GeoQ achieved the lowest pinball loss and highest coverage among the compared methods.The method captured dominant error-growth behavior, although precise localization of fine-scale error structures remained challenging.
  • Future work: Representation maps, calibration-fold construction, and support-score neighbor counts are problem-dependent design choices requiring adaptive or data-driven selection.The paper provides representative ablations rather than comprehensive sensitivity studies across all benchmarks.
  • Future work: Cross-fitting can dominate calibration cost because it requires training an auxiliary surrogate for each fold.Future work could construct approximately out-of-sample error targets without training Kf complete auxiliary surrogates.
  • Future work: The safe-region criterion is empirical, and stronger links between feature-space support and conditional coverage remain open.Future directions also include stability-aware recursive error propagation and using estimates for inverse problems, optimization, and adaptive data acquisition.

Appendix A. Model architectures and training configurations

Appendix A specifies the optimization setup and benchmark-specific surrogate architectures used with GeoQ. The increment models use corresponding architectures while taking geometric features as inputs.

  • Training configuration: All surrogate and uncertainty models use Adam with learning rate 10^-3, mini-batch stochastic optimization, and validation-loss early stopping unless otherwise stated.The experiments use a single NVIDIA GeForce RTX 2080 Ti GPU.
  • Surrogate architectures: The Forrester surrogate is a fully connected network with hidden widths (64, 64, 16, 32), using the 16-dimensional layer as representation map ϕ.ReLU activations are used in hidden layers, followed by a linear output layer.
  • Surrogate architectures: The KS surrogate is a fully connected network with hidden widths (128, 128, 16, 128, 128), using its intermediate 16-dimensional layer as ϕ.ReLU activations are used throughout the network.
  • Surrogate architectures: WeatherBench uses a convolutional encoder-decoder U-Net mapping three atmospheric-history steps to a 40-step forecast trajectory.The encoder has three downsampling levels, a 256-filter bottleneck, and convolutional blocks with GELU activations and dropout rate 0.1.
  • Surrogate architectures: The RMI surrogate is a two-dimensional Fourier Neural Operator mapping a 240 × 480 × 1 initial density field to 12 future density fields.The input is resized to 256 × 512, augmented with coordinate channels, and lifted to 64 channels.
  • Increment models: GeoQ increment models modify corresponding surrogate architectures to accept geometric features ψ=(∆zka, ∆dk).Latent displacement is the primary feature, while scalar support is incorporated by concatenation or conditioning layers.

Appendix B. Ablation studies on KS equation

KS ablations show that GeoQ performance depends on representation, feature construction, anchor size, and cross-fitting folds. The strongest configurations favor learned geometry, directional displacement, support information, and moderate cross-fitting.

  • Ablation design: Table B.4 compares representation maps, geometric features, anchor-neighborhood sizes, and cross-fitting-fold choices on the KS equation.The detailed sensitivity analysis is restricted to this representative nonlinear spatiotemporal benchmark.
  • Representation map: Using the surrogate encoder representation Fencoder improves all metrics compared with the identity map.The learned latent representation provides more informative geometry for anchor selection and error-increment modeling than raw input coordinates.
  • Feature construction: Signed representation displacement ∆zka together with support score ∆dk gives the best overall balance among pinball loss, correlation, and coverage.Replacing signed displacement with its magnitude degrades all metrics, while removing ∆dk also reduces performance.
  • Feature construction: Using only ||∆zka||2 and ∆dk yields slightly higher coverage but substantially lower correlation than the signed-displacement features.This indicates that scalar displacement magnitude loses directional information useful for calibration.
  • Anchor neighborhood: ka = 1 gives the strongest performance among the tested anchor-neighborhood sizes.Increasing ka worsens performance, suggesting that overly large neighborhoods oversmooth local geometry.
  • Cross-fitting folds: Kf = 5 provides the best tradeoff between calibration diversity and surrogate fidelity.Too few folds reduce stable training data, whereas too many weaken the pseudo-OOD calibration signal through less geometric separation.

Appendix C. Baseline uncertainty methods and implementation details

Appendix C describes Gaussian-process, MC-dropout, deep-ensemble, and split-conformal baselines for uncertainty estimation. Their implementations differ in stochastic or predictive-variance construction and in scalability across output types.

  • Baselines: The comparison includes Gaussian process regression, MC dropout, deep ensembles, and split conformal prediction.These methods are used as baselines for GeoQ.
  • Gaussian process: Gaussian-process uncertainty uses a constant-plus-RBF-plus-white-noise kernel and converts predictive standard deviation to bE(x)=1.96σ(x).The resulting estimate increases away from training data under the chosen kernel geometry but has limited scalability for high-dimensional fields.
  • MC dropout: MC dropout keeps dropout active during inference and converts variability across repeated stochastic predictions into bE(x)=1.96σ(x).For autoregressive settings, multiple stochastic rollout trajectories are generated.
  • Deep ensembles: Deep ensembles train independently initialized surrogates and convert ensemble standard deviation into bE(x)=1.96σ(x).For spatial outputs, standard deviation is computed coordinatewise to produce an uncertainty field matching the surrogate output shape.
  • Conformal prediction: Split conformal prediction calibrates pointwise absolute residuals into fixed, input-independent bounds.The bound is scalar for scalar outputs and coordinatewise for spatially distributed outputs.
  • Inference: Baseline inference times include all stochastic or ensemble trajectories, whereas GeoQ uses a deterministic surrogate rollout followed by deterministic error estimation.This distinguishes the computational procedure used for runtime comparison.

Appendix D. Surrogate model accuracy

Table D.5 compares surrogate-model relative L2 errors across uncertainty-estimation approaches. GeoQ uses surrogate predictions with accuracy broadly comparable to neural-network baselines, while GP has a low-dimensional advantage on Forrester and scalability limits elsewhere.

  • GeoQ, MC dropout, deep ensembles, and conformal prediction use essentially the same surrogate architectures, with dropout layers distinguishing MC dropout.
  • Across high-dimensional benchmarks, these neural-surrogate models have broadly comparable relative L2 errors.
  • Table D.5 reports surrogate-model accuracy using relative L2 error across different approaches.
  • GP performs much better on the low-dimensional Forrester problem because Gaussian process regression is well suited there.
  • For KS, GP surrogate error is comparable to neural surrogates, but GP is not reported for WeatherBench and RMI because of scalability limitations.
  • Overall, GeoQ is evaluated with surrogate prediction accuracy similar to neural-network baselines, making the uncertainty comparison more meaningful.
Loading 2608.21652v1…