Learn Only Where You Can't Compute: Structured Corrections to Gaussian Process Posteriors

1 INTRODUCTION

• Scientific inference combines measurements, boundary conditions, physical laws, numerical models, and historical data, while often leaving the quantity of interest only partially determined.The resulting conditional distribution must combine available information and remain computationally tractable.Source: §1 · p. 1
• Gaussian processes analytically condition on linear information and provide predictive means and structured uncertainty, but their assumptions may miss nonlinear effects, multimodality, or model mismatch.Learning the entire conditional law with a generative model can require it to represent structure already available from a Gaussian posterior.Source: §1 · p. 1

Gaussian approximation can already be computed?

• The framework starts from an analytically conditioned Gaussian, shifts its mean toward the target, then calibrates covariance and learns residual spread and non-Gaussian shape in a selected low-dimensional subspace.The complement retains its Gaussian law.Source: Gaussian approximation can already be… · p. 2
• The paper combines Gaussian and subspace KL decompositions, data-driven selection by validation log-score improvement, and evaluations across regression, smoke transport, and trajectory reconstruction.The experiments compare learning curves, run times, and distributional scores, including when full-field models remain preferable.Source: Gaussian approximation can already be… · p. 2

2 METHOD

• The target conditional law is approximated from a tractable Gaussian reference by mean correction, pooled covariance calibration, and a flexible law on a selected low-dimensional block.The calibrated Gaussian supplies the law on the complement.Source: §2 · p. 2
• The KL discrepancy separates location mismatch, covariance mismatch, and non-Gaussian structure; the identity assumes finite second moments and divergences and nonsingular covariances.Numerically rank-deficient scaffolds are handled in resolved coordinates with a positive covariance floor.Source: §2 · pp. 2–3
• A full-field mean correction removes location mismatch before richer distributional corrections are chosen, using a precision-weighted regression loss whose unrestricted population minimizer also matches unweighted squared error.The fitted shift defines the corrected mean and residuals.Source: §2 · p. 3
• After mean correction, residuals are whitened by scaffold covariance and pooled across contexts; estimated variance above or below one signals scaffold underestimation or overestimation in that direction.Directions are calibrated according to their potential log-score gain.Source: §2 · p. 3
• The probabilistic correction learns a conditional law for projected coordinates while leaving the orthogonal complement Gaussian, reducing learned stochastic dimension when k is much smaller than d.Even an exact block law leaves complement mismatch and block–complement dependence unresolved.Source: §2 · pp. 3–4
• Subspace selection estimates held-out log-score gains with a lightweight product of one-dimensional Gaussian mixtures, avoiding diffusion fits for every candidate block.Matched isotropic noise with standard deviation τ softens discrepancies smaller than τ before gain evaluation.Source: §2 · p. 4
• The search proposes directions from residual PCA, independent components, and excess-variance spans, then refines promising frames and selects directions above a practical validation-gain threshold.The threshold is not a statistical guarantee accounting for the adaptive search.Source: §2 · p. 4
• The selected block is fitted with a mixture or conditional diffusion model, which can capture joint dependence among selected coordinates; setting k to zero gives the calibrated Gaussian and k to d gives a full residual model.The experimental full-residual baselines also differ in calibration and architecture.Source: §2 · p. 4

3 RELATED WORK

• Gaussian processes and probabilistic numerical methods condition on observations, linear physical constraints, differential equations, and numerical computations to construct structured Gaussian references.Prior work also includes sparse Gauss Markov formulations and decompositions for nonlinear observations of finitely many linear functionals.Source: §3 · p. 4
• Low-dimensional correction methods motivate modeling high-dimensional targets through a smaller discrepant subspace, while this framework adds mean correction, covariance calibration, and data-driven block selection for scientific Gaussian posteriors.A lightweight conditional mixture estimates validation log-score gain before the selected block is fitted with a mixture or diffusion model.Source: §3 · p. 4
• Bayesian calibration and learned residual models address simulator–observation discrepancy; unlike full-field residual approaches, this work begins with a probabilistic scaffold and retains a calibrated Gaussian complement.The experiments examine what this restriction preserves and where full-field models remain preferable.Source: §3 · pp. 4–5

Methods.

• Comparisons use three seeds, 500 test contexts, and 100 draws per context; full-field and selected methods also differ in architecture and representation.CRPS and Cramér distance assess coordinate-wise predictions, energy score assesses fields, and PCD complements predictive scores without fully characterizing the distribution.Source: Methods · p. 5

4.1 Synthetic

• From ntrain=500, Gain-Mix and Gain-Diff beat Exact-GP in marginal distance; Gain-Diff separates bump modes by n=1000, before Full-Diff at n=2000.The selector retains 16 coordinates on every seed, matching the 16 hidden bumps.Source: §4.1 · p. 5, Figure 2 · p. 6

4 EXPERIMENTS

• Selected corrections improve on Gaussian scaffolds across Synthetic, Smoke, and CR3BP, recovering modes, improving smoke predictions, and lowering trajectory error.Table 1 reports the main results at n=2000, 8192, and 7500 for the three tasks, respectively.Source: §4.1 · p. 5, Table 1 · p. 6, §4.2 · p. 6

4.2 Smoke

• Gain-Diff closes 90% of Smoke’s scaffold-to-exact CRPS gap using only 38–39 of 1024 field coordinates.Gain-Mix closes 85% of the gap; mean correction closes 33%, rising to 42% after covariance calibration.Source: §4.2 · p. 6
• Gain-Diff has near-Exact-GP CRPS (0.0457 versus 0.0459) but lower PCD (154 versus 1256); direct Diff leads learned methods on CRPS, Cramér distance, and PCD.Smoke samples show more localized plumes with Gain-Diff, though one displayed draw remains diffuse.Source: §4.2 · p. 7, Figure 4 · p. 7
• In Smoke, Full-Diff has slightly lower CRPS than Gain-Diff, while Gain-Diff has lower PCD at n=8192; selected corrections trade predictive accuracy for reduced sampling cost.Figure 3 reports costs at n=8192, d=1024; training includes selection and excludes evaluation, and sampling produces 100 draws for one context.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 CR3BP

• Gain-Diff reaches 0.0815 CRPS, about half the solver’s 0.1588; direct conditional diffusion has higher CRPS than the scaffold.Gain-Diff and Gain-Mix also achieve lower field CRPS, energy scores, and Jacobi errors than both full-field alternatives.Source: Figure 5 · p. 7, §4.3 · p. 8

• The targeted correction recovers hidden modes in Synthetic, improves predictive accuracy and sample structure in Smoke, and reduces numerical trajectory error in CR3BP.Source: §5 · p. 8
• Approximation remains in the Gaussian complement and cross-subspace dependence; pooled calibration, selector transfer, context variation, and data-dependent rank constrain the correction.CR3BP selection reaches its imposed rank cap, so it does not establish an intrinsic correction dimension; its point-mass target also falls outside finite-KL assumptions.Source: §5 · p. 8
• In Smoke, restricted correction improves predictive accuracy over the Gaussian reference, while direct diffusion achieves higher accuracy at greater sampling cost.Gain-Diff adds fitting overhead but reduces generative-model dimensionality and simplifies training.Source: §5 · p. 8
• The framework uses computation to build a reference, residual data to locate where richer laws add value, and generative modeling only in those directions.The Gaussian law remains on the complement.Source: §5 · p. 8

A.1 Derivations

• A product contrast can measure only a lower bound on the gain available to a joint block law; the difference is dependence within the block.Source: §A.1 · p. 13

A.2 Covariance mismatch correction

• Whitened residual variances are corrected only when they exceed both sampling-noise thresholds; other directions retain unit variance or share an average variance.The thresholding accounts for Marchenko–Pastur spread in finite samples.Source: §A.2 · p. 13

A.3 Subspace selection

• Subspace search starts from residual PCA, FastICA, and excess-variance proposals, then refines the best pool by gradient ascent on orthonormal frames.Source: §A.3 · pp. 13–14
• Held-out gains determine retained coordinates, but reusing the validation split for early stopping makes reported gains optimistic and the threshold does not control an error rate.Source: §A.3 · p. 14

A.4 Block laws and sampling

• Gain-Mix fits a product Gaussian mixture and Gain-Diff a conditional DDPM on the selected block; directions outside resolved calibration remain Gaussian.On Synthetic, both block laws are unconditional.Source: §A.4 · p. 14

B.1 Methods

• The method roster distinguishes center, Gaussian component, and learned law; KO uses a shared-context GP discrepancy, while Diff generates the whole field without the scaffold.Source: Table 2 · p. 15, §B.1 · p. 15

B.2 Training protocol

• Mean, calibration, selection, and law stages are fitted sequentially on the same training split and frozen between stages; final comparisons use three seeds.Source: §B.2 · p. 15

B.3 Evaluation and metrics

• Evaluation uses marginal and joint scores, while PCD compares score distributions; matching those distributions does not establish matching posterior distributions.PCD values are comparable only within a problem because their scale depends on the problem and smoothing choice.Source: §B.3 · pp. 16–17

C DATASET: SYNTHETIC

• Synthetic readings leave the bump-sign modes unresolved, so the missing non-Gaussian structure lies in the same directions across contexts.Source: C DATASET: SYNTHETIC · p. 17

C.1 Data, exact posterior and scaffold

• The exact Synthetic posterior adds fresh uniform bump signs to a Gaussian draw, whereas the scaffold uses only the coarse GP and misses fine-scale structure and bumps.Source: §C.1 · pp. 17–18

C.2 Methods and training

• Synthetic uses a pooled contrast at τ=0 and unconditional block laws; the physical-consistency score evaluates the convolved exact mixture at τ≈0.08.Source: §C.2 · p. 18, §C.3 · p. 18

C.4 Additional results

• As training size grows, gain methods and full-residual diffusion approach the exact reference; Figure 7 adds marginal, joint, and physical-consistency metrics across those sizes.Figure 2 uses three seeds per plotted method at each of eight training sizes from 100 to 15,000.Source: §C.4 · p. 18, Figure 7 · p. 19
• From ntrain = 500, the selector keeps k = 16 coordinates on every seed; at ntrain = 100, it falls back to the Gaussian.The 16 coordinates correspond to the number of bumps.Source: §C.4 · p. 18
• The sample displays show three method draws at n = 2000, with Figure 9 zooming into four sensor gaps and adding exact draws.Figure 8 includes the generating function, readings, and exact posterior pointwise 90% intervals; readings do not determine bump signs.Source: §C.4 · p. 19, Figure 8 · p. 20, Figure 9 · p. 20
• At n = 2000, Table 5 reports median fit-plus-evaluation time, so learning fewer stochastic coordinates does not by itself establish lower full computational cost.The table reports three-seed medians on an RTX 2080 Ti, includes selection, and excludes data preparation.Source: §C.4 · p. 19, Table 5 · p. 19

D DATASET: SMOKE

• Smoke uses nine readings to constrain wind direction, while the exact posterior is a direction mixture and the scaffold is a Gaussian with the prior’s first two moments.The readings may constrain direction sharply or leave several sectors plausible.Source: D DATASET: SMOKE · p. 21

D.1 Data, exact posterior and scaffold

• The Smoke field combines a Gaussian background with a plume from a point source; the setup uses a 32 × 32 grid and nine noisy sensor readings.The plume comes from a steady wind with speed 1 and diffusivity 0.003; the exact posterior uses a grid of 360 wind angles.Source: §D.1 · p. 21
• The scaffold conditions the prior Gaussian on readings, but its prior moments cannot determine context-specific conditional means, covariances, or wind sectors.Consequently, scaffold draws can superpose several rays.Source: §D.1 · p. 21

D.2 Methods and training

• Smoke corrections use an observation MLP for the mean, pooled covariance calibration, and a conditional contrast density to select a block; KO collapses to the scaffold.The mean network is 2 × 128, and calibration covers 309 range directions and 197 complement spikes.Source: §D.2 · p. 21

D.3 Evaluation

• The physical-consistency score evaluates the exact mixture density convolved with Gaussian noise, with Q integrated out; results are also viewed at the source, upwind, and downwind.The noise scale is τ = 0.2 · 0.163 ≈ 0.033; the source view is a disk of radius 0.2, upwind is x1 < 0.15, and downwind is the remainder.Source: §D.3 · p. 21
• Figure 10 pairs the wind-direction prior with six plume shapes and three field draws, illustrating rays whose directions vary across draws.Stars mark the source and circles mark sensors.Source: Figure 10 · p. 22
• Table 6 covers field and source-disk scores across 500 test contexts and 100 draws, reporting mean ± standard deviation over three seeds at ntrain = 8192.Its metrics include Cramér distance, PCD, RMSE, and selected block size k.Source: Table 6 · p. 22

D.4 Additional results

• Figure 11 compares retained Smoke methods with two exact references; Exact-GP matches the exact posterior’s conditional mean and covariance.The draws use the same readings, training size, seed, and first three draws as Figure 4, but rows are not paired realizations.Source: §D.4 · p. 22, Figure 11 · p. 23
• The Smoke learning curve shows no crossover favoring gain methods; direct diffusion still leads at n = 8192, although the gap between Gain-Diff and direct diffusion narrows as data increase.The authors present the restricted model as a practical alternative, not as evidence of general scaffold-driven sample efficiency.Source: §D.4 · p. 22, Figure 12 · p. 23
• At n = 8192, gain methods have lower combined fit-plus-evaluation times than full-field U-Nets, but separate timings show this is not a fitting advantage.Table 7 reports three-seed median wall time on an L40S, including selection but excluding data preparation.Source: §D.4 · p. 22, Table 7 · p. 23
• Figure 12 plots the share of the scaffold-to-exact CRPS gap closed, and scratch diffusion leads throughout the measured range.The curve begins at n = 1024; scratch diffusion has two seeds at n = 1024 and 4096, while filled points have three seeds.Source: Figure 12 · p. 23

E DATASET: CR3BP

• For CR3BP, the reference law is a point mass at the deterministic trajectory, so the paper’s KL identities for target densities with finite divergences do not apply.Predictions are scored against a high-accuracy reference trajectory.Source: E DATASET: CR3BP · p. 24

E.1 Data, reference trajectory and scaffold

• The planar CR3BP state is z = (x, y, vx, vy), with the experiment using the Arenstorf orbit of period T = 17.0652.The trajectory reference is integrated over one period and stored at 501 equally spaced times, yielding d = 2004.Source: §E.1 · p. 24
• Initial states are phase-shifted Arenstorf states with Gaussian perturbations; a shared pool is split per seed, and runs use 7,500 training trajectories.Trajectories failing coarse-solver thresholds are rejected.Source: §E.1 · p. 24
• The scaffold is a probabilistic ODE filter whose floored covariance is based on centered pathwise draws; the context is the initial state z0.It uses a twice-integrated Wiener prior, linearization, calibration, adaptive steps, and smoothing.Source: §E.1 · p. 24

E.2 Methods and training

• The mean correction is an MLP trained on trajectory residuals, while covariance calibration uses the full pooled basis and subspace selection searches at most 64 directions.The block DDPM and full-field diffusion models are MLPs conditioned on z0.Source: §E.2 · p. 24

E.3 Evaluation

• Evaluation scores CRPS, energy score, and coverage against the reference trajectory, and uses Jacobi-constant drift to assess physical consistency.The Jacobi-error summary is the median over contexts and draws of the absolute drift difference from the reference.Source: §E.3 · p. 24

E.4 Additional results

• Table 8 lists retained methods in the order used in Table 1.Source: §E.4 · p. 24
• Table 8 reports CR3BP results at d = 2004 and n = 7500 as mean ± standard deviation over seeds 10–12, using 500 contexts × 100 draws.Jacobi error is defined in Appendix E.3, and k denotes block size.Source: Table 8 · p. 25
• Table 9 reports median fit-plus-evaluation time over three seeds on an RTX 2080 Ti, including selection but excluding data preparation.Separate sampling timings are unavailable.Source: Table 9 · p. 25, §E.4 · p. 25