PaperlayerView exact revision on arXiv

Source-linked AI summary

An end-to-end framework for uncertainty-aware validation of generative models

Giada Badaracco, Sean Benevedes, Christina Reissel, Philip Harris, Gaia Grosso

arXiv:2610.37398v1stat.MLcs.LGhep-exhep-ph

TL;DR

Finite training data leave fitted generative models uncertain, while standard goodness-of-fit tests treat them as exact. SIGMA models fit uncertainty and profiles it in a composite test, which can remain calibrated on test sets up to an order of magnitude larger than the point-null equivalent, subject to faithful uncertainty coverage.

  • Problem

    Standard goodness-of-fit tests treat a fitted generative model as exact, although finite training data leave statistical uncertainty that should be accounted for when validating it.

    Source: §I · p. 2, §II · p. 3

  • Method

    SIGMA uses a Gaussian mixture model with tractable parameter uncertainty and profiles that uncertainty as nuisance parameters in a composite goodness-of-fit test.

    Source: §VI · p. 14

  • Results

    The composite-null test is calibrated on datasets an order of magnitude larger than the point-null equivalent, provided propagated uncertainties cover the fit error.

    Source: §I · p. 2, §VI · p. 14

  • Takeaways & Limitations

    The framework provides a statistically grounded basis for using generative models in physical sciences when fit uncertainty is propagated into validation.

    Source: §VI · p. 14

    SIGMA is suited to low-dimensional spaces rather than direct high-dimensional use, and it does not resolve model bias when the model family cannot represent the data.

    Source: §VI · p. 14

Abstract

from arXiv · show

Density estimation with generative models is a common task in the physical sciences, for example in background estimation in particle physics. Existing machine learning techniques, however, lack methods to quantify uncertainties that can be propagated into a statistical test. That limits the validation of generative models to standard goodness-of-fit tests, in which the generative model is treated as exact (point-null hypothesis). But a model fitted to a finite training set is never exact, so if its uncertainties are neglected, any sufficiently powerful goodness-of-fit test will eventually reject it once the test set is large enough to resolve this estimation error. We propose SIGMA, an end-to-end framework combining density estimation with principled uncertainty quantification. We use Gaussian mixture models, for which uncertainties are tractable, and further construct a new goodness-of-fit test in which these uncertainties are profiled as nuisance parameters to account for the statistical error of the fit, promoting the conventional point-null test to a composite one. A rejection then signals genuine mismodeling rather than a test set large enough to resolve the error of the fit, provided the propagated uncertainties faithfully cover that error. We demonstrate the framework on a controlled two-dimensional toy model and on a collider physics benchmark modeling the QCD background process. With uncertainties included, the same model can be trusted on test sets up to an order of magnitude larger than the standard test allows. Our work thus provides a statistically grounded basis for a robust use of generative models in the physical sciences.

I. INTRODUCTION

Generative models can make expensive scientific simulations more efficient, but finite-training error must be accounted for when their fitted density serves as a reference in statistical inference. SIGMA pairs tractable fit uncertainties with a goodness-of-fit test that profiles them, and retains its nominal false-positive rate beyond the training-sample size when those uncertainties cover the fit error.

I. INTRODUCTION

  • When a fitted density is the reference for inference, its estimation error can be indistinguishable from a signal and must enter the statistical test.By contrast, errors may be tolerable when they affect only efficiency or statistical power, or when downstream checks or qualitative validation suffice.Source: §I · p. 2, Figure 1 · p. 2
  • Uncertainty estimation for modern density estimators is technically challenging, particularly with high-dimensional inputs and many non-identifiable parameters.Source: §I · p. 2
  • SIGMA combines a Gaussian-mixture density model with tractable fit uncertainties and an uncertainty-aware goodness-of-fit test.The test profiles nuisance parameters representing fit uncertainty, promoting the conventional point-null test to a composite one.Source: §I · p. 2
  • The composite test retains its nominal false-positive rate when test samples exceed the training sample, provided the uncertainties faithfully cover the fit error.Source: §I · p. 2
  • The paper evaluates its approach on a controlled two-dimensional toy study and a QCD-background model in a learned embedding of particle jets.Source: §I · p. 2

II. POWERFUL TESTS REJECT EVERY GENERATIVE MODEL

A fitted generative model differs from the true density because of finite training data, even when its model family contains the truth. Standard goodness-of-fit tests treat the fit as exact and can therefore reject it as test samples grow; accounting for fit uncertainty instead tests compatibility within the spread of possible fits.

II. POWERFUL TESTS REJECT EVERY GENERATIVE MODEL

  • The study separates model bias, which persists with more training data, from statistical uncertainty due to finite samples and optimization variability.The work addresses statistical uncertainty; an incorrect model's bias is not solved by propagating fit uncertainties.Source: §II · p. 3
  • A consistent goodness-of-fit test eventually detects finite-sample fit error as the test set grows, even when the model family contains the truth.In the Gaussian example, the fitted model is rejected with high significance once Ntest grows past Ntrain.Source: §II · p. 3
  • This degradation is a generic, quantifiable consequence of information limits, and no architectural or training improvement makes a density estimated from Ntrain samples exact.Its rate depends on test sensitivity and model design, but the degradation itself cannot be avoided.Source: §II · p. 3
  • The point-null test conflates genuine mismodeling with unavoidable fit fluctuations, whereas the composite null asks whether data are compatible within the statistical uncertainty of the fit.With uncertainties included, the median p-value remains close to its nominal 0.5 over the probed test sizes.Source: §II · p. 3

III. RELATED WORK

Prior generative-model validation and uncertainty methods leave open how to test a fitted density while accounting for its estimation uncertainty. This work combines a parametrized uncertainty description with a composite goodness-of-fit test and bootstrap calibration.

III. RELATED WORK

  • The Gaussian example's composite-null test accounts for estimation error and remains satisfied to the largest probed test size, while the point-null is eventually rejected.Source: Figure 2 · p. 4
  • Ensemble uncertainty spans only trained-model weights and covers the true density only under the assumption that it lies within the ensemble members' span.SIGMA instead spans the generative model's full parameter space, at the cost of requiring a tractable likelihood.Source: §III · p. 4
  • Earlier validation tests generally treat the fitted density as exact, so they eventually reject even a model performing as well as its training statistics allow when Ntest is sufficiently large.Source: §III · p. 4
  • The paper's mixture-based contribution is to use tractable scores and Fisher information to extend the point-null test to a composite one.The mixture machinery itself is established; the contribution is bringing fit uncertainty into the test.Source: §III · p. 5
  • The bootstrap prescription follows the classical parametric-bootstrap procedure, paired with an explicit nuisance parametrization for fit uncertainty.Source: §III · p. 5

IV. METHOD

The method combines a generative model, a standardized description of its estimation uncertainty, and a goodness-of-fit test that allows the model to vary within that uncertainty. Test calibration is treated as a further component.

IV. METHOD

  • The method has three parts: an unconstrained generative-model parametrization, a standardized estimation-uncertainty description, and a goodness-of-fit test allowing movement within that uncertainty.Source: §IV · p. 5
  • The goodness-of-fit test is calibrated using the procedure discussed separately in Sec. IV D.Source: §IV · p. 5
  • The uncertainty-aware test lets the model move within its estimated uncertainty when assessing goodness of fit.Source: §IV · p. 5

A. Generative model

The generative model uses a Gaussian mixture whose unconstrained parametrization keeps every parameter setting a valid density. Its analytic scores make fit uncertainty tractable, particularly for low-dimensional densities.

A. Generative model

  • GMMs satisfy the requirements that uncertainty be propagated across all fluctuating parameters and every parameter setting define a valid density.The parameter count grows as P ∼ KD^2/2, making GMMs heavily overparameterized in high dimensions but expressive in low dimensions.Source: A. Generative model · p. 5
  • Weights use softmax logits and covariances use Cholesky factors with log-parametrized diagonals, giving unconstrained real-valued parameters.Fixing one logit removes the common-shift invariance of the softmax.Source: A. Generative model · p. 5
  • The GMM is fitted by maximum likelihood using EM, which alternates component assignments and closed-form parameter updates until reaching a stationary point.Because the likelihood is highly multimodal, the resulting fit is generally a local maximum.Source: A. Generative model · pp. 5–6
  • After fitting, the score function is fixed, and the uncertainty-aware test is built in the tangent space it spans around the best fit.Source: A. Generative model · p. 6, Figure 3 · p. 6

B. Estimation uncertainty

The fit’s statistical uncertainty is estimated from the Fisher information and expressed in standardized coordinates. Directions that are poorly constrained by training data are discarded because the local expansion may fail along them.

B. Estimation uncertainty

  • The composite null asks whether test data are consistent with a density whose parameters lie near the fitted values within their training-sample variation.A Gaussian constraint in standardized coordinates encodes these fluctuations in the test statistic.Source: B. Estimation uncertainty · p. 6
  • Expectations under the fitted model are estimated with a large reference set used for Fisher information, density normalization, and choosing alternative components.Source: B. Estimation uncertainty · p. 6
  • The fitted parameter covariance is estimated from the Fisher information, using its pseudoinverse with a small ridge, under the correctly specified model’s asymptotic covariance.The covariance estimate uses the reference set drawn from the fitted model.Source: B. Estimation uncertainty · p. 6
  • Eigen-directions with eigenvalues above Λmax are discarded because training data barely constrain them and the local expansion can break down along them.Source: B. Estimation uncertainty · p. 7
  • The retained directions define standardized coordinates that measure each parameter displacement in units of its uncertainty.Asymptotically, these coordinates are uncorrelated with unit variance, so a unit-norm displacement is a one-sigma shift.Source: B. Estimation uncertainty · p. 7

C. Uncertainty-aware goodness-of-fit test

The test profiles fit uncertainty as nuisance parameters in a composite-null likelihood-ratio test, then compares that null family with alternatives allowing localized density distortions. Its statistic isolates incompatibility that fit uncertainty cannot explain.

C. Uncertainty-aware goodness-of-fit test

  • The null family tilts the fitted density along uncertainty-normalized score directions, using an exponential tilt to keep the test convex.To first order, the tilt parameters are standardized parameter displacements.Source: C. Uncertainty-aware goodness-of-fit… · p. 7
  • The alternative adds exponential tilts along a fixed dictionary of localized Gaussian kernels, with width σ setting the scale of detectable departures.The dictionary should tile the model support; the effective degrees of freedom depend jointly on kernel width and ridge penalty, and results are stable under moderate changes in J.Source: C. Uncertainty-aware goodness-of-fit… · p. 7
  • Profiling the fit-uncertainty parameters uses a Gaussian constraint, while a ridge penalty and bounded kernel distortions constrain the alternative.The resulting objective is strictly concave, giving each maximization a unique optimum.Source: C. Uncertainty-aware goodness-of-fit… · pp. 7–8
  • The denominator lets the model move within its uncertainty, while the numerator also permits localized distortions, so the statistic measures incompatibility unexplained by fit uncertainty.The uncertainty-blind point-null baseline fixes v = 0.Source: C. Uncertainty-aware goodness-of-fit… · pp. 7–8
  • Experimental and theoretical systematic uncertainties can be profiled jointly with fit uncertainty when their variations admit a density parametrization, but this paper focuses on fit uncertainty.Source: C. Uncertainty-aware goodness-of-fit… · p. 8

D. Calibration

Calibration uses pseudo-experiments under the composite null, with either Gaussian parameter fluctuations or bootstrap refits. The prescriptions differ in assumptions and optimization variability captured, and the test’s calibration is assessed using pseudo-experiments rather than relying on its approximate asymptotic limit.

D. Calibration

  • Bootstrap calibration assumes fitting-procedure spread is similar for data from the truth and from the fitted model, and can fail when kernel width is too narrow.Source: D. Calibration · p. 8
  • The parametric bootstrap refits the full GMM on simulated training samples, rebuilding the reference set, Fisher information, standardization, and kernel dictionary for each replicate.Source: D. Calibration · p. 8
  • The Gaussian prescription moves the data-generating model while holding the fit fixed; bootstrap moves the fit while holding the data-generating model fixed.Bootstrap also captures optimization variability and yields uncertainty bands comparable to or larger than Gaussian bands, but neither accounts for model bias.Source: D. Calibration · pp. 8–9
  • The Gaussian prescription remains calibrated to larger test samples, while its assumptions hold; it also avoids refits and explicitly parametrizes uncertainty for propagation.Compared with bootstrap, it depends more strongly on the truncation choice Λmax.Source: D. Calibration · p. 9
  • The approximate χ2_J prediction holds when asymptotics are reliable and Gaussian and bootstrap prescriptions agree, but breaks down outside that regime.Calibration results are therefore evaluated using pseudo-experiments.Source: D. Calibration · p. 9
  • The test is independent of the density-estimation and uncertainty-estimation methods, provided uncertainty is reliably estimated and propagated into the composite test.Source: D. Calibration · p. 9

V. NUMERICAL EXPERIMENTS

The experiments apply the method to a controlled 2D toy model and a 4D particle-physics problem. The toy closure test compares three goodness-of-fit tests across test-set sizes and mixture-component counts.

  • The experiments cover a 2D toy model with known analytic ground truth and a four-dimensional particle-physics problem.Source: §V · p. 9
  • The toy closure plot shows median null p-values versus Ntest for K = 16, 32, 64, 128 at fixed Ntrain = 105.Source: Figure 5 · p. 10
  • A median p-value of 0.5 marks calibration; the plot also marks Ntest = Ntrain and shows 68% confidence intervals on the median.Source: Figure 5 · p. 10

A. Toy model

In the 2D toy study, bootstrap calibration preserves closure to larger test samples than the Gaussian prescription. The study also illustrates how point-null testing fails when it treats the fitted density as exact.

A. Toy model

  • The factorized toy pairs a separated bimodal feature with a strongly skewed feature, probing mode resolution and long-tail reproduction.Source: A. Toy model · p. 10
  • With Ntrain = 105 and four K choices, fresh test samples are drawn for each replica; bootstrap uncertainty bands are slightly wider where Gaussian bands underestimate fit spread.Source: A. Toy model · p. 10
  • Gaussian uncertainty restores calibration only at intermediate test sizes: median p-values remain compatible with 0.5 through about Ntest = Ntrain/5, but fall below 0.2 at 2Ntrain for all K.Source: A. Toy model · p. 10
  • Bootstrap calibration remains closed through Ntest = 2Ntrain for every K, capturing optimization variability and non-Gaussian fit scatter missed by the Gaussian description.Source: A. Toy model · pp. 10–11
  • At Ntest = Ntrain, the K = 64 point-null statistic lies almost entirely above its null, whereas the observed composite statistic is reproduced by the bootstrap null.Source: Figure 6 · p. 11, A. Toy model · p. 11

B. Embedded particle jets

The particle-jet study models the QCD background in a learned four-dimensional embedding. For K ≥ 256, bootstrap composite testing extends closure to Ntest = Ntrain/5, while calibration degrades at larger test sizes.

B. Embedded particle jets

VI. CONCLUSION

The conclusion and appendices discuss how SIGMA’s uncertainty-aware goodness-of-fit test handles finite-sample fit uncertainty, including its conditions of success and limitations. The toy studies show that whether propagated ensemble-weight uncertainty prevents rejection depends on whether it covers the fit error.

VI. CONCLUSION

  • SIGMA combines Gaussian-mixture parameter uncertainties with a composite-null goodness-of-fit test and bootstrap calibration, calibrating tests on datasets an order of magnitude larger than the point-null equivalent.Linearizing around the best fit yields parameter covariance from analytic score functions and Fisher information, making the test a convex problem.Source: §VI · p. 14

Appendix A: Additional discussion on the limits of generative models

  • The fitted density’s finite-training-sample error can eventually trigger rejection by a powerful point-null test, even when the model is correctly specified; beyond Ntest ≳ Ntrain/P, fit uncertainties must be incorporated.The appendix frames this as a sensitivity bound; actual rejection depends on a test’s power and calibration. Under model misspecification, an irreducible bias remains that uncertainty cannot absorb.Source: Appendix A · p. 16

Appendix B: Normalizing flows and the limits of ensemble uncertainties

  • For M = 128, the flow ensemble restores closure up to and beyond Ntest ≃ Ntrain in the two-dimensional toy, but fails in the four-dimensional embedding, where it is rejected well before Ntest = Ntrain.In the embedding, propagated weight uncertainty is negligible compared with flow misspecification; the composite test collapses onto the point-null.Source: Appendix B · p. 16

1. Density estimation with normalizing flows

  • Flow ensembles are formed by drawing M members from a pool of 512 independently trained flows, with ten repeated ensemble choices used to reduce dependence on the particular draw.Each flow is trained on its own bootstrap replica; the test profiles the same per-choice weight uncertainty shown in the reported uncertainty band.Source: §1 · p. 17

2. Uncertainty quantification via wifi ensembling

  • Weight-based ensembling captures reweightings of fixed flow members but not changes within individual flows, limiting which uncertainties it can represent.The appendix identifies this restriction as the reason the composite test approaches uniform null p-values only in the two-dimensional toy, not in the four-dimensional embedding.Source: §2 · p. 17

3. Uncertainty-aware goodness-of-fit test

  • The flow test profiles mixture weights under a Gaussian uncertainty constraint and calibrates p-values empirically with pseudo-experiments that sample weights and datasets.A full bootstrap that refits the flow ensemble for every pseudo-experiment could improve calibration but is computationally prohibitive; empirical calibration is used rather than Wilks’ theorem.Source: §3 · p. 18

4. Results

  • In the two-dimensional toy, increasing ensemble size tightens fitted marginals and produces uncertainty bands that widen in low-statistics tails; in the four-dimensional embedding, structured discrepancies persist while bands remain negligible.The four-dimensional uncertainty therefore does not cover the discrepancy between the ensemble and held-out data.Source: §4 · p. 18
  • In 2D, the M=128 composite test has median p≈0.43 at Ntest=5×10^4 and remains near nominal through Ntest=2×10^5, unlike the point-null test.At Ntest=5×10^4, the point-null median p-value is ≈0.16; the M=128 composite test does not reject even when Ntest exceeds Ntrain. For M=32, the composite and point-null p-values are ≈0.09 and 0.07 at Ntest=5×10^4, and the test reaches the floor by Ntest=Ntrain=10^5.Source: §4 · pp. 18–19
  • In 4D, propagating weight uncertainty does not separate composite from point-null p-values, and even M=128 is rejected by Ntest≈2×10^4, well below Ntrain=10^5.The ensembles degrade by Ntest≈10^4, while the single flow degrades by ≈2×10^3.Source: §4 · p. 19

c. Discussion

  • Because pseudo-experiments calibrate each null, the large-Ntest p-value loss indicates insufficient uncertainty coverage, not a test-statistic artifact.In 4D, the covariance spans only re-weightings of frozen flows, leaving uncertainty negligible relative to the data–ensemble discrepancy; the profiled weights therefore barely move and the composite null effectively reduces to the point-null.Source: §4 · p. 19

Appendix C: Implementation details

The appendix specifies numerical choices for fitting SIGMA, normalizing the tilted alternative, and calibrating the test. It also reports how regularization choices affect the two calibration approaches.

Appendix C: Implementation details

  • Scores are computed analytically or by central finite differences, which require 2P log-density evaluations per sample.Source: Appendix C · p. 19
  • Normalization and Fisher estimation use 10Ntest and 10^6 = 10Ntrain reference samples, respectively; features are standardized over the full 2 × 10^6 pool.The pool is drawn from the best-fit SIGMA model.Source: Appendix C · p. 19
  • Kernel centers are obtained by k-means clustering, with uniform isotropic widths σ = 0.4 in 2D and σ = 0.53 in 4D.The setup uses J = 500 centers.Source: Appendix C · p. 19
  • Regularization bounds each kernel’s log-density distortion by A = 10, and both regularizations are applied identically to data and pseudo-experiments.This corresponds to local distortions within a factor e±10.Source: Appendix C · p. 19
  • With λα = 1 and Λmax = 0.008, the Gaussian-calibrated null is strongly dependent on Λmax, while the bootstrap-calibrated null is more robust.For the bootstrap-calibrated null, Λmax affects the test statistic definition but not null sampling.Source: Appendix C · p. 19

Appendix D: Null distribution of the test statistic

Appendix D derives the composite test statistic’s null behavior and compares its calibration across test-set sizes. Appendix E shows that kernel widths finer than the data’s natural scale can cause spurious calibration failure.

Appendix D: Null distribution of the test statistic

  • As M grows, the fitted two-dimensional marginals tighten onto the target, while the weight-uncertainty band widens in low-statistics tails.The tightening is most visible in the bimodal valley of x1; the band is narrow in the bulk.Source: Appendix D · p. 20
  • Under the Gaussian composite null, tcomp asymptotically follows χ2 with J degrees of freedom, or rank(S) degrees when kernels are linearly dependent on R0.The result is independent of a and the correlations Cαv; the M standardized directions drop out because the constraint on v matches the distribution of z.Source: Appendix D · pp. 20, 23
  • At Ntest = 1k, all three empirical nulls agree with observed statistics; by Ntest = 20k, composite nulls remain close, with bootstrap somewhat outperforming Gaussian.The point-null distribution has already shifted below the observed statistic at Ntest = 20k.Source: Appendix D · p. 23
  • At Ntest = Ntrain, bootstrap calibration remains fairly close to observed statistics, while point-null and Gaussian calibration fail dramatically.The comparison concerns the four-dimensional particle-jets problem with K = 256 and Ntrain = 105.Source: Appendix D · p. 23, Figure 15 · p. 24

Appendix E: Effects of kernel width

  • With σ = 0.1, Gaussian calibration is miscalibrated for Ntest ≥10k, while bootstrap median p-values rise above 0.5 at large Ntest.The point null is insufficient and is rejected except at the smallest Ntest; the bootstrap deviation is attributed to spurious calibration.Source: Appendix E · pp. 23–24
  • Small-scale tests can exploit SIGMA’s overfit fine structure, so the authors advise against kernels substantially finer than the data’s natural length scales.Finite-sample GMM overfitting is treated as variance to capture, whereas strong regularization would induce model bias that the uncertainty scheme cannot capture.Source: Appendix E · p. 24
Loading 2610.37398v1…