Scalable AI Uncertainty Quantification via Generalized Laplace Active Subspaces

1 Introduction

• Scientific and engineering applications often need predictions with defined bounds of uncertainty, while plausibility alone is insufficient for quantitative domains.The paper contrasts systematic validation and uncertainty analysis for physics-based models with the greater challenge of applying them to AI systems.Source: §1 · p. 2
• AI systems have many free parameters without physical interpretation, and their overconfident predictions may fail on previously unseen data.This makes parameter uncertainty difficult to assess using the kinds of mathematical-model insights available for physics-based approaches.Source: §1 · pp. 2–3
• Full Bayesian inference over neural-network weights is computationally infeasible at the extremely large parameter dimensions, making scalable posterior approximations necessary.The Bayesian ideal places a prior on weights, conditions on data, and propagates the posterior through the network.Source: §1 · p. 3
• Variational approaches such as Bayes by Backprop optimize a Gaussian posterior approximation, but balancing data fit against prior KL divergence can require specialized training schedules.The paper contrasts this with the post-hoc Laplace approximation centered at pretrained weights.Source: §1 · p. 3
• The paper’s generalized low-rank Laplace approximation uses dominant Gauss–Newton curvature directions of the empirical mean loss and matrix-free products, avoiding formation of the full covariance.It provides closed-form posterior variances in the retained subspace and calibrates prior variance using a generalized-evidence criterion.Source: §1 · p. 3
• Mean-loss scaling avoids data-size-driven posterior contraction, whereas standard scaling can require weak-curvature directions that increase cost and reduce predictive coherence.Propagating posterior samples through the nonlinear network can shift the predictive mean or distort uncertainty intervals.Source: §1 · p. 4

2.1 Generalized Laplace active subspaces

• Eigenpairs are computed using automatic differentiation, matrix-free vector products Gv, and the Lanczos algorithm, avoiding explicitly forming G.Source: §2.1 · p. 5
• The method approximates uncertainty locally around pretrained weights in a low-rank subspace of dominant generalized Gauss-Newton directions, with retained variances in closed form and prior variance calibrated empirically.The subspace dimension is small relative to the full weight dimension.Source: §2.1 · pp. 5–6

2.2 Regression with known observational noise

• With N = 50, generalized scaling reaches coverage with d = 2 and coherent intervals, while standard scaling needs d = 31 and yields wider, less coherent intervals.At N = 1000, generalized scaling remains essentially unchanged, whereas standard scaling requires d = 43 and prediction and interval quality degrade.Source: §2.2 · p. 7, Figure 3 · p. 9, Figure 4 · p. 10

2.3 Regression with unknown observational noise

• In the calibration study, generalized scaling reached target coverage at d = 3, whereas standard scaling never did within the tested range and covered approximately 60% at d = 50 for 90% CIs.With the linear predictive posterior and 95% intervals f(x; θ0) ± 2σf, generalized scaling met the coverage criterion at d = 4; standard scaling stagnated.Source: §2.3 · p. 10
• For the concrete-strength network, the 1000 most-sensitive weights—0.1% of the total—account for 95% of curvature sensitivity, leaving 99.9% to account for 5%.The active subspace in this case has d = 3 dimensions and is localized in weight space.Source: §2.3 · pp. 12–13, Figure 8 · p. 13

2.4 Autoregressive classification for molecular generative modelling

• Subsampling the molecular training data yields reasonable alignment with a reference dominant eigenvector using a small data fraction, though an outlier was attributed to likely Lanczos non-convergence.Subsequent analysis used Nsub = 2000 SMILES, corresponding to N = 93k data points.Source: §2.4 · p. 14
• In molecular generation, standard scaling leaves validity near baseline, whereas generalized scaling lowers validity and, at d = 3, increases the chance of reaching the 256-token sequence limit.These comparisons concern generated SMILES strings and, for sequence length, valid strings only.Source: §2.4 · p. 15, Table 1 · p. 15, Figure 10 · p. 15
• With d = 2 in molecular generation, the first 7.6% of cumulative activity scores account for 95% of total sensitivity.The cumulative activity scores saturate less quickly than in the concrete-strength case.Source: §2.4 · pp. 15, 17, Figure 12 · p. 17

3.1 Posterior scaling and active-subspace dimension

• In nonlinear regression, adding weak-curvature directions can produce wider, less stable intervals and move the predictive mean away from the pretrained model.The larger perturbations may leave the local region where the Laplace approximation is accurate.Source: §3.1 · p. 18
• Generalized scaling achieved desired coverage with d = 4 or fewer, while standard scaling required d = 43 at N = 1000.Under standard scaling, leading-direction variances contract with data size, pushing coverage calibration toward weak-curvature directions.Source: §3.1 · p. 18
• Standard scaling requires higher posterior rank at N = 1000 than at N = 50, increasing the curvature-mode computation and Lanczos iterations.The authors note that data subsampling can be used when computing eigenmodes for large datasets.Source: §3.1 · pp. 18–19
• For molecule generation, standard scaling stays close to the pretrained baseline, whereas generalized scaling increasingly changes sequence lengths and broadens molecular-property distributions as d grows.At d = 3, the generalized-scaling generator produces long, carbon-rich molecules and SlogP values above 5 more often.Source: §3.1 · pp. 19–20
• The REINVENT experiments are a sensitivity study of posterior weight perturbations in free-running generation, not an evaluation of improved predictive distributions.The pretrained model was already reasonably calibrated at the token level.Source: §3.1 · p. 20

3.2 Related work

• Unlike subspace inference in, this method selects dominant generalized Gauss–Newton curvature directions and uses mean-loss scaling to avoid contraction driven solely by dataset size.It yields a closed-form Gaussian posterior in the active subspace rather than requiring MCMC sampling or variational optimization there.Source: §3.2 · p. 21
• Unlike, the method constructs a generalized-Bayes posterior directly in a curvature-informed subspace and propagates samples through the nonlinear network.The authors report calibrated coverage with a handful of directions, compared with subspace dimensions in the tens to hundreds in.Source: §3.2 · pp. 21–22
• The active subspace uses data-averaged local curvature at pretrained weights, rather than prior-averaged model-gradient covariance used in related approaches.Scalability comes from restricting the posterior to this low-rank subspace, rather than imposing layer-wise Kronecker independence assumptions as in.Source: §3.2 · p. 22

• The methods section presents a low-rank generalized Laplace framework for quantifying uncertainty in neural networks.Source: §4 · p. 22

4.1 Statistics

• The framework targets scalable uncertainty quantification by propagating a low-rank posterior over pretrained network weights through the network.It replaces the intractable full Hessian with a positive-semidefinite generalized Gauss–Newton approximation and extracts dominant eigenvectors using matrix-free automatic-differentiation products.Source: §4.1 · p. 22

4.1.1 Generalized Bayes formulation

• Generalized Bayes connects data and parameters through empirical mean loss, retaining a prior-to-posterior update without requiring a traditional likelihood.For regression, standard scaling uses β = Nσ^-2 while mean squared loss uses β = σ^-2; for classification, the corresponding values are β = N and β = 1.Source: §4.1.1 · pp. 22–24

4.1.2 Laplace approximation of the generalized posterior

• The Laplace approximation uses a Gaussian centered at pretrained weights, with covariance determined by local loss curvature and a quadratic expansion around that anchor.It is a pragmatic local estimate, not a claim that the full neural-network posterior is Gaussian; forming and inverting the full Hessian is intractable for neural networks.Source: §4.1.2 · pp. 24–25

4.1.3 Low-rank posterior variance

• Retaining only the d dominant eigenvalues gives a low-rank posterior, whose quality depends on curvature approximation and prior variance.Source: §4.1.3 · p. 25
• With mean-loss scaling, curvature eigenvalues reflect average local sensitivity instead of growing linearly with N, avoiding sample-size-driven contraction of leading directions.Under standard scaling, posterior variances decrease as N grows; in a low-rank projection this can make leading active directions excessively narrow.Source: §4.1.3 · p. 25
• Large-curvature directions contract more strongly, while the prior variance supplies baseline uncertainty in directions with little data information.The prior variance sets the overall uncertainty scale and is estimated using an empirical Bayes procedure.Source: §4.1.3 · pp. 25–26

4.1.4 Approximate Hessians

• The generalized Gauss–Newton matrix G approximates the Hessian while avoiding second-order derivatives of the network output.It coincides with the true Hessian only under specific assumptions.Source: §4.1.4 · pp. 26–27
• Ignoring the second-derivative term is justified when residuals are small, and exact when all residuals are zero.The text notes that zero residuals are unrealistic and would indicate overfitting.Source: §4.1.4 · pp. 26–27
• Using G is equivalent to replacing the network with its first-order Taylor expansion, yielding linearized Laplace predictive distributions.This alternative justification is attributed to.Source: §4.1.4 · p. 27
• The active subspace is the dominant eigenspace of G for the empirical mean loss, a slight departure from its original definition.The paper motivates this focus by evidence that loss gradients align with leading Hessian eigenvectors [38].Source: §4.1.4 · p. 27

4.1.5 Matrix-free computation of dominant eigenpairs

• Each contribution to Gv can be computed with one forward and one backward network pass, without explicitly forming G or the output Jacobian.The procedure uses mini-batches in practice.Source: §4.1.5 · p. 28
• Lanczos uses repeated Gv products to approximate leading eigenvalues and eigenvectors through a low-dimensional tridiagonal matrix.Its Krylov basis is formed from successive applications of G to an initial vector.Source: §4.1.5 · pp. 27–28
• The projection uses the first d columns of VlUl, with l>d; the reported setting is l=d+10.A sufficiently large basis l is important for reducing the approximation error shown in Figure 2.Source: §4.1.5 · p. 29

4.1.6 Empirical generalized Bayesian subspace inference for the prior variance

• Empirical Bayes is performed in projected coordinates w=P^Tθ because full-space optimization is driven toward excessively high prior variance by flat, uninformative directions.The projected space has dimension d and is spanned by dominant eigenvectors of G.Source: §4.1.6 · p. 29
• The projected prior variance is chosen by maximizing generalized evidence after a Laplace approximation around w0=P^Tθ0.The approximation assumes w0 is sufficiently close to a stationary point to neglect the first-order Taylor term.Source: §4.1.6 · pp. 29–30
• The low-rank determinant calculation uses only the d dominant eigenvalues, with the projected curvature approximated by P^TGP.This replaces the full Hessian in the projected calculation.Source: §4.1.6 · p. 30
• The resulting prior scale makes the projected parameter norm consistent with the amount of variance reduction in the active subspace.The derivation is performed using the projected norm rather than the full parameter norm.Source: §4.1.6 · pp. 30–31
• The scalar equation has a unique solution because the relevant function is smooth and strictly decreasing in α, and it is solved by bisection.The equation defining the prior variance is implicit and nonlinear.Source: §4.1.6 · p. 31

4.1.7 Calibrating d

• For regression, d is chosen to minimize the mismatch between confidence-interval coverage and nominal coverage on a calibration set.Each candidate d requires empirical Bayes estimation, so d and σ0 are estimated jointly.Source: §4.1.7 · p. 31
• For classification, calibration means that the correct class lies in the 1−γ set with approximately probability 1−γ.A similar calibration procedure can be applied to classification networks [29].Source: §4.1.7 · p. 31
• Weight perturbations may diminish calibration when a classification network is already well calibrated.The text identifies this as a possible effect of the additional stochasticity.Source: §4.1.7 · p. 31

4.1.8 Posterior prediction

• Orthogonal directions retain prior variance, and their vast number can collectively produce overly wide predictive intervals.The retained posterior variance covers only the dominant curvature eigenspace.Source: §4.1.8 · p. 31
• The low-rank approximation omits orthogonal directions and concentrates uncertainty in the data-informed subspace.The authors state that this choice follows approaches such as.Source: §4.1.8 · p. 31
• Monte Carlo forward passes propagate posterior weight samples through the network, but nonlinear propagation may move the predictive mean away from the MAP prediction.The samples approximate predictive expectations.Source: §4.1.8 · p. 32
• Linearized Laplace locally approximates network outputs, keeps the predictive mean at the MAP prediction, and yields a closed-form covariance from the Jacobian and low-rank posterior covariance.For scalar regression, the resulting pointwise predictive variance is also expressed in closed form.Source: §4.1.8 · p. 32

Supplementary information.

• No Supplementary Materials are available for this article.Source: Supplementary information · p. 32

Funding

• P.V.C. reports funding from the UK Engineering and Physical Sciences Research Council for the UK Consortium on Mesoscale Engineering Sciences project.Source: Funding · p. 32

Competing interests

• The authors declare no competing interests.Source: Competing interests · p. 32

Data and code availability

• Training data, a pretrained REINVENT model, and reproduction notebooks are available at.Source: Data and code availability · p. 33

Declaration on the use of artificial intelligence tools

• The authors used ChatGPT 5.5 for language editing and manuscript discussion, and state that they reviewed and finalized the scientific content.Source: Declaration on the use of artificial… · p. 33

Appendix A ChEMBL vocabulary

• Table A1 lists the vocabulary used by the autoregressive SMILES generator.Source: Table A1 · p. 34, Appendix A · p. 34
• The vocabulary includes start and stop tokens, bond and branch symbols, ring-closure indices, atom tokens, and charged or aromatic atom tokens.Source: Table A1 · p. 34
• Table A2 presents 10 SMILES strings from the ChEMBL database.Source: Table A2 · p. 34, Appendix A · p. 34