PaperlayerView exact revision on arXiv

Source-linked AI summary

Theoretical and Practical Advances in Hierarchical Bayesian Deep Learning Methodology

Neil K. Chada, Simon Cotter, Kody J. H. Law, Xinzhu Liang

arXiv:2610.19074v1cs.LGstat.COstat.ME

TL;DR

Strong dependencies between neural-network parameters and prior hyperparameters make joint posterior inference challenging for conventional MCMC. The paper develops MLPMMH using coupled multilevel SMC and PMMH, and a practical SBMC-based approximation; it proves a canonical MSE–cost rate for MLPMMH and reports empirical support for its methods.

  • Problem

    Strong dependence between network parameters and prior hyperparameters can make joint-posterior inference difficult for generic MCMC.

    Source: §1 · p. 2

  • Method

    MLPMMH combines PMMH hyperparameter inference with coupled SMC across a TNN-induced hierarchy, while S-MLPMMH approximates its sampling components with scalable Bayesian Monte Carlo.

    Source: §2.1 · p. 4, §3.2 · p. 7

  • Results

    Under the stated assumptions, MLPMMH has MSE O(ε^2) at expected cost O(ε^-2), and the conclusion reports comparable predictive accuracy and NLL for moderate width decay versus unregularized and standard weight-decay baselines.

    Source: §4.1 · p. 7, §6 · p. 12

  • Takeaways & Limitations

    The synthetic regression experiments support the predicted multilevel regularity and canonical MSE–cost rate, while MNIST experiments demonstrate the practical approximation’s predictive and uncertainty-quantification behavior.

    Source: §5.3.3 · p. 11, §6 · p. 12

    S-MLPMMH is not an exact pseudo-marginal sampler for the original joint posterior, and MLPMMH’s theoretical results do not directly apply to this approximation.

    Source: §3.2 · p. 7

Abstract

from arXiv · show

Reliable uncertainty quantification remains an important challenge in AI. Hierarchical Bayesian methods provide a principled framework for this by representing unknown quantities through posterior distributions. Joint inference of neural-network parameters and prior hyperparameters is challenging for conventional Markov chain Monte Carlo (MCMC) methods due to strong dependencies. In this work, we introduce the multilevel particle marginal Metropolis-Hastings (MLPMMH) method, which combines pseudo-marginal inference for the hyperparameters with coupled sequential Monte Carlo (SMC) across hierarchy of network-width truncations for conditional parameter inference. The hierarchy is induced by a trace-class neural network (TNN) prior with width-dependent decay, motivated empirically by the compressible structure observed in trained neural-network parameters. Under standard multilevel coupling assumptions, we establish that MLPMMH achieves mean-square error (MSE) $\mathcal{O}(\varepsilon^2)$ at expected computational cost $\mathcal{O}(\varepsilon^{-2})$, and verify the predicted rate on a Bayesian regression problem. To address larger-scale applications, we further introduce S-MLPMMH, a practical approximation that replaces the exact sampling components with a PCA-based hierarchy and scalable Bayesian Monte Carlo. Experiments on MNIST demonstrate competitive predictive performance and uncertainty quantification while exploiting the parallel structure of the approximation. Software has been made publicly available at https://github.com/submission-code-anon/mlpmmh-anon.

1 Introduction

The introduction frames joint inference of neural-network parameters and prior hyperparameters as difficult for generic MCMC because of strong dependencies. It introduces a multilevel PMMH approach, its theoretical and practical contributions, and the paper’s empirical plan.

1 Introduction

  • Strong parameter–hyperparameter correlations can make joint-posterior estimation difficult for generic MCMC, including HMC.The introduction identifies this dependence as a challenge in hierarchical Bayesian inference.Source: §1 · p. 2

1.1 Contributions

  • MLPMMH combines coupled SMC conditional-parameter inference across resolutions with PMMH hyperparameter estimation to form multilevel increment estimators.The paper places the method in the context of multilevel Monte Carlo for TNN priors.Source: §1.1 · p. 2, §1 · p. 2
  • The authors report a theorem on MLPMMH convergence and empirical tests of regression rates, MNIST classification, weight-and-bias decay, and inner-sampler variants.The introduction’s outline places the theory in Section 4, main experiments in Section 5, and additional MLSMC comparisons in the appendix.Source: §1.1 · pp. 2–3, §1.2 · p. 3
  • The paper also introduces S-MLPMMH, which estimates increments with SBMC ensembles conditional on hyperparameters drawn from a Laplace approximation.This is presented as a practical approximation to the main method.Source: §1.1 · p. 2

1.2 Outline

  • The paper proceeds from model and prior formulation to MLPMMH and S-MLPMMH, then theory, numerical tests, and conclusions.Appendix B is described as containing SMC background, MLSMC methods, supporting theory, experiments, and proofs.Source: §1.2 · p. 3, Figure 1 · p. 3

2 Model Formulation

The model jointly infers neural-network parameters and prior hyperparameters from data. A trace-class prior supplies width-dependent decay, while nested network widths and a truncation map establish the multilevel structure.

2 Model Formulation

  • The model infers network parameters θ and hyperparameters µ jointly from paired input–output data D.The network maps inputs and parameters to outputs in the specified output space.Source: §2 · pp. 3–4

2.1 Trace-class neural network priors

  • TNN weights and biases are defined layer by layer under a prior designed to impose decay in network width.The prior is based on a pushforward of a countably infinite Gaussian random variable and is described as converging strongly.Source: §2.1 · p. 4
  • The width-decay prior provides an inductive bias and strong convergence that the paper uses to support applying MLMC.The decay exponent s is held constant, while µ=(σA, σb) can have a two-dimensional log-normal prior.Source: §2.1 · p. 4

2.2 Multilevel Structure

  • At level l, hidden layers have width n_l=2^l, while the input and output widths are n and m.The parameter vector θ_l comprises the layer weights and biases and has prior π̄_l.Source: §2.2 · p. 4
  • The truncation map T_l takes fine-level parameters to a coarse level by retaining the corresponding lower-width parameter components.The passage describes the map’s assignments for hidden-layer weights and biases and analogous assignments at the input and output layers.Source: §2.2 · p. 4
  • Each level defines a neural-network output and quantity of interest using the same output map Ψ.The level-specific quantity is φ_l(x, θ_l, µ)=Ψ(g_l(x, θ_l, µ)).Source: §2.2 · p. 4
  • The multilevel construction forms an approximate coupling between levels using a joint likelihood chosen as a maximum coupling.The coupling passage identifies the joint likelihood L_l(θ_l, θ_{l−1}|µ) as the maximum coupling.Source: §2.2 · pp. 4–5
  • Although µ is retained in the network notation, once θ is fixed the forward map does not explicitly depend on µ; its distribution depends on µ through the prior.Source: §2.2 · p. 4

3 Hierarchical Bayesian Estimation of Neural Networks with Trace-Class Priors

The paper embeds multilevel SMC within PMMH to jointly estimate neural-network parameters and prior hyperparameters under trace-class priors. It also proposes S-MLPMMH to reduce serial computation, while noting that this approximation is not exact and lacks the direct theoretical guarantee of MLPMMH.

3 Hierarchical Bayesian Estimation of Neural Networks with Trace-Class Priors

  • The main contribution is a hierarchical Bayesian estimation method based on particle MCMC, tailored to trace-class neural-network priors.The section identifies PMMH as the specific particle MCMC framework.Source: §3 · p. 5

3.1 MLPMMH

  • MLPMMH embeds an MLSMC sampler in PMMH to estimate the marginal likelihood while inferring log-normal prior standard-deviation hyperparameters for TNN weights and biases.The level-specific joint posterior is part of the multilevel setup.Source: §3.1 · p. 5
  • The MLPMMH estimator combines level-wise increments, with the coarsest reference increment set to zero.The supplied estimator passages identify the level range and the convention φ_{Lmin−1}:=0.Source: §3.1 · p. 5
  • Within each level, the PMMH procedure initializes a hyperparameter and runs the MLSMC algorithm to obtain particle and normalizing-constant estimates.The algorithm passage also describes resampling and proposing subsequent hyperparameters.Source: §3.1 · p. 6
  • MLPMMH parallelizes across levels and SMC particles, but its PMMH chain remains sequential across iterations.Source: §3.1 · p. 6

3.2 S-MLPMMH

  • S-MLPMMH replaces the serial outer PMMH chain with a Laplace-based SBMC approximation and the inner MLSMC sampler with MAP-centered SBMC.Its hyperparameter approximation is centered at an empirical-Bayes MAP estimate.Source: §3.2 · p. 6
  • S-MLPMMH preserves multilevel corrections and permits independent outer hyperparameter samples and inner computations to run in parallel, reducing serial and inner-sampling costs.Source: §3.2 · p. 7
  • Unlike MLPMMH, S-MLPMMH is not an exact pseudo-marginal sampler for the original joint posterior, and MLPMMH’s theoretical results do not directly apply to it.Source: §3.2 · p. 7

4 Theoretical results

The theoretical results establish an MSE bound for MLPMMH and a canonical computational-cost rate under stated assumptions. They contrast this with a higher PMMH cost exponent that can worsen with dimension.

4.1 Main results

  • Theorem 4.1 assumes E.1, E.3, and E.2 with β > γ, and applies to suitable φ: Θ × S → R.Source: §4.1 · p. 7
  • The MLPMMH estimator has a canonical computational-cost rate of O(ε^-2), alongside the section’s main MSE-bound contribution.Source: §4.1 · p. 7, §4 · p. 7
  • For the same MSE target, PMMH has cost O(ε^(-2-γ/α)); the paper says γ/α increases with dimension, so PMMH often faces the curse of dimensionality.Source: §4.1 · p. 7

5 Numerical Experiments

The experiments test MLPMMH’s convergence on synthetic Bayesian regression, examine TNN width-decay structure, and evaluate a PCA-based practical approximation on MNIST. They report regression convergence, rank-wise weight compressibility, and calibration behavior as the maximum level increases.

5.1.1 Synthetic regression dataset

  • MSE=O(COST^-1) from Lmin=4, matching the predicted MLPMMH convergence rate on synthetic Bayesian regression.The figure plots MSE against computation cost using 10 realizations.Source: §5.1.1 · p. 8, Figure 2 · p. 8

5.2 Empirical verification of TNN prior structure

  • Sorted weight magnitudes show rank-wise decay; width-decay preserves leading-rank structure while strengthening tail decay, especially in hidden layers.Because sorting removes original width indices, this is a compressibility diagnostic, not direct evidence of index-wise TNN structure.Source: §5.2 · p. 9, Figure 3 · p. 9
  • For s ≤4, width-decay maintains accuracy, NLL, Brier score, and ECE comparable to NoReg and weight-decay; s=10 causes instability and poor performance.Table 1 reports MNIST results averaged over five replicates.Source: §5.2 · p. 9, Table 1 · p. 10

5.3.1 Feature matrix from the pre-trained LeNet NN

  • A frozen LeNet produces 256-dimensional features from the second MaxPool layer for the subsequent Bayesian analysis.The LeNet is pretrained on MNIST using supervised learning and then fixed.Source: §5.3.1 · p. 10

5.3.2 PCA hierarchy and multilevel coupling.

  • At level l, the PCA hierarchy retains the first 2^l components, while level l−1 shares the leading coordinates to correlate fine and coarse likelihoods.The feature matrix has 256 columns, and both training and test features are expressed in the PCA basis.Source: §5.3.2 · p. 11

5.3.3 Results

  • ECE remains low for Lmax ≤4 but rises from Lmax=5; the signed calibration gap stays positive, indicating average under-confidence.The authors describe the resulting uncertainty estimates as conservative.Source: §5.3.3 · p. 11
  • The models are small enough that idealized parallel runtime is nearly constant across levels, while estimated FLOPs increase with Lmax.The authors note this experiment does not demonstrate level-dependent wall-clock savings.Source: §5.3.3 · p. 11

6 Conclusion

The paper develops Bayesian neural-network inference for joint parameter and hyperparameter estimation using multilevel Monte Carlo and a trace-class neural-network prior. Its model formulation uses a joint posterior, while the prior imposes width-direction decay and is motivated by scaling limitations of other priors.

6 Conclusion

  • The method combines multilevel Monte Carlo with a trace-class neural-network prior to estimate network parameters and hyperparameters jointly.The prior imposes width-direction decay and is described as strongly convergent.Source: §6 · p. 11

A.1 Model Formulation

  • The Bayesian model targets the joint posterior over network parameters and their hyperparameter, with the network defined through layers of weights, biases, and ReLU activations.The model also defines quantities of interest from network outputs and targets their posterior expectations.Source: §A.1 · p. 15

A.2 Trace-class neural network priors

  • Trace-class neural-network priors were motivated by the poor scaling of Gaussian-field priors in high-dimensional problems and impose decay in the network-width direction.The priors were introduced to mimic Gaussian priors used in function-space inverse problems.Source: §A.2 · p. 15

B State-of-the-Art Estimation Methods for Bayesian Neural Networks

The section reviews SMC and multilevel SMC estimators for conditional parameter inference, then describes PMMH for hyperparameter inference and the S-MLPMMH approximation.

B.1 Sequential Monte Carlo estimator

  • SMC estimates expectations under the conditional parameter posterior by tempering from the prior toward the likelihood-weighted target and applying invariant Markov transitions.The tempering schedule can be selected adaptively to maintain a minimum effective sample size.Source: §B.1 · pp. 16–18

B.2.1 Multilevel Structure

  • Multilevel SMC couples neighboring resolutions to estimate differences across levels, with top-down truncation and bottom-up propagation as the two coupling directions.Top-down estimators can run independently across levels, whereas serial bottom-up sampling depends on the previous level.Source: §B.2.1 · pp. 18–19

B.2.2 Multilevel sequential Monte Carlo estimators

  • The multilevel estimators include self-normalized and ratio forms, with coupled SMC sampling used to estimate level differences.The randomized ratio variant samples levels from a probability distribution rather than fixing a maximum level in advance.Source: §B.2.2 · pp. 19–21, §B.2.3 · p. 21

B.2.4 Serial multilevel sequential Monte Carlo estimator

  • Serial MLSMC propagates samples from each coarse level to the next finer level, making its level-wise procedure non-parallelizable.Particle-wise operations and independent replications can still be parallelized.Source: §B.2.4 · p. 22

C PMMH

  • PMMH infers hyperparameters by using an unbiased SMC estimate of the marginal likelihood within a Metropolis-Hastings scheme.The SMC estimator is unbiased for the marginal likelihood used to target the hyperparameter posterior.Source: C PMMH · pp. 22–23

D Details of the S-MLPMMH approximation

  • S-MLPMMH approximates the hyperparameter posterior with an outer Laplace construction and uses MAP-centered SBMC for conditional parameter sampling.The MAP-centered inner prior shrinks covariance using a tuning parameter; the stated experiments use sθ = 0.1.Source: D Details of the S-MLPMMH approximation · p. 24

E Theoretical results

The theoretical results build on boundedness, rate-decay, and PMMH transition assumptions. The paper relates the multilevel assumptions to the standard bias, variance, and cost conditions used in MLMC.

E Theoretical results

  • The theoretical section presents MLSMC results, a trace-class neural-network variance bound, and a convergence theorem for the joint posterior.The proofs and additional results are provided in appendices.Source: E Theoretical results · p. 25

E.1 Assumptions

  • The multilevel assumptions cover boundedness and rate decay, corresponding to bias, variance, and functional-evaluation cost conditions in standard MLMC.For fixed hyperparameters, the stated assumptions fall within the original MLMC assumptions.Source: §E.1 · p. 25
  • The PMMH transition assumption is stated separately, and verification of the rate-decay and PMMH assumptions is deferred to Appendix H.5.Source: §E.1 · p. 25

E.2 MLSMC results

The section states multilevel estimator results under boundedness and rate-decay assumptions, then gives trace-class and truncation-convergence results for the neural-network prior.

E.2 MLSMC results

  • The MLSMC estimator theorems assume E.1 and E.2 with fixed µ and β > γ, and assert that a suitable level parameter can be chosen.The provided theorem statements cover self-normalized, ratio, randomized, and serial estimators.Source: §E.2 · p. 26
  • Expected COST(bφrMI) ≤ Cε^-2 for the randomized MLSMC ratio estimator, identified as the canonical rate.The theorem statement requires Assumptions E.1 and E.2.Source: §E.2 · p. 26

E.3 Trace-class neural network priors

  • For a TNN prior with s ≥ 1/2, truncation error in the network output has a decay rate that transfers to Lipschitz quantities of interest.The trace-class claim additionally assumes σ(z) ≤ |z| for all real z.Source: §E.3 · p. 27

E.4 Supporting results

This block states supporting theoretical results and outlines the numerical problem settings, including the MNIST data used in classification experiments.

E.4 Supporting results

  • Theorem E.5 is stated under Assumptions E.1 and E.2 for functions φ mapping Θ × S to Rd, with a finite constant C.The supplied statement does not include the bound that follows.Source: §E.4 · p. 27
  • Proposition E.2 assumes E.1–E.3 and a Markov chain with kernel K initialized from π, and asserts a rate β > 0 for bounded Lipschitz functions.The displayed bound following this statement is not included.Source: §E.4 · p. 28

F Additional Numerical Experiments

  • The numerical experiments compare conditional-posterior MLSMC methods, test MLPMMH regularity and complexity on synthetic regression, and assess the TNN width-decay structure.Arithmetic cost is measured by the number of basic floating-point operations in the coupled level.Source: F Additional Numerical Experiments · p. 28

F.1 Problem setting

  • The problem setting uses Gaussian likelihoods for continuous regression outputs and a softmax-based categorical likelihood for discrete classification labels.The regression density is evaluated at each observed response, while classification likelihood uses indicators for the observed class.Source: §F.1 · p. 28

F.1.1 MNIST dataset

  • MNIST contains 60,000 training and 10,000 test images of digits 0–9, represented as standardized 784-dimensional vectors in [0, 1]^784.The text describes using the first Ntr training examples and first Ntest test examples.Source: §F.1.1 · p. 29

F.2 Comparison among MLSMC methods

The experiments test multilevel regularity and complexity, examine burn-in for MLPMMH, and assess the TNN width-decay structure and PCA hierarchy. The reported decay estimates support the hierarchy's convergence conditions, while moderate width-decay strengths retain comparable predictive metrics to the baselines.

F.2 Comparison among MLSMC methods

  • For fixed µ and s = 2, the regularity test reports strong convergence β = 3 and bias decay at least α = β/2 from level l = 4 onward.The first level is treated as a single level and need not follow the rate.Source: §F.2 · p. 29, Figure 5 · p. 30

F.3 Numerical justification for MLPMMH

  • The autocorrelation plots for σA and σb suggest that 50 burn-in samples are reasonable for the MLPMMH MCMC.The regression experiment uses 50 realizations and 100 samples per index.Source: §F.3 · p. 29, Figure 7 · p. 31

F.4 Empirical verification of TNN prior structure

  • The TNN-style width-decay prior is assessed using MAP training alongside unregularized and isotropic-Gaussian weight-decay regimes.The PCA featured-matrix decay is compared with the TNN-style weight decay.Source: §F.4 · pp. 29–30, §F.5 · p. 31

F.2 Comparison among MLSMC methods

  • The regression complexity-test rates are −0.962 for MLSMCSN, −0.916 for MLSMCRE, −0.885 for rMLSMC, and −0.870 for Serial MLSMC.The comparison computes MSE from 30 realizations for each experiment.Source: §F.2 · p. 29, Figure 6 · p. 30

F.5 Practical application of MLPMMH

  • The PCA eigenvalues decay with fitted log-log slope −0.615, corresponding to empirical s = 1.23; prior-predictive variance decay is β = 1.408, exceeding cost rate γ = 1.The text states that β > γ satisfies the conditions for MLPMMH convergence.Source: §F.5 · p. 31

G Practical settings in MLSMC methods

The practical settings select levels and per-level sample counts using observed decay rates and a target error. The randomized estimator samples levels independently, while slower variance decay than cost growth leads to sub-canonical complexity.

G.1 Multilevel sequential Monte Carlo estimators

  • For MLSMC, empirical Lmin is determined from regularity results, and Lmax and per-level sample counts are estimated so MSE is below ε², with variance and bias terms each below ε².Because each level is independent, the algorithm can be parallelized.Source: §G.1 · pp. 31–34
  • When variance decay is no faster than cost growth (β ≤ γ), MLSMC incurs an additional exponent in its ε–COST complexity.The remark specifies bias decay O(2^-αl) and cost growth O(2^γl) for this sub-canonical case.Source: §G.1 · p. 34

G.2 Randomized multilevel sequential Monte Carlo estimators

  • The randomized MLSMC estimator draws level indices from a positive probability distribution and allocates counts by sampled level; levels can be run in parallel.Its sub-canonical regime is also tied to the mismatch between variance decay and cost growth.Source: §G.2 · p. 34

H Proofs of Theoretical results

H.1 Proof relating to Theorem 4.1

The appendix derives the MLPMMH complexity result by controlling bias, variance, and covariance under multilevel assumptions, then verifies supporting bounds and assumptions. It closes by identifying data scale, network depth, and comparisons with generic MCMC and non-fully Bayesian methods as future directions.

H.1 Proof relating to Theorem 4.1

  • Under β > γ and the stated assumptions, the proof chooses levels and sample counts to obtain the canonical total cost O(ε^-2) while controlling the MSE terms.The proof treats bias, variance, and covariance separately and selects the maximum level in relation to |log(ε)|/β.Source: §H.1 · pp. 35–36

H.2 Proof relating to Theorem E.5

  • Theorem E.5's proof expands the coupled-level estimator terms and uses Jensen’s inequality to bound them under the stated assumptions.The argument checks six terms against the desired level-decay order.Source: §H.2 · pp. 37–38

H.3 Proof relating to Lemma E.1

  • The proof of Lemma E.1 establishes the stated algebraic identity by expanding its right-hand-side terms.Source: §H.3 · p. 38

H.4 Proof relating to Proposition E.2

  • The Proposition E.2 proof considers a Markov chain with kernel K and represents its relevant variables as Vk = (θk,l, µ).The proof is described as similar to Proposition A.1 in Jasra et al. [2018].Source: §H.4 · pp. 38–39

H.5 Verification of Assumptions

  • The assumption verification establishes E.3 and then checks the bias and variance bounds in E.2 under the joint posterior.The stated conditions include boundedness and Lipschitz assumptions on the functions involved.Source: §H.5 · pp. 39–41

I Future research directions

  • The authors identify larger, more challenging datasets such as IMDB or CIFAR as future applications beyond examples used to verify theory and justify the approach.Source: I Future research directions · p. 42
  • Extending the analysis from width to network depth is proposed as a way to better understand the prior’s algorithmic impact.Source: I Future research directions · p. 42
  • A comparison with generic methods such as HMC remains open for the multilevel trace-class neural-network setting.The text says the comparison exists with parallel-SMC, but not in this setting.Source: I Future research directions · pp. 42–43
Loading 2610.19074v1…