R\`enyi-Regularized Flow-Based Variational Inference for Bayesian Calibration

1. Introduction.

• Normalizing flows offer flexible distributions with tractable sampling and log-density evaluation, while the standard variational objective considered is the ELBO from reverse Kullback–Leibler divergence.The introduction contrasts variational inference with MCMC methods whose computational cost can be prohibitive when simulator likelihood evaluations are expensive.Source: §1 · p. 1
• Regularization reduced the near-collapse fraction from 50% to 25% in the two-parameter calibration experiment.The experiment compared a connected bimodal posterior with one whose modes were separated by a region of near-zero probability mass.Source: §1 · p. 1
• The study tests whether adding a prior-referenced R´enyi divergence to the negative ELBO encourages approximate posteriors to cover prior support.The benchmark includes both connected and well-separated bimodal cases.Source: §1 · p. 1

2.1. Bayesian model calibration.

• With conditionally independent Gaussian observation errors and a Gaussian prior independent of context, calibration estimates model parameters from observations and their contexts.The prior mean is θb, and B and R are positive-definite covariance matrices.Source: §2.1 · p. 2

2.2. Variational inference with normalizing flows.

• Variational inference approximates the posterior with a parameterized distribution, modeled here as a normalizing flow that transforms a base distribution through a differentiable, invertible map.The variational approximation is obtained by minimizing the Kullback–Leibler divergence, equivalently the negative ELBO.Source: §2.1 · p. 2, §2.2 · p. 2

2.3. Connection with Tikhonov regularization.

• Under the Gaussian calibration assumptions, the negative ELBO is the expected Tikhonov functional minus differential entropy, balancing fit toward low-loss regions against dispersion.A restricted variational family and its optimization may still concentrate near one mode, particularly when modes are separated by low-probability regions.Source: §2.3 · p. 2

3.1. Regularized objective.

• The regularization is intended to discourage too little variational mass in regions with appreciable prior probability.The divergence direction is from the prior to the approximation; its estimation procedure is described in section C.Source: §3.1 · p. 3
• The regularized objective adds λ times the R´enyi divergence from the prior to the variational approximation to the negative ELBO.For ρ > 1, zero approximation density on a set with positive prior probability makes this divergence infinite.Source: §3.1 · p. 3

3.2. Adaptive weight.

• An adaptive weight uses gradient normalization, with its update applied at the next training step and held fixed when differentiating the regularized objective with respect to ϕ.The update uses smoothing coefficient β, target gradient ratio r, and numerical stabilizer ϵ.Source: §3.2 · p. 3

4.1. Synthetic benchmark.

• The synthetic calibration uses one context and a synthetic observation with diagonal Gaussian noise covariance 5 × 10^-2I10.The observation is generated from the forward model with additive noise.Source: §4.1 · p. 3
• Choosing the context and true parameter produces Case A, a curved V-shaped bimodal posterior with a substantial connection, or Case B, with modes separated by very low density.Both cases have zero prior mean and observation covariance 5 × 10^-2I10; the prior covariance is denoted B.Source: §4.1 · p. 3, Table 1 · p. 4
• The reference posterior is computed by normalizing its unnormalized grid density with trapezoidal quadrature, alongside an HMC comparison whose density is estimated by Gaussian kernel density estimation.The grid-normalization procedure is described in Appendix A.Source: §4.1 · p. 4

4.3. Numerical results.

• In Case A, the flow approximation represents both connected modes and follows the grid and MCMC references.Figure 1 shows joint-density contours and marginal densities.Source: §4.3 · p. 4, Figure 1 · p. 5
• In Case B, the flow frequently converges to one mode (near-collapse fraction 0.875), and the Gaussian-mixture approximation also converges to a single mode across eight random seeds.Figure 2 compares the flow and mixture rows against grid and MCMC references.Source: §4.3 · p. 4, Figure 2 · p. 5
• The combination ρ = 0.8 and r = 0.2 gives favorable results for near-collapse fraction, minority-mode mass, and squared energy distance.The paper also reports extended results for other parameter values in Appendix D.Source: §4.3 · pp. 4–5
• Near-collapse falls from 0.50 to 0.250 with R´enyi regularization; forward Kullback–Leibler divergence drops from 4.912 to 3.126 and squared energy distance from 0.259 to 0.227.Reverse divergence rises from 2.401 to 3.358, and total variation rises from 0.554 to 0.617.Source: Table 3 · p. 6, §4.3 · p. 6
• Regularized Case B runs remain variable: R´enyi A represents both modes, whereas R´enyi B nearly collapses.Figure 3 compares the reference configuration with two regularized runs using different random seeds.Source: §4.3 · p. 6, Figure 3 · p. 7

5. Discussion and conclusion.

• A small expected Tikhonov term can favor concentration near either equally favored mode, while entropy maximization alone does not enforce multimodality.The authors relate this to flows capturing connected modes but frequently missing a mode when the modes are well separated.Source: §5 · p. 7
• Mode-seeking results depend on architecture, initialization, and optimization settings; prior-referenced R´enyi regularization changes the variational target and can spread probability into regions less favored by the data.In the comparison, forward divergence and energy distance decrease, while reverse divergence and total variation increase.Source: §5 · p. 7
• R´enyi regularization can improve mode coverage, but its benefit must be weighed against bias from modifying the Bayesian objective and optimization variability.Source: §5 · p. 7

• The unnormalized posterior is evaluated on a Cartesian grid, and trapezoidal quadrature weights are used for normalization and integral approximations.Source: Appendix A · p. 8
• The reference posterior and evaluated approximation are normalized on the same grid domain.Source: Appendix A · p. 8
• Gaussian kernel density estimation is used when density evaluation at arbitrary points is intractable.Source: Appendix A · p. 8

• The matched-flow Case B comparison uses the same architecture for unregularized and Rényi-regularized approximations.Source: Table 4 · p. 8
• The prior-expectation form of the Rényi divergence is estimated using a log-sum-exp estimator [14].Source: Appendix C · p. 9
• The estimator is generally biased because of the logarithm, but is consistent when the expectation is finite.Source: Appendix C · p. 9

• Figure 4 reports metric changes across Rényi orders ρ and gradient ratios r relative to a reference configuration, with green for improvement and red for deterioration.Source: Figure 4 · p. 9