PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiscale Invertible Generative Networks for High-Dimensional Bayesian Inference

Shumao Zhang, Pengchuan Zhang, Thomas Y. Hou

arXiv:2105.05489v1stat.MLcs.CVcs.LGstat.CO

TL;DR

High-dimensional Bayesian inference struggles with disconnected posterior modes and the curse of dimensionality. MsIGN addresses this by generating samples from coarse to fine scales with an invertible network and training it in stages using Jeffreys divergence. It captures multiple modes and improves posterior approximation on inverse problems while outperforming image-synthesis baselines in bits-per-dimension.

  • Problem

    High-dimensional Bayesian inference must approximate posteriors whose modes become increasingly difficult to detect and capture.

    Source: §1 · p. 1

  • Method

    MsIGN uses multiscale invertible transport to generate samples coarse-to-fine and multi-stage training to minimize Jeffreys divergence.

    Source: §1 · p. 2

  • Results

    MsIGN captures multiple modes and accurately approximates high-dimensional posteriors while achieving superior bits-per-dimension among image-synthesis baselines.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Multiscale structure provides a basis for applying invertible generative networks to Bayesian inference problems up to dimension 4096 and to image synthesis.

    Source: §1 · p. 1, §7 · p. 8

    For natural images, the method assumes a simple Gaussian prior with scalar covariance learned from the data.

    Source: §6.2 · p. 7

Abstract

from arXiv · show

We propose a Multiscale Invertible Generative Network (MsIGN) and associated training algorithm that leverages multiscale structure to solve high-dimensional Bayesian inference. To address the curse of dimensionality, MsIGN exploits the low-dimensional nature of the posterior, and generates samples from coarse to fine scale (low to high dimension) by iteratively upsampling and refining samples. MsIGN is trained in a multi-stage manner to minimize the Jeffreys divergence, which avoids mode dropping in high-dimensional cases. On two high-dimensional Bayesian inverse problems, we show superior performance of MsIGN over previous approaches in posterior approximation and multiple mode capture. On the natural image synthesis task, MsIGN achieves superior performance in bits-per-dimension over baseline models and yields great interpret-ability of its neurons in intermediate layers.

1. Introduction

High-dimensional Bayesian inference is difficult because posterior modes become disconnected as dimension grows, while multiscale structure can expose a lower-dimensional approximation. MsIGN exploits this structure with coarse-to-fine invertible generation and multi-stage Jeffreys-divergence training, achieving strong posterior and image-synthesis results.

  • Posterior-mode detection becomes exponentially costly with dimension, creating a mode-collapse challenge for existing Bayesian inference methods.Source: §1 · p. 1
  • Multiscale structure makes high-dimensional inference more tractable when observations depend mainly on coarse-scale structure rather than fine-scale details.A kilometer-scale precipitation model can approximate a meter-scale model while using 10^6 fewer grid points.Source: §1 · p. 1
  • MsIGN maps Gaussian random seeds to posterior samples through invertible coarse-to-fine transport, progressively increasing dimension with prior conditioning and flow refinement.At each scale, the prior conditioning layer upsamples a coarse sample and the invertible flow refines it.Source: §1 · p. 2
  • MsIGN minimizes Jeffreys divergence using multi-stage training, with each coarse-to-fine stage supplying a proposal distribution for importance sampling at the next stage.The objective combines both KL directions, while stagewise training helps avoid mode missing in high-dimensional settings.Source: §1 · p. 2
  • Across two high-dimensional inverse problems and natural image synthesis, MsIGN captures multiple posterior modes, improves posterior approximation, and outperforms baseline models in bits-per-dimension.The paper also reports interpretable neurons in intermediate image-synthesis layers.Source: §1 · p. 2

2. Theoretical Motivation

The theoretical motivation assumes that observations depend more strongly on a low-dimensional coarse representation than on fine-scale details. This supports approximating the full posterior through a recursively constructed coarse-scale posterior and conditional fine-scale prior.

  • A downsampling operator A maps the high-dimensional state x to a coarse representation Ax with dimension dc < d, while B can lift it back to the input dimension.Average pooling is one example of A, and nearest-neighbor upsampling is one example of B.Source: §2 · p. 2
  • The multiscale assumption is that the observation relies more on global coarse-scale structure than local fine-scale structure of x.Source: §2 · p. 2
  • When F(B(Ax)) approximates F(x), the coarse-scale likelihood Lc(y|Ax) can approximate the full likelihood L(y|x).The construction is motivated by spatial or temporal structure and illustrated by the multiscale example in Figure 2.Source: §2 · pp. 2–3
  • Prior factorization separates the full prior into a coarse-scale prior ρc(Ax) and a conditional fine-scale distribution ρ(x|Ax).Source: §2 · p. 3
  • Together, likelihood approximation and prior decoupling yield a coarse-scale posterior qc and a distribution ˜q that can approximate the full posterior q.The coarse posterior is defined from the coarse prior and coarse likelihood, while ˜q combines qc with the conditional prior.Source: §2 · p. 3

3. Network Architecture

MsIGN generates posterior samples from coarse to fine scales by recursively upsampling prior-conditioned samples and refining them with invertible flows. Its multiscale construction preserves invertibility while separating low-dimensional structure from fine-scale detail.

Sampling strategy

  • The sampling pipeline first draws a coarse posterior sample, then samples from the conditional prior, and finally applies an invertible flow to better approximate the target.For image synthesis, the intermediate sample is an upsampled low-resolution image with prior noise before high-resolution refinement.Source: §3 · p. 3

Prior conditioning

  • The prior conditioning layer maps a coarse sample and Gaussian seed to a finer conditional sample while enforcing Ax=xc and maintaining invertibility.For Gaussian priors, the conditional layer has a closed form; for non-Gaussian priors, it can be pre-trained and then fixed.Source: §3 · p. 3

Invertible flow

  • Each invertible flow modifies the prior-conditioned distribution toward the target posterior, using Glow-style invertible blocks that can be initialized as the identity.The overall transport map remains invertible from the concatenated Gaussian seed z to the final sample x.Source: §3 · pp. 3–4

Recursive design

  • MsIGN recursively increases dimension from d1 to dL=d, generating each finer-scale sample by upsampling and refining the previous scale.At scale l, PCl produces a prior-conditioned intermediate sample and Fl transforms it toward the scale-l posterior.Source: §3 · p. 3

Output constraints

  • Bijective output activations can enforce constraints such as image pixels lying in [0, 1] without breaking the seed-to-sample invertible relation.Source: §3 · p. 4

4. Training Strategy

MsIGN combines Jeffreys-divergence optimization with coarse-to-fine multi-stage training. Coarser models provide proposal distributions and initialization for finer stages, enabling posterior approximation without direct samples from the target.

Multi-stage training

  • Multi-stage training targets ql at each scale while training only flows through that scale and keeping prior conditioning layers fixed.The procedure begins at the coarsest scale and progressively incorporates finer-scale flows.Source: §4 · p. 4

Jeffreys divergence

  • Jeffreys divergence penalizes mode missing more strongly than one-sided KL divergence and can remove local minima associated with missing posterior modes.This addresses the optimization difficulty of generative posterior approximation in high-dimensional settings.Source: §4 · p. 4
  • Importance sampling estimates the Jeffreys divergence and its gradient using the prior-conditioned coarse-scale approximation as a proposal, without requiring samples from q.The Monte Carlo estimator also cancels the unknown normalizing constant Z.Source: §4 · p. 4

Optimization

  • Stochastic-gradient optimization uses backpropagation for ∂log pθ/∂θ, while the proposal ˜q comes from the coarser-scale model.Source: §4 · p. 4

Multi-stage training

  • At each finer stage, previously trained flows are retained and the new flow starts as the identity, because the coarser model and conditioning layer already provide a good proposal.The authors report that this strategy significantly stabilizes training and improves final performance.Source: §4 · p. 4

5. Related Work

MsIGN extends invertible generative modeling with a spatially interpretable multiscale structure and applies it to high-dimensional Bayesian inference. Its design targets mode-missing and optimization difficulties that become more severe as posterior dimension grows.

Invertible generative models

  • Unlike earlier invertible models, MsIGN trains different scales separately and uses a spatial multiscale structure intended to make intermediate neurons semantically interpretable.The Glow invertible block is used as a building component but can be replaced without algorithmic changes.Source: §5 · p. 4

Generative models

  • GANs use multiscale ideas, but their lack of invertibility makes them difficult to apply to Bayesian inference problems.Source: §5 · p. 5

Bayesian inference

  • High-dimensional Bayesian inference differs from image synthesis because posterior inference typically provides only an unnormalized density and seeks independent samples.Generative models can produce independent samples efficiently at inference time, but their non-convex optimization becomes harder as dimension increases.Source: §5 · p. 5
  • MsIGN addresses these difficulties by combining Jeffreys-divergence minimization with coarse-to-fine training, which the paper presents as having fewer local minima than KL-based optimization.Source: §5 · p. 5

MCMC and particle methods

  • MCMC methods can suffer from highly correlated samples and low sample efficiency in high dimensions, while the proposed multiscale strategy also benefits particle-based methods.The paper reports observing this benefit for amortized-SVGD.Source: §5 · p. 5

6. Experiment

Experiments evaluate MsIGN on two 4096-dimensional multimodal Bayesian inverse problems and natural-image synthesis, including ablations of its architecture and training strategy. Across these tasks, MsIGN captures modes accurately, approximates distributions well, and achieves strong image-model performance with interpretable latent and intermediate representations.

6.1.1. SYNTHETIC BAYESIAN INVERSE PROBLEMS

  • MsIGN achieves the best mode capture and superior distribution approximation on the synthetic 4096-dimensional Bayesian inverse problem.It is closest to ground truth along the critical direction and is especially accurate at finer scales.Source: Figure 3 · p. 6, §6.1.1 · p. 7

6.1.2. ELLIPTIC BAYESIAN INVERSE PROBLEMS

  • In the elliptic inverse problem, MsIGN and HMC capture both modes, while other methods fail, and MsIGN produces more balanced samples.The similarity between MsIGN and HMC cluster means suggests that both likely capture the correct modes.Source: Figure 4 · p. 6, §6.1.2 · p. 7

6.1.3. ABLATION STUDY

  • The prior conditioning layer is crucial because replacing it with stochastic nearest-neighbor upsampling or Glow prevents successful mode capture.These alternatives are evaluated in the architecture ablation.Source: Figure 5 · p. 6, §6.1.3 · p. 7
  • Single-stage training causes mode collapse, whereas multi-stage training benefits both KL- and Stein-discrepancy objectives.Jeffreys divergence yields more balanced samples, especially for the complicated elliptic problem; single-stage Jeffreys training is infeasible because reverse-KL estimation is difficult.Source: §6.1.3 · p. 7

6.2. Image Synthesis

  • MsIGN achieves superior bits-per-dimension values and uses fewer parameters than Glow and Residual Flow on image synthesis benchmarks.It uses 24.4% fewer parameters than Glow for CelebA 64 and 37.4% fewer than Residual Flow for ImageNet 64.Source: §6.2 · p. 7, Table 2 · p. 8
  • CelebA latent interpolations are semantically meaningful, while internal activations show progressively refined generation across four scales.The visualizations cover synthesized 64 × 64 images, latent interpolation, and checkpoints from low to high resolution.Source: Figure 6 · p. 8, §6.2 · p. 8, Figure 7 · p. 8

7. Conclusion

The paper proposes MsIGN and associated training algorithms for high-dimensional Bayesian inference with multiscale structure. It demonstrates potential up to dimension 4096 and reports superior benchmark performance for image synthesis, while identifying further applications as future work.

7. Conclusion

  • MsIGN approximates high-dimensional posteriors with multiscale generation and demonstrates potential on Bayesian inference problems up to dimension 4096.The conclusion leaves several important directions for future work.Source: §7 · p. 8
  • The network achieves superior performance over benchmarks in various image synthesis tasks.Proposed future applications include multiscale Bayesian deep learning and temporal data assimilation.Source: §7 · p. 8

A. Proof of Theorem 3.1

The proof establishes a closed-form prior conditioning layer for a Gaussian prior and shows that this layer is invertible between the coarse variable, fine-scale noise, and the full state.

A. Proof of Theorem 3.1

  • For a positive-definite Gaussian covariance, the conditional covariance is Σc = Σ − ΣA^T(AΣA^T)^−1AΣ.The proof derives this from the conditional Gaussian mean and covariance.Source: A. Proof of Theorem 3.1 · pp. 11–12
  • The state decomposes into a conditional mean component U_c x_c and a Gaussian residual Wz, yielding the prior conditioning layer.The residual covariance satisfies Σc = WW^T, and z is standard Gaussian.Source: A. Proof of Theorem 3.1 · pp. 11–12
  • The residual x − U_cAx is independent of Ax because their Gaussian cross-covariance is zero.This independence permits the conditional covariance calculation.Source: A. Proof of Theorem 3.1 · pp. 11–12
  • The conditioning layer is invertible: Ax recovers x_c, while the complementary transform recovers z.The proof explicitly verifies both inverse components.Source: A. Proof of Theorem 3.1 · p. 12
  • The construction is nonunique because any orthonormal rotation P of the residual coordinates gives another valid conditioning map.This preserves the induced conditional Gaussian distribution.Source: A. Proof of Theorem 3.1 · p. 12

B. Comparison of the Jeffreys divergence and Kullback-Leibler divergence

The two one-sided KL divergences can be trapped at local minima that miss modes, whereas the Jeffreys divergence is selected as a more robust objective for multimodal posterior learning.

  • DKL(p∥q) can guide training to local minima near (1.5, 1.5) or (−1.5, −1.5), where only one mode of q is captured.The example uses a Gaussian mixture with µ1 = −µ2 = 1.5 and σ = 0.25.Source: B. Comparison of the Jeffreys… · p. 12
  • DKL(p∥q) is small when p avoids regions where q has low density, so it does not strongly enforce capturing every high-density region of q.This zero-forcing behavior explains the one-mode local minima.Source: B. Comparison of the Jeffreys… · p. 12
  • DKL(q∥p) also risks a small value when p captures q’s modes but includes additional modes.This behavior is described as zero-avoiding.Source: B. Comparison of the Jeffreys… · p. 12
  • Figure 8 compares the landscapes of both KL directions and the Jeffreys divergence, marking global and other local minima.Source: Figure 8 · p. 13
  • The Jeffreys divergence combines the two directions and is therefore chosen as a robust learning objective for capturing multiple modes.Source: B. Comparison of the Jeffreys… · p. 13

C. Proof of Theorem 4.1

The theorem establishes Monte Carlo estimators for the Jeffreys divergence and its derivative without requiring the target posterior’s normalizing constant. The recursive multiscale construction defines progressively higher-dimensional posteriors and approximates each through prior-conditioned upsampling.

C. Proof of Theorem 4.1

  • The Monte Carlo estimators for the Jeffreys divergence and its derivative do not require samples from q or knowledge of its normalizing constant.The unknown normalizing constant cancels in the estimator, including when only an unnormalized target is available.Source: C. Proof of Theorem 4.1 · p. 13

D. The Recursive Multiscale Structure

E. More Discussion about Related Work

The related-work discussion distinguishes MsIGN’s deterministic, approximate, recursively corrected multiscale construction from prior approaches using random coarse variables, exact two-scale relations, or dimension-sensitive polynomial flows.

F.1. Experimental Setting of BIPs

The BIP experiments evaluate MsIGN on synthetic and elliptic inverse problems designed to expose multimodality, then test its multiscale training, ablations, and image-synthesis setup. Across the additional results, MsIGN captures modes more reliably than alternatives, while training after prior conditioning materially improves scale-specific posterior approximation.

F.1. Experimental Setting of BIPs

  • The BIP setup uses Gaussian priors and noise, finite-element PDE simulation for the elliptic case, and HMC samples to initialize low-dimensional multi-stage training.At the coarsest scale, the posterior is four-dimensional, making HMC a workable initialization strategy.Source: §F.1 · pp. 15–16
  • The elliptic benchmark uses a mirror-symmetric, double-modal posterior generated through an elliptic PDE forward map with local measurements.The discretized state has dimension 4096, and the measurement functions and forcing term are symmetric along the s2 direction.Source: §F.1 · p. 15

F.2.1. SYNTHETIC BAYESIAN INVERSE PROBLEM

  • In the synthetic problem, SVGD and A-SVGD collapse modes as dimension increases, HMC becomes imbalanced, and pSVGD is biased, whereas MsIGN remains effective.The intermediate-scale comparisons attribute these failures partly to aliasing and larger posterior log-density gaps at rough resolutions.Source: §F.2.1 · pp. 16–17, Figure 11 · p. 18
  • At the final synthetic scale, multi-stage training lowers the Jeffreys divergence from 252 to 56.8, showing improvement beyond merely refining resolution.The learning curve supports continued training after entering the final scale.Source: §F.2.1 · p. 17

F.2.2. ELLIPTIC BAYESIAN INVERSE PROBLEM

  • In the elliptic problem, only MsIGN and HMC detect all modes at intermediate scales, while other methods can become stuck between modes.HMC performs acceptably but can remain imbalanced, and its samples are not treated as a reference solution.Source: §F.2.2 · p. 17, Figure 11 · p. 18
  • Training after prior conditioning is necessary because the learned posterior differs from its surrogate initialization in the location, shape, and scale of features.Prior conditioning provides an initial guess, but each scale must learn additional posterior information.Source: §F.2.2 · p. 17, Figure 12 · p. 19

F.2.3. ABLATION STUDY OF BAYESIAN INVERSE PROBLEM

  • MsIGN outperforms its ablated variants on both Bayesian inverse problems, supporting the prior conditioning layer, network design, multistage training, and Jeffreys divergence.The mode-capturing table uses T, F, and I to distinguish successful, collapsed, and biased mode recovery.Source: §F.2.3 · p. 17, Table 5 · p. 19

G.1. Experimental Setting of Image Synthesis

  • For image synthesis, MsIGN is trained by maximum likelihood on downsampled image distributions, using average pooling for scale transitions and estimated scale-dependent conditioning parameters.The image experiments use an invertible-flow block and Adam optimization, with shorter training at non-final stages.Source: §G.1 · pp. 17, 21

G.2. Additional Results of Image Synthesis

MsIGN generates images from MNIST, CIFAR-10, and CelebA across resolutions up to 128 × 128. Its multiscale internal states progress from global structure at coarse scales to local detail at finer scales.

G.2. Additional Results of Image Synthesis

  • MsIGN synthesizes MNIST, CIFAR-10, and CelebA images at 28 × 28, 32 × 32, and 128 × 128 resolutions, respectively.The displayed samples use temperature 1.0 for MNIST and CIFAR-10, and 0.8 for CelebA.Source: §G.2 · p. 21, Figure 15 · p. 22, Figure 16 · p. 22
  • The 4-scale CelebA model records 16 checkpoints per image to visualize internal neurons during reconstruction and synthesis.Snapshots are taken four times at each scale along the invertible flow.Source: §G.2 · p. 21, Figure 18 · p. 23
  • MsIGN forms global image features at early scales and adds increasingly local details at higher scales.This interpretation applies to the observed internal neurons during both image recovery and new-image synthesis.Source: §G.2 · p. 21, Figure 18 · p. 23
Loading 2105.05489v1…