PaperlayerView exact revision on arXiv

Source-linked AI summary

A-NICE-MC: Adversarial Training for MCMC

Jiaming Song, Shengjia Zhao, Stefano Ermon

arXiv:1706.07561v3stat.MLcs.AIcs.LG

TL;DR

MCMC proposals can be slow to converge when they are general-purpose, while domain-specific proposals may require expert design. A-NICE-MC adversarially trains flexible MCMC kernels, using volume-preserving flows and bootstrap training for target distributions. The paper reports improved performance over traditional methods, including HMC, across varied domains.

  • Problem

    MCMC methods may rely on general-purpose proposals that converge slowly or on problem-specific proposals hand-crafted by an expert.

    Source: §1 · p. 1

  • Method

    A-NICE-MC adversarially trains flexible MCMC kernels, using volume-preserving flows and a bootstrap approach to target empirical or analytic distributions.

    Source: §1 · p. 1, §3 · p. 3, §4.4 · p. 6

  • Results

    A-NICE-MC significantly outperforms traditional methods such as Hamiltonian Monte Carlo in various domains; on four synthetic energy functions, HMC performed well only on ring while A-NICE-MC performed well in all cases.

    Source: §5 · pp. 7–8, §6 · p. 9

  • Takeaways & Limitations

    The resulting chains can target empirical and analytic distributions with domain-specific kernels that combine MCMC guarantees and neural-network expressiveness.

    Source: §6 · p. 9

    For sample-based training, optimization is difficult because a suitable chain length is unknown in advance and gradients require backpropagation through the entire chain.

    Source: §3 · p. 2

Abstract

from arXiv · show

Existing Markov Chain Monte Carlo (MCMC) methods are either based on general-purpose and domain-agnostic schemes which can lead to slow convergence, or hand-crafting of problem-specific proposals by an expert. We propose A-NICE-MC, a novel method to train flexible parametric Markov chain kernels to produce samples with desired properties. First, we propose an efficient likelihood-free adversarial training method to train a Markov chain and mimic a given data distribution. Then, we leverage flexible volume preserving flows to obtain parametric kernels for MCMC. Using a bootstrap approach, we show how to train efficient Markov chains to sample from a prescribed posterior distribution by iteratively improving the quality of both the model and the samples. A-NICE-MC provides the first framework to automatically design efficient domain-specific MCMC proposals. Empirical results demonstrate that A-NICE-MC combines the strong guarantees of MCMC with the expressiveness of deep neural networks, and is able to significantly outperform competing methods such as Hamiltonian Monte Carlo.

1 Introduction

MCMC has not benefited as much as variational inference from deep learning, in part because its iterative nature makes expressive function approximators and practical optimization difficult. A-NICE-MC trains flexible MCMC kernels adversarially and combines neural-network expressiveness with MCMC guarantees.

1 Introduction

  • MCMC has gained less from deep learning than variational inference because its iterative methods do not naturally accommodate expressive function approximators, and its performance metrics and optimization objectives are difficult to handle.The cited passages describe evaluating MCMC and optimizing across a large parameter space as challenging.Source: §1 · p. 1
  • A-NICE-MC trains flexible, potentially neural-network-parameterized MCMC kernels by treating the Markov chain as an implicit generative model and matching its samples to a target distribution adversarially.The approach is presented for both empirical data distributions and analytically specified targets such as Bayesian posteriors.Source: §1 · p. 1
  • A-NICE-MC uses flexible volume-preserving flows and a bootstrap approach to design domain-specific proposals, with training intended to reduce autocorrelation and increase effective sample size.The paper reports that trained operators significantly outperform traditional methods such as Hamiltonian Monte Carlo in various domains.Source: §1 · pp. 1–2

2 Notations and Problem Setup

The setup treats a parameterized transition kernel as a stochastic generative model and seeks a chain whose stationary distribution approximates a target while producing weakly correlated samples. Targets may be specified by samples or by an analytic expression, including one known only up to a normalization constant.

2 Notations and Problem Setup

  • The transition kernel Tθ is time-homogeneous and parameterized by θ; under irreducibility and positive recurrence, it has a unique stationary distribution, with those conditions assumed for all θ.The kernel is defined through an implicit generative model driven by an auxiliary random variable.Source: §2 · p. 2
  • The objective is for the stationary distribution πθ to be close to the target pd while samples from one chain have low autocorrelation.The setup gives targets such as data distributions and intractable Bayesian posteriors.Source: §2 · p. 2
  • The chain is viewed as a stochastic generative model for efficiently producing samples with target-specified characteristics and enabling Monte Carlo estimates.The target may be provided as typical samples or through an analytic expression, possibly only up to a normalization constant.Source: §2 · p. 2

3 Adversarial Training for Markov Chains

The section introduces Markov GAN (MGAN), a likelihood-free approach that trains a Markov chain using samples from the chain and from the target data distribution. Image experiments show chains moving between data modes, while also showing that model architecture can affect transition speed and autocorrelation.

3 Adversarial Training for Markov Chains

  • A direct adversarial approach using a chain run long enough to approximate stationarity is difficult because the suitable run length is unknown and gradients require backpropagation through the whole chain.MGAN is proposed as a more efficient approximation.Source: §3 · pp. 2–3
  • MGAN trains a chain using samples after b transitions from its initial distribution and after m transitions from data samples, rather than backpropagating through a full chain of unknown length.The method combines these two sample types in adversarial training; the first condition encourages convergence toward the target over relatively short runs, while the second enforces the target as a fixed point.Source: §3 · p. 3

3.1 Example: Generative Model for Images

  • The image-chain kernel uses an autoencoding structure that adds Gaussian auxiliary noise to the latent code before decoding the next state.The noise scale β is set to 0.1, and transition probabilities are generally intractable because evaluating them requires integrating over the auxiliary variable.Source: §3.1 · p. 3
  • In MNIST samples, consecutive states can move between modes, including changes in digit identity, rather than generating a single sample ancestrally.The learned class probabilities are fairly uniform, ranging from 0.09 to 0.11.Source: Figure 1 · p. 3, §3.1 · p. 4
  • Shortcut connections can increase the time needed to transition between CelebA modes, raising autocorrelation that can hinder Monte Carlo estimates.The authors suggest that an information bottleneck may encourage transitions by making the chain more likely to forget its previous state.Source: Figure 3 · p. 3, §3.1 · p. 4

4 Adversarial Training for Markov Chain Monte Carlo

A-NICE-MC trains neural-network MCMC proposals for targets specified by an analytic density, where direct target samples are unavailable. It combines Metropolis–Hastings correction with volume-preserving flows, bootstrap training, and a pairwise discriminator to improve sampling proposals and reduce autocorrelation.

4.1 Exact Sampling Through MCMC

  • Metropolis–Hastings acceptance enforces detailed balance so the chain’s stationary distribution is the target, even when its normalization is intractable.Directly optimizing the proposal is difficult because the acceptance objective is nondifferentiable; REINFORCE led to extremely low acceptance rates in the experiments.Source: §4 · p. 4, §4.1 · p. 4

4.2 Hamiltonian Monte Carlo and Volume Preserving Flow

  • Volume-preserving flows provide neural-network proposals with the symmetric behavior needed to improve acceptance relative to the REINFORCE setting.The flow uses invertible transformations; its Jacobian determinant is one.Source: §4.2 · p. 5

4.3 A NICE Proposal

  • The NICE proposal randomly chooses the forward transformation or its inverse, then uses Metropolis–Hastings acceptance and resamples the auxiliary variable after each step.The proposal satisfies the required symmetry for any model parameters, while resampling the auxiliary variable helps prevent random-walk behavior.Source: §4.3 · pp. 5–6

4.4 Training A NICE Proposal

  • Training the proposal targets high-probability states under the joint target p(x, v) = pd(x)p(v), without requiring score-function gradient estimators.The method applies the MGAN objective to pd(x) and matches the generated auxiliary-variable distribution to its prior.Source: §4.4 · p. 6

4.5 Bootstrap

  • A bootstrap process iteratively trains the model on chain samples and uses the trained transition operator to generate the next training samples.It begins with samples from a potentially slow-mixing operator whose stationary distribution is the target; successive samples should improve in quality.Source: §4.5 · p. 6

4.6 Reducing Autocorrelation by Pairwise Discriminator

  • The pairwise discriminator scores two samples together; correlated fake pairs make autocorrelation detectable against independent real pairs.The discriminator is intended to encourage samples with little autocorrelation, improving effective sample size.Source: §4.6 · p. 7
  • Pairwise discrimination significantly reduces autocorrelation in samples from a shortcut-connected model, as shown by their greater variety.The discriminator compares pairs, using independent bootstrapped samples as real data and chain-related pairs as fake data.Source: §4.6 · p. 7, Figure 5 · p. 7

5 Experiments

Across four synthetic 2D targets, A-NICE-MC performs well while HMC performs well only on the single-connected ring and struggles when modes are distant. On Bayesian logistic regression, HMC has higher ESS, but A-NICE-MC's cheaper proposals yield higher ESS per second; training also improves ESS over iterations and bootstrap rounds.

5 Experiments

  • HMC performs well only on ring, while A-NICE-MC performs well across ring, mog2, mog6, and ring5, where flexible neural proposals move between distant modes.For ring5, HMC fails to estimate the statistics from a Gaussian-centered initialization, and its R hat is 1.26 versus 1.002 for A-NICE-MC.Source: §5 · pp. 7–8
  • ESS increases in all tested cases with more training iterations and bootstrap rounds, indicating reduced autocorrelation.Figure 8 plots ESS against training iterations.Source: §5 · p. 8, Figure 8 · p. 9
  • In ring5, training reaches ESS 121.54 in approximately 550 seconds, after which A-NICE-MC catches up with HMC in under 2 seconds of sampling.The reported training time uses 2500 iterations on one CPU thread and includes bootstrap; training adds initial computational cost.Source: §5 · p. 8
  • On Bayesian logistic regression, HMC has higher ESS, but A-NICE-MC's proposal costs almost an order of magnitude less and yields higher ESS per second.The comparison uses german, heart, and australian datasets; the supplied passages do not give Table 2's numeric values.Source: §5 · pp. 8–9, Table 2 · p. 9

6 Discussion

The discussion presents A-NICE-MC as a likelihood-free way to train parametric MCMC operators with good mixing for empirical or analytic targets. It identifies high-dimensional inference and data-adaptive proposals as future directions, while noting that untrained proposals may struggle to find low-energy regions.

6 Discussion

  • The paper presents a likelihood-free method for training parametric MCMC operators with good mixing, targeting empirical and analytic distributions through domain-specific transition kernels.The kernels use flexible neural networks and volume-preserving flow models.Source: §6 · p. 9
  • The authors identify high-dimensional inference, including Bayesian neural networks, as potential applications, but say initial samples may need to come from another method such as variational inference.They link this boundary to the reduced chance that untrained proposals stumble on low-energy regions under the curse of dimensionality.Source: §6 · p. 9
  • Future directions include bypassing bootstrap by training directly on U(x) and developing proposals that adapt rapidly to data, potentially through meta-learning.A cited use case is inferring a data point's latent variable in a variational autoencoder.Source: §6 · p. 9

A Estimating Effective Sample Size

The appendix defines effective sample size (ESS) by matching the variance of an MCMC sample mean to that of independent samples, and gives an autocorrelation-based estimator. It also motivates the Equation 3 objective through conditions on convergence to and stationarity at the target distribution.

A Estimating Effective Sample Size

  • ESS is the number of independent target-distribution samples needed to match the variance of the mean estimated from correlated MCMC samples.The practical estimator uses autocorrelations and truncates their sum when autocorrelation falls below 0.05.Source: A Estimating Effective Sample Size · p. 12

B Justifications for Objective in Equation 3

  • The Equation 3 objective is motivated by two conditions for the target distribution: short-run convergence toward it and its being a fixed point of the transition operator.The text connects total-variation convergence with Jensen–Shannon divergence and GAN training.Source: B Justifications for Objective in… · pp. 12–13
  • Under the stated assumptions, the sequence of ergodic Markov-chain stationary distributions converges to the target in total variation.The appendix states this conclusion in Proposition 1 and repeats it at the end of its proof.Source: B Justifications for Objective in… · p. 12

C Proof of Theorem 1

The appendices justify the Metropolis–Hastings ratio for a volume-preserving proposal, describe pairwise-discriminator training samples, and provide image-model architecture and energy-function details.

C Proof of Theorem 1

  • For a volume-preserving transformation, the proof establishes the proposal-density relation used to apply the Metropolis–Hastings ratio p(x′, v′)/p(x, v).The Jacobian determinant is one.Source: C Proof of Theorem 1 · p. 13

D Details on the Pairwise Discriminator

  • Pairwise-discriminator training uses real-to-chain pairs (x, z1) and generated chain pairs (z2, z3), with equal weights; the figure notes blocking the gradient for z2 to parallelize and speed training.The chains run for B steps to obtain z1 and z2, then for M additional steps to obtain z3.Source: D Details on the Pairwise Discriminator · p. 13, Figure 9 · p. 13

E.1 Architectures for Generative Model for Images

  • The image-model appendix specifies layer notation and architectures, including residual and highway connections for one CelebA model and doubled convolutional filters for the pairwise discriminator.The highway output blends the input image and decoder output using a sigmoid-activated transposed-convolution layer.Source: §E.1 · p. 14

E.2 Analytic Forms of Energy Functions

  • The appendix lists analytic energy-function forms for ring, mog2, mog6, and ring5 without further substantive results in the supplied passages.Source: §E.2 · pp. 14–15

E.3 Benchmarking Running Time

The authors benchmark HMC and A-NICE-MC in TensorFlow and examine how batch size and computation-graph optimization affect wall-clock measurements. They choose batch sizes near the runtime threshold to avoid substantial efficiency underestimates.

E.3 Benchmarking Running Time

  • The HMC–A-NICE-MC comparison uses TensorFlow to reduce differences caused by machine type, language, and low-level optimizations.Source: §E.3 · p. 15
  • Runtime is relatively linear in the number of steps but not exactly linear in batch size, partly because TensorFlow optimizes computation graphs.With a fixed number of steps, runtime is constant below a batch-size threshold and then increases approximately linearly.Source: §E.3 · p. 15
  • Benchmarks use optimized graphs with batch sizes set before execution: 2000 for energy functions and 64 for Bayesian logistic regression.Source: §E.3 · p. 15

E.4 Hyperparameters for the Energy Function Experiments

The appendices specify training and evaluation settings for energy-function and Bayesian logistic-regression experiments, describe the A-NICE-MC architecture, and provide extended image samples.

E.4 Hyperparameters for the Energy Function Experiments

  • Both methods start from x0 ∼ N(0, I), run 1000 burn-in steps, and evaluate the following 1000 samples; A-NICE-MC uses three coupling layers and bootstrapped training.For the energy-function experiments, HMC uses 40 leapfrog steps with step size 0.1; bootstrapping replaces half the samples every 500 iterations, and models train for 20000 iterations.Source: §E.4 · p. 15

E.5 Hyperparameters for the Bayesian Logistic Regression Experiments

  • For Bayesian logistic regression, HMC step sizes are tuned per dataset, while A-NICE-MC uses three coupling layers and 50-dimensional v.The models train for 20000 iterations with B = 16, M = 2, batch size 32, and learning rate 5 × 10^-4.Source: §E.5 · p. 15

E.6 Architecture Details

  • The architecture diagrams show NICE configurations for energy functions and Bayesian logistic regression; normalization is omitted because it slows execution without much ESS improvement.Source: §E.6 · p. 16

F Extended Images

F.1 Extended Images for Figure 1

  • Figure 11 shows MNIST samples π1 through π50, with each row representing samples from one chain.Source: Figure 11 · p. 17

F.2 Extended Images for Figure 3

The extended image figures show CelebA samples from chains under different model configurations, including shortcut connections and the presence or absence of a pairwise discriminator.

F.2 Extended Images for Figure 3

  • Figure 12 shows CelebA samples π1 through π50, with each row representing one chain; Figure 13 shows samples from a model with shortcut connections.Source: Figure 12 · p. 18, Figure 13 · p. 18

F.3 Extended Images for Figure 5

Related papers

Loading 1706.07561v3…