PaperlayerView exact revision on arXiv

Source-linked AI summary

Cooperative Training of Descriptor and Generator Networks

Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, Ying Nian Wu

arXiv:1609.09408v3stat.MLcs.CV

TL;DR

The paper addresses how to train energy-based descriptor and latent-variable generator models jointly when their maximum-likelihood procedures require costly MCMC. It interweaves their learning so the generator initializes descriptor sampling and then learns the descriptor’s revisions through MCMC teaching, producing highly realistic generative models in the reported experiments.

  • Problem

    The two models’ maximum-likelihood learning algorithms require MCMC, whose Langevin steps may converge slowly and become bottlenecks; separately learning highly varied images is challenging.

  • Method

    CoopNets interweaves descriptor and generator training: generator samples initialize descriptor Langevin revision, and the generator reconstructs the descriptor’s revised synthesized examples.

  • Results

    The cooperative algorithm learns highly realistic generative models, with experiments evaluating generated-image synthesis quality and reconstructed-image recovery errors.

  • Takeaways & Limitations

    The descriptor and generator communicate through synthesized data, connecting MCMC sampling with direct ancestral sampling in a jointly trained system.

  • Takeaways & Limitations

    The paper leaves analysis of the finite-capacity situation for future investigation and relies on empirical evaluations.

Abstract

from arXiv · show

This paper studies the cooperative training of two generative models for image modeling and synthesis. Both models are parametrized by convolutional neural networks (ConvNets). The first model is a deep energy-based model, whose energy function is defined by a bottom-up ConvNet, which maps the observed image to the energy. We call it the descriptor network. The second model is a generator network, which is a non-linear version of factor analysis. It is defined by a top-down ConvNet, which maps the latent factors to the observed image. The maximum likelihood learning algorithms of both models involve MCMC sampling such as Langevin dynamics. We observe that the two learning algorithms can be seamlessly interwoven into a cooperative learning algorithm that can train both models simultaneously. Specifically, within each iteration of the cooperative learning algorithm, the generator model generates initial synthesized examples to initialize a finite-step MCMC that samples and trains the energy-based descriptor model. After that, the generator model learns from how the MCMC changes its synthesized examples. That is, the descriptor model teaches the generator model by MCMC, so that the generator model accumulates the MCMC transitions and reproduces them by direct ancestral sampling. We call this scheme MCMC teaching. We show that the cooperative algorithm can learn highly realistic generative models.

1 INTRODUCTION

The paper proposes cooperative training for an energy-based descriptor network and a latent-variable generator network. The generator initializes the descriptor’s Langevin sampling, while the descriptor’s revisions teach the generator through MCMC teaching.

  • 1 INTRODUCTION: The descriptor network maps images to energy through a bottom-up ConvNet, whereas the generator maps latent factors to images through a top-down ConvNet.The descriptor is an energy-based model; the generator is a latent-variable or directed graphical model.
  • 1 INTRODUCTION: Both models’ maximum-likelihood algorithms require MCMC, whose Langevin revision and inference steps may converge slowly and become bottlenecks.Algorithm D revises synthesized images, while Algorithm G infers latent factors for observed images.
  • 1 INTRODUCTION: CoopNets jointly trains bottom-up energy-based descriptor and top-down latent-variable generator networks for image modeling and synthesis.The approach connects undirected and directed models, as well as MCMC and ancestral sampling.
  • 1 INTRODUCTION: The generator supplies initial synthesized examples to the descriptor’s Langevin revision, and the descriptor updates from observed-versus-synthesized differences.This modifies contrastive divergence by initializing finite-step MCMC from generator samples rather than observed examples.
  • 1 INTRODUCTION: Through MCMC teaching, the generator reconstructs the descriptor-revised samples and learns to absorb the descriptor’s MCMC transitions by direct ancestral sampling.The generator therefore learns from revised synthesized data rather than only from observed examples.

2 TWO MODELS AND TWO ALGORITHMS

The paper develops two ConvNet-based generative models: a bottom-up energy-based descriptor and a top-down latent-variable generator. Their maximum-likelihood algorithms use MCMC, with Langevin dynamics approximating intractable expectations for both models.

  • 2.1 Energy-based model and maximum likelihood learning: The descriptor network defines an energy-based distribution by exponentially tilting a reference distribution with a bottom-up ConvNet.Its energy combines a quadratic reference-distribution term with the negative ConvNet output.
  • 2.1 Energy-based model and maximum likelihood learning: Maximum-likelihood learning of the descriptor network uses finite-step Langevin dynamics to approximate an analytically intractable expectation.The dynamics lowers energy through a gradient term while Gaussian noise increases entropy.
  • 2.1 Energy-based model and maximum likelihood learning: The descriptor algorithm alternates Langevin revision of synthesized examples with density shifting that updates parameters toward observed examples.Back-propagation supports both image-gradient and parameter-gradient computations, whose chain-rule steps largely overlap.
  • 2.1 Energy-based model and maximum likelihood learning: The descriptor algorithm becomes persistent contrastive divergence when synthesized examples are initialized from observed examples, and its zero-temperature limit becomes gradient descent toward energy minima.The zero-temperature formulation also yields a minimax interpretation in which revision decreases and density shifting increases the value function.
  • 2.2 Latent variable model and maximum likelihood learning: The generator models images as noisy transformations of lower-dimensional latent factors through a top-down ConvNet and supports direct ancestral sampling.Sampling first draws latent factors from a known Gaussian prior, then maps them to images through the generator.
  • 2.2 Latent variable model and maximum likelihood learning: Maximum-likelihood learning of the generator requires MCMC inference because its latent-variable expectation and image marginal are analytically intractable.Langevin inference samples from the posterior and solves a nonlinear least-squares reconstruction problem with an ℓ2 penalty.

3 COOPERATIVE TRAINING

CoopNets interweaves generator sampling, Langevin revision, descriptor learning, and generator reconstruction so each model teaches the other through MCMC transitions. The framework has theoretical convergence properties under idealized assumptions, while finite-capacity analysis is left to empirical evaluation.

  • 3.1 Cooperation of two algorithms: CoopNets lets the generator initialize finite-step Langevin revision for the descriptor, then trains the generator to reproduce the revised examples through MCMC teaching.The generator supplies initial synthesized examples, the descriptor revises them, and the generator learns from paired latent factors and revised outputs.
  • 3.1 Cooperation of two algorithms: The cooperative iteration alternates initial generation, Langevin revision, optional Langevin inference, descriptor density shifting, and generator reconstruction.The algorithm accepts training examples, Langevin step counts, and learning iterations, and returns descriptor and generator parameters with synthesized examples.
  • 3.2 Theoretical understanding: The authors report that disabling Langevin inference works well, one-step Langevin revision is efficient and works well, and variational learning performs as well as maximum-likelihood learning empirically.These are reported special-case and experimental observations rather than general theoretical guarantees.
  • 3.1 Cooperation of two algorithms: The descriptor transforms initial samples S1 into low-energy revised samples S2, while the generator reconstructs S2 as S3 and uses their closeness to assess learning.This interaction is described as associative-memory feedback from the descriptor to the generator.
  • 3.2 Theoretical understanding: MCMC teaching alternates Markov transition from qα(t) to p(t+1) with projection to qα(t+1), allowing the generator to accumulate past transitions and rejuvenate chains by direct sampling.The generator thereby represents the evolving MCMC marginal distribution and can implicitly power many parallel chains.
  • 3.2 Theoretical understanding: At an idealized fixed point with infinite generator capacity, qα equals the descriptor's stationary distribution pθ, the second KL divergence vanishes, and θ becomes a maximum-likelihood estimate.The result assumes convergence to local minima and infinite generator capacity; finite-capacity analysis is left for future work.

4 EXPERIMENTS

Experiments apply CoopNets to textures, objects, scenes, handwritten digits, large-scale image benchmarks, inpainting, and dynamic textures. Across these settings, the method produces realistic samples and outperforms or approaches baseline models on reported evaluations.

  • Large-scale benchmark evaluation: CoopNets achieves higher softmax probabilities, lower top-5 classification errors, and higher pairwise structural similarity than DCGAN, VAE, and separate training.The results are averaged over five object categories and vary with the number of training examples.
  • Handwritten digits: Both CoopNets networks outperform DBN, Stacked CAE, Deep GSN, and GAN baselines on MNIST Parzen-window log-likelihood estimates.The descriptor improves the generator during cooperative training, and the generator eventually approaches the descriptor’s performance.
  • Large-scale benchmark evaluation: Both the generator and descriptor outperform DCGAN, W-GAN, and VAE on FID across LSUN bedrooms, CelebA, and Cifar-10.The generator is effective standalone, while Langevin revision further improves the descriptor’s synthesis results.
  • Large-scale benchmark evaluation: Human subjects rank CoopNets as generating more realistic images than the other baseline methods on LSUN bedrooms, CelebA, and Cifar-10.The comparison uses randomized model order and 16 synthesized images per model for each subject.
  • Dynamic textures: CoopNets synthesizes realistic dynamic textures, while evaluation compares its PSNR and SSIM with established dynamic-texture models.The spatial-temporal model learns from 64-frame videos and uses the same parameters across all categories without tuning.

5 CONCLUSION

The conclusion presents cooperative training as a way to connect descriptor and generator models through mutual exchange of synthesized data and interwoven MCMC learning. It highlights MCMC teaching, extensions to conditional generation, and a fast-policy interpretation of the learned generator.

  • 5 CONCLUSION: Cooperative training lets descriptor and generator networks exchange synthesized data while learning jointly.The generator provides initial samples, the descriptor revises them, and the generator learns from the revised samples.
  • 5 CONCLUSION: MCMC teaching makes the generator accumulate the descriptor’s MCMC transitions and reproduce them through direct ancestral sampling.This distills the descriptor’s MCMC algorithm into the generator and turns generator learning into supervised learning in the authors’ interpretation.
  • 5 CONCLUSION: The cooperative method generalizes to conditional generation for outputs given labels, text descriptions, sketches, or images.The proposed conditional version uses conditional descriptor and generator networks for high-dimensional conditional distributions.
  • 5 CONCLUSION: The authors relate the descriptor to a cost function, MCMC to optimal control, and the generator to a learned policy that enables faster thinking.They describe this connection as future work rather than a completed analysis.
Loading 1609.09408v3…