Source-linked AI summary
A Theory of Generative ConvNet
Jianwen Xie, Yang Lu, Song-Chun Zhu, Ying Nian Wu
TL;DR
The paper addresses whether discriminative ConvNets can also provide generative and unsupervised learning models. It derives a generative ConvNet by combining a classification-based exponential tilt with ReLU and Gaussian white-noise assumptions, obtaining a piecewise Gaussian model with auto-encoding means. The model links sampling and contrastive-divergence learning to reconstruction and can synthesize realistic natural image patterns.
Problem
The paper asks whether successful discriminative ConvNets can be turned into generative and unsupervised learning machines, especially for small or sparsely labeled datasets.
Method
The paper derives a generative random-field ConvNet from multi-category classification by using one base category with a reference distribution, then assumes ReLU nonlinearity and Gaussian white noise.
Results
The resulting model is piecewise Gaussian with auto-encoding means, and its Langevin sampling and contrastive-divergence learning are driven by reconstruction; maximum likelihood synthesizes realistic natural image patterns.
Takeaways & Limitations
Generative ConvNet provides a unified framework in which discriminative filters serve as bottom-up encoders and top-down basis functions for generative reconstruction.
Takeaways & Limitations
The auto-encoding representation depends on Gaussian white noise, while the piecewise Gaussian form and exact reconstruction-gradient equivalence depend on ReLU piecewise linearity.
Abstract
from arXiv · showhide
We show that a generative random field model, which we call generative ConvNet, can be derived from the commonly used discriminative ConvNet, by assuming a ConvNet for multi-category classification and assuming one of the categories is a base category generated by a reference distribution. If we further assume that the non-linearity in the ConvNet is Rectified Linear Unit (ReLU) and the reference distribution is Gaussian white noise, then we obtain a generative ConvNet model that is unique among energy-based models: The model is piecewise Gaussian, and the means of the Gaussian pieces are defined by an auto-encoder, where the filters in the bottom-up encoding become the basis functions in the top-down decoding, and the binary activation variables detected by the filters in the bottom-up convolution process become the coefficients of the basis functions in the top-down deconvolution process. The Langevin dynamics for sampling the generative ConvNet is driven by the reconstruction error of this auto-encoder. The contrastive divergence learning of the generative ConvNet reconstructs the training images by the auto-encoder. The maximum likelihood learning algorithm can synthesize realistic natural image patterns.
1. Introduction
The paper asks whether discriminative ConvNets can become generative, unsupervised models, and derives a generative ConvNet from classification with a reference-distribution base category. With ReLU and Gaussian white noise, the model becomes piecewise Gaussian with auto-encoding means and filter-based decoding.
- Generative ConvNet is derived from a multi-category discriminative ConvNet by treating one category as a base category generated by a reference distribution.
- ReLU and Gaussian white noise yield a piecewise Gaussian energy-based model whose Gaussian means are defined by an auto-encoder.
- Bottom-up filters become top-down basis functions, while binary activations become coefficients in top-down deconvolution.
- Gaussian white noise supplies the featureless reference and the ℓ2 energy term needed for the auto-encoding representation.
- ReLU supplies binary activation variables and piecewise linearity, enabling the piecewise Gaussian form and exact correspondence with auto-encoder reconstruction gradients.
2. Related Work
Related energy-based and auto-encoding models motivate the generative ConvNet, but the paper distinguishes its explicit Gaussian reference and internal auto-encoding structure. It also positions the model as directly derived from discriminative ConvNets rather than explicit-latent-variable hierarchies.
- Earlier exponential-tilting work proposed the model form but did not study its internal representational structure, while FRAME learning used pre-learned ConvNet filters.
- Generative ConvNet can be viewed as a hierarchical FRAME, Product of Experts, or Field of Experts model, but adds an explicit Gaussian white-noise reference.
- The explicit Gaussian reference distinguishes generative ConvNet by supporting an internal auto-encoding representation in which filters act as basis functions.
- Score-matching connections between latent-variable energy models and auto-encoders require analytically computable free energy, which generally fails for deep multi-layer latent-variable models.
- Unlike hierarchical Boltzmann-machine models with explicit binary latent variables, generative ConvNet is directly derived from the discriminative ConvNet.
3. Generative ConvNet
The generative ConvNet defines image distributions by exponentially tilting Gaussian white noise with scores from a hierarchical convolutional ConvNet. A proposition establishes that discriminative and generative ConvNets can be derived from each other, including an unlabeled-learning setting.
- A ConvNet recursively composes locally supported linear filters, nonlinear transformations, and optional subsampling across layers.
- The model uses ReLU nonlinearity, h(r)=max(r,0), in the modern ConvNet architecture.
- The discriminative ConvNet is a multinomial logistic regression defining class probabilities from category-specific scores, weights, and biases.
- The generative ConvNet exponentially tilts a reference distribution q(I), assumed to be Gaussian white noise, using a ConvNet scoring function and a normalizing constant.
- Generative and discriminative ConvNets can be derived from each other by choosing category priors and, in one direction, designating a base category generated by q(I).
- For unlabeled data, the exponential-tilting form can learn filters at a selected convolutional layer, and the convolutional model unfolds the prototype horizontally and hierarchically.
4. A Prototype Model
The prototype model exposes the generative ConvNet’s internal structure: ReLU activation patterns partition image space into Gaussian pieces whose means reconstruct images from activated filters. Langevin synthesis is therefore driven by auto-encoder reconstruction.
- The prototype learns filters or basis functions from small image patches, with each filter response given by an image-filter inner product.
- ReLU activation patterns divide image space into 2^K regions according to binary variables determined by filter responses and biases.
- Within each activation-pattern region, the model has a Gaussian density whose mean reconstructs images through the activated filters.
- Langevin dynamics samples the model using an auto-encoding reconstruction error based on the binary activation variables, linking synthesis to reconstruction.
- At a local energy minimum, bottom-up encoding infers binary activations and top-down decoding reconstructs the image as a weighted sum of filters.
5. Internal Structure of Generative ConvNet
The generative ConvNet has a piecewise Gaussian structure whose means are auto-encoding reconstructions. Bottom-up activation patterns determine top-down basis-function decoding, while Langevin sampling follows reconstruction error.
- Hierarchical construction: The generative model is obtained by horizontally unfolding filters into convolutions and vertically unfolding them into a multilayer hierarchy.The prototype model’s internal structure extends accordingly to the generative ConvNet.
- Piecewise Gaussian structure: ReLU divides image space into 2^K regions defined by binary activation patterns, making the model piecewise Gaussian under Gaussian white noise.Each region corresponds to a fixed pattern of filter activations.
- Piecewise Gaussian structure: On each activation region, the density is a truncated Gaussian with mean Bw,δ, which approximates reconstruction of images in that region.The scoring function is linear within a fixed activation pattern, and combining it with the Gaussian reference energy yields a quadratic energy.
- Auto-encoding representation: Bottom-up convolution computes binary activation variables, while top-down deconvolution combines filters as basis functions to construct Bw,δ.The resulting auto-encoding path is I → δ → Bw,δ.
- Sampling and modes: Langevin sampling is driven by the reconstruction error I − Bw,δ(I;w), and its deterministic component converges toward local energy minima.These local minima correspond to auto-encoding modes under the stated conditions.
- Sampling and modes: A local maximum of the model density is exactly auto-encoded through bottom-up activation inference followed by top-down reconstruction.The proposition generalizes to nonlinear h(), although the piecewise-Gaussian theorem requires piecewise linear nonlinearities such as ReLU.
6. Learning Generative ConvNet
The paper analyzes maximum-likelihood and contrastive-divergence learning for generative ConvNet. One-step contrastive divergence becomes reconstruction learning when observed and synthesized images retain the same activation pattern.
- Learning procedure: Maximum-likelihood learning estimates parameters from training images, with expectations approximated by Monte Carlo samples from Langevin dynamics.The paper summarizes this procedure in its learning and sampling algorithm.
- Contrastive divergence: Contrastive divergence can start Langevin dynamics from observed images and thereby tends to learn the generative ConvNet’s auto-encoder.This initialization is proposed for learning from big data.
- Contrastive divergence: When synthesized images are near the Gaussian-piece mean and that mean is a local mode, contrastive divergence tends to reconstruct observed images by that mode.The associated gradient is driven by the difference between the observation and its auto-encoding reconstruction.
- Contrastive divergence: If one-step Langevin dynamics preserves the activation pattern, its expected contrastive-divergence gradient is proportional to the reconstruction gradient.Bias updates match activation-pattern statistics and help preserve the shared pattern condition.
- Connection to score matching: The paper connects this reconstruction result to score matching, with the piecewise-linear ConvNet removing complicated second-derivative terms.This is presented as a sharpened specialization of the one-step Langevin connection.
- Empirical illustrations: Figures compare training images with generated object or texture patterns, and compare training images with corresponding one-step reconstructed images.The object figure uses four training and four generated images per category; the texture figure uses one training image and two generated images.
7. Synthesis and Reconstruction
The experiments show that generative ConvNets can learn and generate realistic natural image patterns, while contrastive divergence reconstructs observed images. The experiments use models learned from scratch across texture and object-pattern settings.
- Generative ConvNets learn and generate realistic natural image patterns, providing an empirical proof of the model’s generative capacity.The paper also reports that contrastive divergence reconstructs observed images, empirically validating Proposition 3.
- The experiments learn generative ConvNets from scratch rather than relying on pre-learned filters from existing ConvNets.
- Experiment 1: Generating texture patterns: For texture-pattern experiments, each category can be modeled from a single training image, with generated images compared against the training image.
- Experiment 2: Generating object patterns: For aligned object patterns, a four-layer model uses a fully connected top layer with a single filter covering the whole image.The experiments use around 10 Internet-collected training images per category and display generated images alongside training images.
- Synthesis and Reconstruction: A small contrastive-divergence experiment uses 10 Internet-collected training images, one Langevin iteration, and 1200 learning iterations.The ConvNet structure matches experiment 1, and all filter layers are learned simultaneously for computational efficiency.
8. Conclusion
The paper derives a generative ConvNet from a discriminative ConvNet and identifies an internal representational structure described as unique among energy-based models. It also points to potential learning from large unlabeled datasets.
- The generative ConvNet is derived from the discriminative ConvNet and reveals an internal representational structure unique among energy-based models.
- The paper’s framework connects generative modeling with the established discriminative ConvNet formulation.
- The generative ConvNet has potential for learning from big unlabeled data through contrastive divergence or reconstruction-based methods.