Source-linked AI summary
Generative Image Modeling Using Spatial LSTMs
Lucas Theis, Matthias Bethge
TL;DR
Modeling natural images requires capturing long-range dependencies while retaining tractable likelihoods. The paper introduces RIDE, a recurrent image model combining spatial LSTMs with conditional mixture models. RIDE shows superior quantitative performance and captures diverse texture statistics, while the broader problem of jointly modeling high- and low-level image statistics remains unresolved.
Problem
Natural-image distributions contain strong dependencies over large distances, creating a challenge for models that must remain both flexible and tractable.
Method
RIDE combines spatial LSTMs with factorized mixtures of conditional Gaussian scale mixtures to predict pixels using broad spatial context.
Results
RIDE shows superior performance in quantitative comparisons and captures many different statistical patterns in texture applications.
Takeaways & Limitations
The model demonstrates that deep recurrent architectures can support tractable generative image modeling and texture synthesis.
Takeaways & Limitations
The paper states that further conceptual work is needed for a model that handles both abstract high-level and low-level image statistics.
Abstract
from arXiv · showhide
Modeling the distribution of natural images is challenging, partly because of strong statistical dependencies which can extend over hundreds of pixels. Recurrent neural networks have been successful in capturing long-range dependencies in a number of problems but only recently have found their way into generative image models. We here introduce a recurrent image model based on multi-dimensional long short-term memory units which are particularly suited for image modeling due to their spatial structure. Our model scales to images of arbitrary size and its likelihood is computationally tractable. We find that it outperforms the state of the art in quantitative comparisons on several image datasets and produces promising results when used for texture synthesis and inpainting.
1 Introduction
Generative image models offer a principled route to unsupervised learning and applications such as reconstruction and compression, but must balance tractability with flexibility. The paper introduces a recurrent, spatially structured model combining multi-dimensional LSTMs with conditional mixture models.
- Motivation: Unsupervised learning can exploit abundant unlabeled data, including multimodal data that may benefit unsupervised approaches.
- Motivation: Generative models provide a principled approach to unsupervised learning by predicting parts of images from other parts.The same framework can support semi-supervised learning when extended with labels.
- Challenge: Generative image modeling must balance the flexibility of deep neural networks with computational tractability.The paper presents its model as fully tractable yet highly flexible.
- Approach: The proposed model combines multi-dimensional recurrent neural networks with mixtures of experts, using spatial LSTMs as its backbone.Spatial LSTMs are presented as naturally suited to generative image modeling because of their multidimensional structure.
- Approach: Mixtures of conditional Gaussian scale mixtures model pixel distributions given hidden states and support sampling images of arbitrary size.The model treats images as instances of a stationary stochastic process to capture correlations in arbitrarily large images.
2 A recurrent model of natural images
The recurrent image model factorizes image density into tractable pixel conditionals while using spatial recurrence to incorporate information from broad image regions. Its factorized MCGSM output models pixel intensities, and spatial LSTMs propagate context across a two-dimensional grid.
- 2.1 Factorized mixtures of conditional Gaussian scale mixtures: The image distribution is factorized into conditional distributions for pixels under a chosen ordering.This factorization follows the probability chain rule and does not itself impose independence assumptions.
- 2.1 Factorized mixtures of conditional Gaussian scale mixtures: MCGSMs improve representational power by assigning conditional distributions their own parameters, while causal neighborhoods and stationarity control parameter growth.Stationarity permits one parameter set to be reused across pixel locations and supports arbitrary image sizes.
- 2.1 Factorized mixtures of conditional Gaussian scale mixtures: The factorized MCGSM reduces parameter growth through additional sharing, enabling larger causal neighborhoods and more mixture components.
- 2.2 Spatial long short-term memory: Spatial LSTMs maintain memory and hidden units on a two-dimensional grid, with each memory unit receiving two preceding states and corresponding forget gates.Input and output gates regulate how inputs affect memory and how memory states produce hidden units.
- 2.3 Recurrent image density estimator: RIDE feeds spatial-LSTM hidden vectors into a factorized MCGSM to predict each corresponding pixel while preserving the autoregressive factorization.Recurrent connections let the estimator use pixels from a much larger region and nonlinearly transform the available context.
- Related work: Unlike DRAW, RIDE uses deterministic likelihood evaluation and models pixel intensities with an MCGSM rather than relying on stochastic latent variables.The cited comparison notes that DRAW's likelihood must be approximated.
3 Experiments
Experiments evaluate RIDE against MCGSM and other generative image models across natural-image and dead-leaf datasets. Results show advantages on large-image likelihood modeling, while performance depends on dataset representation, neighborhood size, and ensemble transformations.
- Experimental setup: RIDE uses stochastic-gradient training followed by MCGSM fine-tuning with L-BFGS, early stopping, and typically a 5-pixel neighborhood with 32 components.The reported training setup used batch size 50, momentum 0.9, and a decreasing learning rate.
- Preprocessing: Conditionally whitening the data speeds training and requires adding the log-Jacobian when evaluating conditional log-likelihood.The preprocessing also introduces a shortcut connection from the pixel neighborhood to the predicted pixel.
- Ensembles: An eight-transformation ensemble of rotations and flips produces a noticeable performance boost without training separate models.The ensemble is formed by mixing transformed-image probabilities with the corresponding determinant factors.
- Natural images: On van Hateren images, RIDE outperforms MCGSM by a large margin when images are treated as stochastic processes, whereas zero-padded 63-dimensional patches reduce RIDE performance.The stochastic-process evaluation uses increasingly sized image patches for RIDE and reports average log-likelihood rates.
- Natural images: On BSDS300, an MCGSM with 16 components on large images captures more correlations than models applied to small patches, and ensembles of RIDEs improve further.The factorized MCGSM uses approximately 3,000 parameters, compared with approximately 400,000 for a 200-component GMM.
- Dead leaves: For dead leaves, larger MCGSM neighborhoods quickly saturate and do not match the improvement from SLSTM nonlinear transformations; RIDE improves several textures but struggles with periodic structure.Some RIDE samples are nearly indistinguishable from real textures, while periodicity failures may reflect LSTM limitations or weak likelihood penalties.
4 Conclusion
RIDE is a deep, tractable recurrent image model based on spatial LSTMs that shows superior quantitative performance and captures diverse texture statistics. The authors view the broader field as still lacking a likely solution for modeling both high-level and low-level natural-image statistics.
- RIDE is a deep but tractable recurrent image model based on spatial LSTMs.
- RIDE shows superior performance in quantitative comparisons.
- RIDE captures many different statistical patterns, as demonstrated through its application to textures.The authors note that natural images can be viewed as collections of textures at an intermediate abstraction level.
- A factorized MCGSM enables more experts and larger causal neighborhoods, while remaining compact, easy to train, and effective as an image model.The authors suggest it may extend models such as DRAW or video models.
- Further conceptual work is needed for a generative image model that handles both abstract high-level and low-level natural-image statistics.