Source-linked AI summary
Density Modeling of Images using a Generalized Normalization Transformation
Johannes Ballé, Valero Laparra, Eero P. Simoncelli
TL;DR
The paper asks whether useful representations and density models for natural images can be learned without supervision. It introduces an invertible generalized divisive normalization transform, optimized to Gaussianize image data. GDN improves Gaussianization, denoising, and sampling relative to related models and can be cascaded for unsupervised deep-network learning.
Problem
Unsupervised learning seeks representations and density models from observed examples without relying on large labeled datasets.
Method
GDN composes a linear transformation with generalized divisive normalization and learns its parameters by minimizing the non-Gaussianity of transformed natural-image data.
Results
GDN significantly improves Gaussianization, denoising, and sampling compared with ICA-MG and RG.
Takeaways & Limitations
The invertible GDN density model supports likelihood evaluation, sampling, image denoising, and cascaded unsupervised representation learning.
Abstract
from arXiv · showhide
We introduce a parametric nonlinear transformation that is well-suited for Gaussianizing data from natural images. The data are linearly transformed, and each component is then normalized by a pooled activity measure, computed by exponentiating a weighted sum of rectified and exponentiated components and a constant. We optimize the parameters of the full transformation (linear transform, exponents, weights, constant) over a database of natural images, directly minimizing the negentropy of the responses. The optimized transformation substantially Gaussianizes the data, achieving a significantly smaller mutual information between transformed components than alternative methods including ICA and radial Gaussianization. The transformation is differentiable and can be efficiently inverted, and thus induces a density model on images. We show that samples of this model are visually similar to samples of natural image patches. We demonstrate the use of the model as a prior probability density that can be used to remove additive noise. Finally, we show that the transformation can be cascaded, with each layer optimized using the same Gaussianization objective, thus offering an unsupervised method of optimizing a deep network architecture.
1 INTRODUCTION
The paper frames unsupervised representation learning as learning an invertible transformation that maps image data to a fixed target density. It introduces GDN to Gaussianize natural-image patches more effectively than marginal and radial alternatives, while supporting density modeling and cascaded learning.
- Motivation: An invertible, differentiable transformation can map data to a fixed target density whose inverse image defines a density model for inputs.
- Related approaches: PCA, ICA, ICA-MG, and RG can be interpreted as transformation-based density models with increasingly nonlinear or structured Gaussianization.
- Related approaches: ICA-MG applies marginal nonlinearities, while RG applies a radial Gaussianizing nonlinearity and induces elliptically symmetric input distributions.
- Proposed approach: GDN generalizes divisive normalization to Gaussianize local natural-image patches and address dependencies that typical normalization forms do not capture.
- Contributions: GDN includes ICA-MG and RG as special cases and outperforms them in pairwise statistics, image-patch sampling, and denoising, with further gains from a two-stage cascade.
2 PARAMETRIC GAUSSIANIZATION
The paper fits a differentiable transformation that maps data toward a standard normal distribution and thereby defines an input density through the change-of-variables relation. Parameters are learned by minimizing transformed-data non-Gaussianity, with stochastic optimization equivalent to maximizing induced-model likelihood.
- Target distribution: The transformation parameters are selected so that the input vector maps to a standard normal random vector with zero mean and identity covariance.
- Density model: With a standard-normal target, the transformation determines the shape of the input density and induces a density model specified by its parameters.
- Optimization objective: Density estimation is formulated as minimizing the KL divergence, or negentropy, between the transformed density and the standard normal.
- Optimization: The objective can be evaluated over data samples and optimized with stochastic gradient descent, which is equivalent to maximizing the induced density model’s log likelihood.
- Evaluation: The difference in negentropy between input and output provides a practical measure of how much more Gaussian the data become after transformation.
3 DIVISIVE NORMALIZATION TRANSFORMATIONS
The proposed GDN transform combines a linear operation with generalized divisive normalization, extending several existing models through flexible exponents and cross-response interactions. Its continuity, invertibility, constraints, and efficient inversion support its use as a density transformation.
- Divisive normalization: Divisive normalization divides responses by pooled activity from neighboring responses as a form of sensory-neuron gain control.
- Prior limitations: Earlier weighted normalization reduced dependencies and produced approximately Gaussian responses, but was mainly effective for spatially local filter-response groups.
- GDN transform: GDN composes a linear transformation with generalized divisive normalization and uses parameters β, ε, H, α, and γ, totaling 2N + 3N^2 parameters.
- Special cases: Specific parameter choices recover classic divisive normalization, ICA-MG, RG, Lp-symmetric distributions, and ISA as special cases.
- Well-definedness and invertibility: Well-definedness requires a nonsingular H, finite partial derivatives, nonnegative exponents, and a positive denominator; invertibility follows when the normalization Jacobian is positive definite everywhere.
- Constraints: The optimization enforces αij ≥ 1, βi > 0, γij ≥ 0, and 0 ≤ εi ≤ αi^-1, while Jacobian singularities are penalized by the objective.
- Inversion: GDN can be efficiently inverted using a fixed-point iteration, whereas earlier iterative inverse solutions applied only to special cases.
4 EXPERIMENTS
Experiments evaluate GDN against ICA-MG and RG on wavelet coefficients, image patches, denoising, likelihood, and cascaded Gaussianization. Across these tests, GDN better captures varied dependencies, reduces negentropy more, produces natural-looking samples, and supports density-based denoising and cascading.
- 4.1 JOINT DENSITY OF PAIRS OF WAVELET COEFFICIENTS: GDN captures both elliptically symmetric and nearly marginally independent wavelet-coefficient distributions, whereas RG and ICA each fail in one regime.The comparison uses pairwise densities at selected spatial offsets.
- 4.1 JOINT DENSITY OF PAIRS OF WAVELET COEFFICIENTS: 2.43 nats: GDN reduces negentropy, compared with 2.04 nats for ICA-MG and 2.11 nats for RG.Negentropy difference is evaluated as an overall model-fit metric.
- 4.1 JOINT DENSITY OF PAIRS OF WAVELET COEFFICIENTS: GDN comes close to Gaussianizing both marginals and the radial component, while RG fails on marginals and ICA-MG fails on the radial component.The radial component is expected to follow a Chi distribution with degree 256, while marginals are expected to be standard normal.
- 4.2 JOINT DENSITY OVER IMAGE PATCHES: GDN samples reproduce nearly constant patches and sparse organic structures more faithfully than ICA-MG samples.ICA-MG samples are described as more jumbled and filled with random mixtures of oriented elements.
- 4.3 TWO-STAGE CASCADED MODEL: The GDN-induced density is used for Bayesian least-squares removal of additive Gaussian noise from image data.The model estimates the noisy-data density, which is a Gaussian-smoothed version of the clean-image density.
- 4.2 JOINT DENSITY OVER IMAGE PATCHES: 151.5 nats: GDN's cross-validated average log likelihood exceeds ICA-MG's 126.8 nats on mean-removed BSDS300 patches.Without mean removal, GDN achieves 3.47 bits/pixel, essentially equal to the best reported performance in the cited comparison.
- 4.3 TWO-STAGE CASCADED MODEL: A GDN cascade jointly Gaussianizes coefficient groups and replaces ICA rotations with convolutional ICA, reducing stages and avoiding block-boundary artifacts.Convolutional weight sharing also avoids the larger parameter count required by simply increasing transformation size.
5 CONCLUSION
The paper introduces an invertible nonlinear density model that Gaussianizes natural-image data and supports likelihood evaluation, sampling, and inference. Its generalized normalization improves Gaussianization and factorization relative to marginal and radial alternatives, while remaining connected to sensory-coding and cascaded representation learning.
- 5 CONCLUSION: The model combines a linear operation with generalized divisive normalization and optimizes its parameters to Gaussianize natural-image data.The fitting objective minimizes the KL divergence between transformed data and a Gaussian target.
- 5 CONCLUSION: The induced density is not closed-form because it requires evaluating the Jacobian determinant, but it supports direct likelihood computation, sampling, and inference.
- 5 CONCLUSION: The generalized transformation includes ICA/ISA and elliptically symmetric models as special cases while extending divisive-normalization models.
- 5 CONCLUSION: The transformed data are significantly closer to Gaussian than after marginal or radial Gaussianization, yielding a more factorized representation.
- 5 CONCLUSION: The approach combines efficient sensory-coding principles with projection-pursuit Gaussianization and can support cascaded representations such as deep neural networks.
6 APPENDIX
The appendix develops theoretical and practical details for the transformation, including a change-of-variables treatment of negentropy and an invertibility condition based on a positive-definite Jacobian. It also documents preprocessing for the van Hateren and Kodak image datasets.
- 6 APPENDIX: Negentropy of transformed data can be expressed as an expectation over the original data through a change of variables.
- 6 APPENDIX: A continuous transformation is invertible when its Jacobian is positive definite everywhere.
- 6 APPENDIX: The invertibility argument compares distinct inputs and outputs, using Jacobian integration along the line connecting two inputs.
- 6 APPENDIX: The van Hateren preprocessing removed images with more than 0.1% of pixels in the highest-valued histogram bin and then applied a pointwise nonlinearity to 2904 remaining images.
- 6 APPENDIX: Kodak RGB values were converted to linear luminance, followed by a fitted pointwise nonlinearity that removed the mean and marginally Gaussianized intensities.