Source-linked AI summary
i-RevNet: Deep Invertible Networks
Jörn-Henrik Jacobsen, Arnold Smeulders, Edouard Oyallon
TL;DR
The paper asks whether information loss is necessary for deep networks to learn representations that generalize well. It builds the fully invertible i-RevNet and finds that information can be preserved while representations progressively contract and separate with depth.
Problem
It was unclear whether the information loss observed in successful deep classifiers is necessary for classification and progressive representation changes.
Method
The paper constructs i-RevNets from homeomorphic layers with an explicit inverse, preserving input information through the final convolutional representation.
Results
i-RevNets achieve the same classification accuracy as comparable RevNet and ResNet architectures on ILSVRC-2012 while retaining recoverable input information.
Takeaways & Limitations
Successful representations can preserve all input information while achieving progressive contraction and linear separation with depth.
Takeaways & Limitations
Estimating the apparent low-dimensional embedding is difficult because local dimensionality estimation in high dimensions remains an open problem.
Abstract
from arXiv · showhide
It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network architectures. In this paper we show via a one-to-one mapping that this loss of information is not a necessary condition to learn representations that generalize well on complicated problems, such as ImageNet. Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded. Building an invertible architecture is difficult, for one, because the local inversion is ill-conditioned, we overcome this by providing an explicit inverse. An analysis of i-RevNets learned representations suggests an alternative explanation for the success of deep networks by a progressive contraction and linear separation with depth. To shed light on the nature of the model learned by the i-RevNet we reconstruct linear interpolations between natural image representations.
1 INTRODUCTION
The paper challenges the belief that successful classification requires progressively discarding input information. It proposes preserving information through homeomorphic layers while achieving progressive contraction and class separation.
- Recovering images from hidden representations is difficult in many common architectures, motivating the question of whether information loss is necessary for classification.
- i-RevNets preserve mutual information between inputs and hidden representations, so information loss can occur only at the final projection.
- The proposed explanation for effective representations is progressive contraction with respect to a meaningful ℓ2 metric in intermediate representations.
- Prior work links invariance and separation, suggesting that contraction should be gradual enough to avoid removing discriminative information.
- The i-RevNet retains input information while progressively separating and contracting signals with depth, including on ImageNet.
2 RELATED WORK
Related work documents information loss and progressive invariance in deep networks, while existing invertible approaches require architectural or optimization constraints. i-RevNet addresses these issues through exact invertibility by construction.
- Prior ImageNet classifiers show increasing information loss with depth, but it remains unclear whether this loss is necessary for progressive separation and contraction.
- Existing invertible frameworks impose conditions such as norm-constrained linear operators or invertible nonlinearities, which are typically unavailable together.
- Parseval representations are recoverable only if their nonlinearities are invertible, while scattering reconstruction relies on gradient descent without guaranteed convergence.
- RevNets avoid storing intermediate activations but remain non-invertible because they use max-pooling and downsampling operators.
- The paper introduces i-RevNet as an invertible RevNet architecture designed to perform competitively on ImageNet.
3 THE i-REVNET
i-RevNet replaces non-invertible network components with invertible splitting, downsampling, and reversible blocks. The resulting models retain recoverable representations while achieving competitive ImageNet performance, with an accuracy–parameter trade-off.
- 3.1 AN INVERTIBLE ARCHITECTURE: i-RevNet alternates additive reversible blocks, nonlinear convolutional operators Fj, and spatial downsampling operators Sj.
- 3.1 AN INVERTIBLE ARCHITECTURE: Invertible modules Sj replace max-pooling and strides by reducing spatial resolution while increasing channels and preserving layer size.
- 3.1 AN INVERTIBLE ARCHITECTURE: The final representation is spatially averaged, passed through ReLU, and projected onto class probes; an explicit inverse or left-inverse recovers the input before projection.
- 3.1 AN INVERTIBLE ARCHITECTURE: The inverse network is obtained by replacing Sj and Fj with their inverses and negated nonlinear operators, requiring minimal adaptation.
- 3.2 ARCHITECTURE, TRAINING AND PERFORMANCES: The i-RevNet (a) has about 6 times more parameters than RevNet and ResNet but achieves similar ImageNet validation accuracy.
- 3.2 ARCHITECTURE, TRAINING AND PERFORMANCES: The bijective i-RevNet (b) has roughly matched baseline parameter counts but performs 1.5% lower in absolute ImageNet accuracy than the RevNet baseline.
4 ANALYSIS OF THE INVERSE
The i-RevNet’s explicit inverse enables reconstruction despite severe local ill-conditioning. Feature-space interpolation produces recognizable but noisy image sequences, while depth-wise representations support progressively better classification.
- Analysis setup: The analysis evaluates the i-RevNet and its inverse using local stability, reconstruction, feature interpolation, and depth-wise classification experiments.The depth-wise classification protocol applies linear SVM and nearest-neighbor classifiers to spatially averaged representations.
- An ill-conditioned inversion: The differential’s singular values decay rapidly: the first 10^3 and 10^4 account for 80% and 97% of cumulative energy, respectively.This indicates strong local contraction into a smaller space, although the resulting dimensionality remains large.
- An ill-conditioned inversion: Rapid singular-value decay makes inversion ill-conditioned, so the paper uses a formal reconstruction algorithm rather than obtaining the inverse implicitly.The differential is a very large matrix, making direct computation expensive.
- Linear interpolation and reconstruction: Feature-space linear interpolation is reconstructed into image sequences whose intermediate images are noisy but mostly recognizable.The sequences connect endpoint images through convex paths in representation space and include examples from face, texture, and ImageNet datasets.
- Linear interpolation and reconstruction: Some geometric transformations, including apparent 3D rotation, seem to become linearized in the learned representation.The paper presents this observation as motivation for examining how linear separation changes with depth.
5 A CONTRACTION
The i-RevNet develops representations that become progressively more linearly separable and contracted with depth, despite retaining input information, while classification remains possible in a much smaller PCA subspace.
- 5 A CONTRACTION: The i-RevNet study examines depth-wise classification improvement and whether its feature space can be reduced to a smaller discriminative subspace.The experiments use localized or linear classifiers and PCA-based dimensionality analysis.
- 5.1 PROGRESSIVE LINEAR SEPARATION AND CONTRACTION: Both ResNet and i-RevNet representations become progressively more linearly separable and contracted with depth, including for the information-preserving i-RevNet.This supports progressive contraction and separation as an alternative to information discarding.
- 5.1 PROGRESSIVE LINEAR SEPARATION AND CONTRACTION: Classifiers improve similarly with depth, while the final three layers show a stronger performance jump consistent with earlier preparation for contraction and linear separation.The linear SVM performs slightly better than nearest neighbor, and the i-RevNet CNN reaches 77% classification accuracy in the reported setting.
- 5.1 PROGRESSIVE LINEAR SEPARATION AND CONTRACTION: The proposed interpretation suggests that the representation may have a low-dimensional embedding, but this is difficult to validate because local dimensionality estimation in high dimensions remains an open problem.The paper treats the dimensionality claim as suggestive rather than fully established.
- 5.2 DIMENSIONALITY ANALYSIS OF THE FEATURE SPACE: A PCA of the final representation preserves classification-relevant information in a small subspace, with accuracy decreasing significantly only when d ≤200.The experiment projects features onto the first d principal components and evaluates nearest-neighbor and linear-SVM classifiers on the 100-class task.
6 CONCLUSION
The conclusion presents i-RevNet as a fully invertible CNN that matches comparable architectures on ImageNet while retaining information until the final class projection. It argues that progressive separation and contraction, rather than information loss, can explain successful representations.
- 6 CONCLUSION: Figure 7 plots linear-SVM and nearest-neighbor accuracy against the number of retained principal components.The figure concerns the dimensionality of the representation used for classification.
- 6 CONCLUSION: The paper reports empirical evidence that invertible representations can learn successfully on large-scale supervised problems without discarding input information.This is presented as the first such empirical evidence to the authors’ knowledge.
- 6 CONCLUSION: i-RevNet exactly recovers the input from its last convolutional layer and achieves the same classification accuracy as comparable RevNet and ResNet architectures on ILSVRC-2012.The inverse network is obtained during training with only minimal adaptation for recovering inputs.
- 6 CONCLUSION: The proposed explanation for generalization is progressive separation and contraction with depth, demonstrated on ImageNet despite the absence of information loss.The conclusion directly contrasts this account with the belief that discarding information is essential for generalization.