Invertible Residual Networks

Jens Behrmann; Will Grathwohl; Ricky T. Q. Chen; David Duvenaud; Jörn-Henrik Jacobsen

Source-linked AI summary

Invertible Residual Networks

Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, Jörn-Henrik Jacobsen

arXiv:1811.00995v3Nov 2, 2018cs.LGcs.AIcs.CVstat.ML

TL;DR

The paper addresses the divide between discriminative ResNets and specialized generative architectures by making residual networks invertible through normalization. It combines contractive blocks, fixed-point inversion, and a tractable Jacobian log-determinant approximation, achieving competitive classification and generative-model performance with one architecture.

Problem
Discriminative residual networks and specialized generative architectures differ substantially, complicating the design of a single model for classification and generative tasks.
Method
i-ResNets change ResNet normalization to enforce contractive residual blocks, use fixed-point inversion, and approximate the Jacobian log-determinant for likelihood computation.
Results
i-ResNets perform competitively with non-invertible counterparts for MNIST, CIFAR10, and CIFAR100 classification and with flow-based generative models.
Takeaways & Limitations
One architecture can support discriminative and generative tasks while retaining free-form layer architectures and tractable density estimates.
Takeaways & Limitations
The log-determinant estimator is biased, and designing networks under a Lipschitz constraint is challenging.

Abstract

from arXiv · show

We show that standard ResNet architectures can be made invertible, allowing the same model to be used for classification, density estimation, and generation. Typically, enforcing invertibility requires partitioning dimensions or restricting network architectures. In contrast, our approach only requires adding a simple normalization step during training, already available in standard frameworks. Invertible ResNets define a generative model which can be trained by maximum likelihood on unlabeled data. To compute likelihoods, we introduce a tractable approximation to the Jacobian log-determinant of a residual block. Our empirical evaluation shows that invertible ResNets perform competitively with both state-of-the-art image classifiers and flow-based generative models, something that has not been previously achieved with a single architecture.

1. Introduction

i-ResNets bridge discriminative and generative modeling by making ResNet architectures invertible through a normalization change, while retaining flexible residual blocks and competitive classification performance.

Motivation: A single architecture is desirable across related tasks, but discriminative residual networks and specialized generative models have diverged.This divide complicates architecture selection and limits discriminative tasks’ ability to benefit from unsupervised learning.
Approach: i-ResNets construct invertible residual networks by changing the normalization scheme of standard ResNets.The construction uses the ResNet-as-ODE-discretization viewpoint.
Classification: Each residual block can use an unconstrained architecture provided its Lipschitz constant is smaller than one.Under this restriction, classifiers perform on par with non-invertible counterparts on MNIST, CIFAR10, and CIFAR100.
Generative modeling: i-ResNets support maximum-likelihood generative training on unlabeled data using a tractable approximation to each residual block’s Jacobian determinant.Their free-form Jacobians can learn more expressive transformations than triangular mappings used in other reversible models.
Evaluation: The empirical evaluation finds i-ResNets competitive with both state-of-the-art image classifiers and flow-based generative models.The paper presents this as progress toward general-purpose architectures.

2. Enforcing Invertibility in ResNets

Invertibility follows from viewing ResNet blocks as ODE discretizations and enforcing contractive residual mappings, with inversion obtained numerically by fixed-point iteration and stability supported by Lipschitz bounds.

ODE perspective: ResNet dynamics can be interpreted as Euler discretizations, while solving them backward in time corresponds to inverting the network.A sufficient condition makes the backward dynamics solvable.
Invertibility condition: The sufficient condition is Lip(gθt) < 1 for every residual block.This condition guarantees invertibility but is not necessary, since alternative methods use dimension partitioning or autoregressive structures.
Numerical inversion: Because no analytic inverse is available, each layer is inverted with fixed-point iteration initialized at the layer output.The fixed point is unique, and initializing with y is effective because the residual is a bounded perturbation of the identity.
Stability: Convergence is exponential in iteration count, and smaller Lipschitz constants yield faster convergence.Contractive residual blocks also provide forward and inverse Lipschitz stability guarantees.
Spectral normalization: The implementation enforces the block constraint by directly estimating each convolutional layer’s spectral norm and normalizing it.Power iteration on the parameter matrix provides only a bound for larger-than-1 × 1 filters, so the method estimates the spectral norm using the layer and its transpose.
Spectral normalization: The normalization coefficient c < 1 is only an estimate-based control because the estimated spectral norm underestimates the true norm.Exact post-training inspection can verify whether Lip(g) < 1 holds.

3. Generative Modelling with i-ResNets

i-ResNets use invertibility to define normalizing-flow generative models, while a tractable stochastic power-series approximation makes Jacobian log-determinants scalable to high dimensions.

Generative modelling: i-ResNets parameterize the invertible transformation in a normalizing flow, enabling likelihood evaluation and sampling by inversion.The model samples z and computes x = F^-1(z), while likelihoods use the change-of-variables formula.
Scaling to higher dimensions: Exact Jacobian log-determinants cost O(d^3), making naive likelihood computation impractical for high-dimensional data.This computational barrier motivates the approximation introduced for i-ResNet residual blocks.
Scaling to higher dimensions: A Lipschitz-constrained residual block permits a convergent log-determinant power series because its Jacobian norm is below one.The construction uses the identity ln det(A) = tr(ln(A)) and expands the matrix logarithm as a power series.
Stochastic approximation of log-determinant: The method estimates power-series traces stochastically with Hutchinson’s estimator and vector-Jacobian products, then truncates the series to bound computation.Truncation introduces bias, but the resulting error can be bounded and the gradient approximation has a stated convergence rate.
Implementation: The approximation is implemented during the forward pass with spectral normalization and a selected number of power-series terms for each residual block.The procedure uses the finite series PS(Jg, n) to estimate ln |det(I + Jg)|.
Error of power-series truncation: 5-10 power-series terms suffice in practice for bias below .001 bits per dimension.The paper notes that this is typically reported to .01 precision.

4. Related Work

i-ResNets provide a reversible architecture with free-form mappings and Lipschitz bounds, contrasting with dimension-splitting flows and adaptive ODE solvers. Their inverse is computed iteratively rather than analytically.

Reversible Architectures: NICE, i-RevNet, Real-NVP, and Glow use dimension splitting or autoregressive structures to obtain analytic inverses, imposing architectural restrictions.Glow additionally requires invertible 1 × 1 convolutions, which are inverted numerically.
Reversible Architectures: i-ResNet forward mappings are analytic, while inverse mappings are computed through fixed-point iteration.This places i-ResNet between architectures with analytic inverses and methods requiring numerical ODE solvers.
Reversible Architectures: i-ResNets use free-form Jacobians and provide Lipschitz bounds for both forward and inverse mappings.These properties distinguish them from other reversible approaches by design.
Neural ODEs: Neural ODEs support free-form dynamics but rely on adaptive solvers, leaving scalability to very high-dimensional images unclear.i-ResNets instead define discrete dynamics through residual blocks.
Spectral Sum Approximations: The log-determinant literature includes Taylor approximations, stochastic trace estimation, and Chebyshev-polynomial methods for spectral sums.These approaches motivate the approximation techniques used for i-ResNet likelihood computation.

5. Experiments

Experiments validate i-ResNet invertibility, classification performance, comparison with Glow, and generative modeling. The method preserves competitive classification while generative performance remains below Glow and FFJORD.

Experimental scope: The experimental survey verifies invertibility, evaluates image classification, compares invertible architectures, and studies generative modeling.Experiments cover MNIST, CIFAR10, and CIFAR100 classification, plus MNIST and CIFAR10 generation.
Validating Invertibility and Classification: Larger layer-wise Lipschitz coefficients yield classification performance competitive with baselines while preserving provable invertibility.Very conservative normalization increases classification error across all tested datasets.
Validating Invertibility and Classification: Unconstrained ResNets are not reliably invertible after CIFAR10/100 training, whereas the proposed model guarantees an inverse without significantly harming classification.Unconstrained MNIST ResNets were invertible after training, showing that invertibility without constraints is possible but unpredictable.
Validating Invertibility and Classification: 5-20 fixed-point iterations make reconstruction errors imperceptible, costing 0.15-0.75 seconds for 100 CIFAR10 images.The reconstruction cost corresponds to 5-20 times the forward pass.
Comparison with Other Invertible Architectures: i-ResNets perform competitively with Glow as classifiers and are easier to train using SGD with momentum and learning rate 0.1.Tested Glow variants required Adam or Adamax with much smaller learning rates to avoid divergence.
Generative Modeling: i-ResNet generative models are competitive with highly engineered flow models but underperform Glow and FFJORD.The authors attribute much of the performance gap to their biased log-determinant estimator.

6. Other Applications

i-ResNets are positioned for applications combining discriminative and generative objectives. Their Lipschitz bounds may also support stability-related uses, although those benefits are presented as plausible rather than established.

Application areas: i-ResNets can support applications combining primary discriminative tasks with secondary unsupervised or generative objectives.Examples include joint classification and detection, semi-supervised learning, inverse problems, and adversarial robustness.
Application areas: The proposed architecture is relevant to hybrid density-discriminative models, downstream unsupervised learning, semi-supervised learning, inverse problems, and likelihood-based robustness.These are presented as promising application areas rather than demonstrated application results in this section.
Stability implications: The Lipschitz bounds may aid gradient stability during optimization and adversarial robustness.The paper states this as plausible rather than reporting a direct evaluation of either benefit.

7. Conclusions

i-ResNets combine free-form layer architectures with tractable density estimates, while their log-determinant estimator and Lipschitz-constrained design leave important challenges for future work.

i-ResNets support free-form layer architectures while providing tractable density estimates.
Their unrestricted Jacobians permit expansion and contraction through residual blocks, unlike partitioning-based models that require affine and scaling layers for non-volume preservation.
The log-determinant estimator is biased, although unbiased estimators may improve generative-model performance.
Learning and designing networks under a Lipschitz constraint is challenging because each linear layer must be constrained rather than the block directly.

A. Additional Lemmas and Proofs

The supplementary results establish invertibility, positive Jacobian determinants, fixed-point inversion, and convergence under contractive residual blocks, while stating the assumptions behind these guarantees.

Each residual block is invertible when its residual function has Lipschitz constant below one, enabling inversion through fixed-point iteration.
The fixed-point iteration converges because a residual function with Lip(gθt) < 1 is a contraction on a Banach space.
The proofs compare the condition with prior work that restricts the residual domain and linearizes g because inversion uses a convergent Neumann series.
For F(x) = x + g(x), the inverse mapping obeys bounds derived from the Lipschitz constant of g and the reverse triangle inequality.
If Lip(g) < 1, all eigenvalues of the residual-layer Jacobian have positive real parts and its determinant is positive.
The analysis assumes continuously differentiable activations, bounded inputs, Lipschitz derivatives, and bounded quantities needed to establish convergence rates.

B. Verification of Invertibility

Experiments verify that spectral normalization makes i-ResNet blocks contractive and that the resulting models can be inverted accurately, unlike standard ResNets on CIFAR10.

Roughly two thirds of layers in vanilla and BatchNorm ResNets have singular values above one, making their residual blocks non-contractive.
Spectral normalization keeps every tested i-ResNet residual block contractive.
Fixed-point iteration inverts i-ResNets, whereas inversion is not guaranteed for standard ResNets.
On CIFAR10, vanilla ResNet reconstructions fail completely, while i-ResNet reconstructions are qualitatively and quantitatively exact.
On MNIST, reconstructions are close to exact for both vanilla and i-ResNet models despite no explicit Lipschitz enforcement for the vanilla model.

C. Experimental Details

The experiments compare i-ResNets with Glow and specify convolutional, padding, optimization, density-estimation, and activation choices used for classification and generation.

Discriminative i-ResNets are around 1.2–2.1 times faster than Glow, whereas generative i-ResNets are around 2.6 times slower.
The classification models use 39 pre-activation bottleneck blocks, ELU nonlinearities, and ActNorm before each invertible residual block.
Injective padding adds 13 zero channels before the first residual block; removing it decreased CIFAR10 accuracy by approximately 2%.
Generative MNIST and CIFAR10 models use three scale-blocks, each containing 32 i-ResNet blocks, with invertible squeeze operations between scale-blocks.
Training estimates the log-determinant with a power-series approximation using ten terms for MNIST and five terms for CIFAR10.
Evaluation controls estimator accuracy using bounds and Hutchinson samples to report bit/dim within a tolerance of 0.0002.
Continuous-derivative nonlinearities such as ELU or softplus are recommended because differentiating the estimator requires second derivatives.

D. Fixed Point Iteration Analysis

The inverse is computed efficiently by fixed-point iteration, with reconstruction error decaying quickly. The log-determinant estimator converges with increasing power-series terms, showing negligible bias after 10 terms and numerically zero bias after 20.

Fixed-point iteration: 5-20 fixed-point iterations are sufficient for visually perfect reconstructions on CIFAR10.One iteration corresponds to one forward pass, making inversion approximately 5-20 times slower than inference.
Log-determinant estimation: After 10 terms, the log-determinant estimator’s bias is negligible, and after 20 terms it is numerically 0.The comparison uses brute-force true values and averages over 1000 test examples.
Log-determinant estimation: The log-determinant approximation error converges as the number of power-series terms increases.The estimator variance comes from the stochastic trace estimator.