PaperlayerView exact revision on arXiv

Source-linked AI summary

MintNet: Building Invertible Neural Networks with Masked Convolutions

Yang Song, Chenlin Meng, Stefano Ermon

arXiv:1907.07945v2cs.LGcs.CVstat.ML

TL;DR

Many invertible networks trade architectural flexibility against efficient inversion and exact Jacobian computation. MintNet composes triangular modules into flexible networks with parallelizable inversion and tractable determinants; it reports competitive classification and improved generative likelihood results across the tested image datasets.

  • Problem

    Many invertible-network applications need efficient inversion and Jacobian determinants, but existing approaches can impose architectural constraints or approximate determinants.

    Source: §1 · p. 1

  • Method

    MintNet composes triangular building blocks into invertible modules whose Jacobians are triangular and whose inverses can be computed with a parallelizable fixed-point iteration.

    Source: §1 · p. 1, §6 · p. 9

  • Results

    MintNet matches a similarly architected ResNet at 99.6% MNIST accuracy and reports lower bpd than prior best results on MNIST, CIFAR-10, and ImageNet 32×32, including 0.98 bpd on MNIST.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The reported results show that MintNet combines ResNet-like architecture with competitive classification and generative performance on the evaluated image datasets.

    Source: §1 · p. 2, §6 · p. 9

    The theorem's non-singularity condition is sufficient but not necessary, so it does not certify every invertible function.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

We propose a new way of constructing invertible neural networks by combining simple building blocks with a novel set of composition rules. This leads to a rich set of invertible architectures, including those similar to ResNets. Inversion is achieved with a locally convergent iterative procedure that is parallelizable and very fast in practice. Additionally, the determinant of the Jacobian can be computed analytically and efficiently, enabling their generative use as flow models. To demonstrate their flexibility, we show that our invertible neural networks are competitive with ResNets on MNIST and CIFAR-10 classification. When trained as generative models, our invertible networks achieve competitive likelihoods on MNIST, CIFAR-10 and ImageNet 32x32, with bits per dimension of 0.98, 3.32 and 4.06 respectively.

1 Introduction

MintNet addresses the tension between flexible architectures and efficient inversion and exact Jacobian determinants by composing triangular building blocks. Its ResNet-like design performs competitively on classification and achieves better reported image-model likelihoods than prior best results.

1 Introduction

  • Flexible invertible networks often face trade-offs among efficient inversion, efficient Jacobian determinants, and architectural freedom.Some flows constrain hidden dimensions or use dimension partitioning; FFJORD and i-ResNets have fewer architectural constraints but approximate Jacobian determinants.Source: §1 · p. 1
  • MintNet composes triangular modules into a flexible, ResNet-like architecture that supports parallelizable inversion and exact, efficient Jacobian computation.The paper also reports fewer parameters and lower training-resource use than the cited generative-model baselines.Source: §1 · pp. 1–2
  • MintNet reports 99.6% MNIST accuracy, matching a similarly architected ResNet, and 91.2% CIFAR-10 accuracy versus ResNet's 92.6%.For generative modeling, its bpd values are 0.98 on MNIST, 3.32 on CIFAR-10, and 4.06 on ImageNet 32×32, below the cited prior bests.Source: §1 · p. 2

2 Background

The background defines invertible networks and explains their roles in classification and generative modeling. Invertibility enables tracing feature representations and using change of variables to evaluate and sample from density models.

2 Background

  • For a continuous invertible map from data to latent representations, input and latent dimensions must match, and compositions of invertible maps remain invertible.The paper focuses on classification and generative modeling.Source: §2 · p. 2

2.1 Classification with invertible neural networks

  • Invertible classifiers treat the feature extractor as the invertible component, since the class count usually differs from the input dimension.The paper notes that latent representations can be inverted to trace predictions.Source: §2.1 · p. 2

2.2 Generative modeling with invertible neural networks

  • Change of variables relates data density to latent density and the Jacobian determinant, so tractable determinants permit density evaluation when the base distribution is known.The Jacobian must be non-singular for the log absolute determinant to be defined.Source: §2.2 · p. 2
  • An invertible density model can be trained by maximum likelihood and generate data by drawing from a base distribution and applying the network inverse.Invertible components can also be stacked; their inverses are applied sequentially and their Jacobian determinants multiply.Source: §2.2 · p. 2

3 Building invertible modules compositionally

The section builds invertible modules from triangular-Jacobian blocks, using composition rules and a non-singularity condition, then describes a fixed-point method for inversion. Triangular Jacobians also make determinant computation efficient.

3.1 The basic module

  • A triangular linear transformation is the basic module: its determinant is the product of its diagonal entries, reducing computation from O(D^3) to O(D).Masked convolutions create the triangular structure through causal filter ordering.Source: §3.1 · p. 3

3.2 The calculus of building invertible modules

  • Addition and composition preserve lower-triangular Jacobians, and element-wise differentiable activations can be composed with functions in F.These rules recursively produce continuous, nonlinear modules; addition alone does not guarantee invertibility.Source: §3.2 · pp. 3–4
  • Theorem 1 guarantees invertibility for functions in F when their Jacobians are non-singular throughout the domain, but this condition is sufficient, not necessary.For example, x^3 is invertible despite having a singular Jacobian at zero.Source: §3.2 · p. 4

3.3 Efficient inversion of the invertible modules

  • A fixed-point iteration computes the inverse and is locally convergent when 0 < α < 2, provided its initial value is close to the true inverse.Each update uses the function value and the diagonal of its Jacobian.Source: §3.3 · pp. 4–5
  • The inversion method is GPU-parallelizable with cost roughly O(T), and experiments report fast convergence when T ≪ D.This contrasts with sequential autoregressive inversion requiring at least O(D) iterations; the method's theoretical convergence guarantee remains local.Source: §3.3 · pp. 4–5

4 Masked Invertible Networks

MintNet builds invertible networks from masked-convolution Mint layers and composition rules, with paired layers and squeezing used to form deeper architectures. The paper contrasts this design with alternatives that constrain capacity, partition dimensions, invert sequentially, or approximate Jacobian determinants.

4.1 Building the Masked Invertible Layer

  • Mint layers use masked convolutions to form invertible modules, and MintNet stacks them into a deep network.The Mint layer is the basic layer; the full network is constructed from paired Mint layers and squeezing layers.Source: §4.1 · p. 5, §4.2 · p. 6

4.2 Constructing the Masked Invertible Network

  • Pairing lower- and upper-triangular Mint layers makes the combined Jacobian non-triangular and eliminates masked-convolution blind spots.The authors state that constraining the network Jacobian to remain triangular would limit capacity.Source: §4.2 · p. 6
  • Squeezing reduces each feature map’s height and width by a factor of 2 while increasing channels by a factor of 4, with an identity Jacobian.This invertible reshaping enlarges the receptive field without using pooling or strided convolutions.Source: §4.2 · p. 6

4.3.1 Models based on identities of determinants

  • MintNet adds a nonlinear convolutional layer to Sylvester flow and avoids its bottleneck on the maximum latent-representation dimension.Sylvester flow’s M ≤ D constraint is described as a bottleneck that limits expressive power.Source: §4.3.1 · p. 6

4.3.2 Models based on dimension partitioning

  • Unlike NICE, Real NVP, and Glow, MintNet’s Jacobian diagonal is not partially restricted to 1s by dimension partitioning.The cited coupling-layer approaches rely on heuristic partitions, and their performance is sensitive to that choice.Source: §4.3.2 · pp. 6–7

4.3.3 Models based on autoregressive transformations

  • MintNet uses fixed-point iteration for inversion rather than the sequential solving required by autoregressive models.The cited autoregressive inversion has a naïve cost of Ω(D) computations; MintNet’s architecture is described as more flexible.Source: §4.3.3 · p. 7

4.3.4 Free-form invertible models

  • Unlike FFJORD’s ODE solver and stochastic trace estimator, MintNet computes Jacobian determinants exactly.The comparison also notes that i-ResNet approximates its log-determinant with a truncated power series and stochastic estimators.Source: §4.3.4 · p. 7

4.3.5 Other models using masked convolutions

  • On CIFAR-10, MintNet inversion takes 120 iterations versus 3,072 for emerging convolutions, and can be 25 times faster on powerful GPUs.Emerging convolutions and MaCow use forward/back substitutions and are built on Glow; MintNet instead uses a ResNet architecture with causal convolutions.Source: §4.3.5 · p. 7

5 Experiments

MintNet is evaluated for classification, density estimation, inversion, and sampling on image datasets. It matches ResNet test accuracy on MNIST, sets reported bpd records on three datasets, and its reconstruction error converges within 120 iterations.

5.1 Classification

  • MintNet reaches 99.6% test accuracy on MNIST, matching ResNet, and 91.2% on CIFAR-10 versus ResNet’s 92.6%.Both models achieve 100% training accuracy on MNIST and CIFAR-10.Source: §5.1 · p. 7

5.2 Density estimation and verification of invertibility

  • MintNet sets new bits-per-dimension records on MNIST, CIFAR-10, and ImageNet 32×32, using fewer parameters than prior best models.The MNIST model uses 30% fewer parameters than FFJORD; the CIFAR-10 and ImageNet 32×32 models use 60% and 74% fewer parameters than Glow, respectively. The reported bpd values concern uniformly dequantized images and are not directly comparable to discrete-distribution results.Source: §5.2 · p. 8, Table 1 · p. 8
  • Normalized L2 reconstruction error converges within 120 iterations on all three datasets considered.The step sizes were 3.5 for MNIST, 1.1 for CIFAR-10, and 1.15 for ImageNet 32×32; the MNIST setting lacks the stated theoretical local-convergence guarantee.Source: §5.2 · p. 8
  • MintNet sampling is comparable in speed to i-ResNet and roughly 5 times faster than autoregressive sampling on MNIST, and 25 times faster on CIFAR-10 and ImageNet 32×32.Sampling used 120 inversion iterations for each dataset.Source: §5.2 · p. 8

6 Conclusion

MintNet combines masked-convolution building blocks and composition rules to create deep invertible networks. Its modules support efficient inversion and exact determinant computation, and the paper reports strong classification and generative results.

6 Conclusion

  • Composition rules build flexible, nonlinear invertible modules from triangular-matrix transformations, provided their Jacobians are non-singular.The composed modules have triangular Jacobians, supporting efficient, exact determinant computation.Source: §6 · pp. 8–9
  • Figures show uncurated samples across three datasets and an accuracy analysis of Algorithm 1, including reconstruction and original-image rows.Figure 4’s second, fourth, and sixth rows are reconstructions; the other rows are originals.Source: Figure 4 · p. 9, Figure 3 · p. 9
  • MintNet performs well on MNIST and CIFAR-10 classification and achieves new state-of-the-art generative performance on MNIST, CIFAR-10, and ImageNet 32×32.The supplied conclusion does not give the benchmark values.Source: §6 · p. 9
  • Masked convolutions form Mint layers that can be stacked into a deep network following a ResNet-block architecture.The resulting MintNet is inverted with a fixed-point iteration algorithm.Source: §6 · p. 9

A Proofs

The proofs establish closure rules for continuously differentiable functions with triangular Jacobians, use triangular structure and monotonicity to prove invertibility, and show local convergence of the inversion iteration.

A Proofs

  • The function class with continuously differentiable, lower-triangular Jacobians is closed under addition, composition, and elementwise continuously differentiable activations.The proofs verify that these operations preserve continuous differentiability and lower-triangular Jacobians.Source: A Proofs · p. 12
  • A lower-triangular Jacobian means output component i depends only on input components x_1 through x_i.This dependency structure lets the inverse solve coordinates in sequence.Source: A Proofs · pp. 12–13
  • The diagonal-sign condition makes each output component monotonic in its corresponding input when earlier inputs are fixed.The proof derives this from the sign of the corresponding diagonal Jacobian entry.Source: A Proofs · p. 12
  • A function in this class is invertible when its Jacobian is non-singular throughout the domain.The proof uses triangular dependencies and coordinate-wise monotonicity to establish a unique inverse.Source: A Proofs · p. 13
  • Algorithm 1 is locally convergent when 0 < α < 2.The proof analyzes the iteration map at the solution and shows its spectral radius is less than one under this condition.Source: A Proofs · p. 13

B Masked convolutions

Masked convolutions provide the basic triangular module, and MintNet implements its L(x) transformation with three masked convolutional layers.

B Masked convolutions

  • The masked-convolution setup uses odd R × R kernels, zero-padding of ⌊R/2⌋, and equal input and output channel counts to preserve shape.The mask is applied elementwise to the convolution weights before use.Source: B Masked convolutions · p. 13
  • MintNet implements L(x) with three masked convolutions, using intermediate channel widths KC and KC before returning to C channels.The three weight tensors have shapes (KC, C, R, R), (KC, KC, R, R), and (C, KC, R, R).Source: B Masked convolutions · p. 13
  • The expression for L(x) combines masked convolutions with elementwise activations and biases b1, b2, and b3.The symbol ⊛ denotes discrete 2D convolution.Source: B Masked convolutions · p. 14

C Interpolation of hidden representations

The interpolation procedure combines features of four dataset images in latent space, then maps the result back to the image domain.

C Interpolation of hidden representations

D Experiment setup and network architecture

This section reports training and architecture settings for classification and density estimation, along with sampling-step tuning and a reconstruction-error check.

Training and compute settings

  • Classification uses the standard MNIST and CIFAR-10 train/test split, while density-estimation CIFAR-10 and ImageNet 32×32 runs use two Titan XP GPUs.Other runs use one Titan XP GPU; density estimation follows the cited settings, with dequantized images transformed by the logit transformation and models trained using AMSGrad.Source: D Experiment setup and network… · p. 14

Classification setup

  • For classification, MintNet and ResNet are trained for 200 epochs with AMSGrad, cosine learning-rate decay, an initial rate of 0.001, and batch size 128.Images are padded to 16 channels with zeros; both classifiers use global average pooling followed by a fully connected layer.Source: D Experiment setup and network… · p. 14

Classification architecture

  • MintNet uses 19 grouped invertible layers comprising 38 residual invertible layers, while the ResNet has 38 pre-activation residual blocks; Table 2 covers MintNet’s classification architecture.Both use three 3 × 3 convolutions per block, and batch normalization before MintNet layers preserves invertibility at test time.Source: D Experiment setup and network… · p. 14, Table 2 · p. 16

Density estimation setup

  • Density-estimation schedules run for 400 epochs on MNIST, 200 epochs on CIFAR-10, and 350k steps on ImageNet 32×32, all with initial learning rate 0.001.MNIST decays the rate by 10 at epochs 250 and 350; CIFAR-10 and ImageNet 32×32 use cosine decay.Source: D Experiment setup and network… · p. 15

Density estimation architecture

Sampling step size

  • The selected sampling α values are 3.5 for MNIST, 1.1 for CIFAR-10, and 1.15 for ImageNet 32×32.Grid search first identified fastest-converging intervals of [3, 4], [1] [2], and [1] [2], respectively, then refined the search in those intervals.Source: D Experiment setup and network… · p. 15

Invertibility verification

  • Invertibility is checked using normalized L2 reconstruction error on 128 images per dataset, with Fig. 4 plotting the exponential of the mean log error.The shaded area represents the exponential of the standard deviation of log reconstruction errors.Source: D Experiment setup and network… · p. 15

E Additional tables

The additional tables cover comparisons with common invertible models and sampling times across specified methods.

Additional comparisons

  • Table 5 presents a comparison with common invertible models.Source: Table 5 · p. 19

Sampling time

  • Table 6 compares sampling times for 64 samples across MintNet, i-ResNet, and an autoregressive method on the same model architectures, measured on an NVIDIA TITAN Xp.Source: Table 6 · p. 19

F More Samples

The section provides additional uncurated MintNet samples for MNIST, CIFAR-10, and ImageNet 32×32.

More samples

Related papers

Loading 1907.07945v2…