PaperlayerView exact revision on arXiv

Source-linked AI summary

Sylvester Normalizing Flows for Variational Inference

Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomczak, Max Welling

arXiv:1803.05649v2stat.MLcs.AIcs.LGstat.ME

TL;DR

Variational inference needs tractable, expressive posterior families because simple parametric approximations may be insufficient. The paper introduces Sylvester normalizing flows, which generalize planar flows and compare favorably with planar flows and IAFs across several datasets.

  • Problem

    Variational inference needs tractable, expressive variational families because simple parametric approximations may inadequately represent true posteriors.

  • Method

    The paper introduces Sylvester normalizing flows, using Sylvester’s determinant identity and orthogonal or triangular parameter matrices to construct invertible flows.

  • Results

    Sylvester normalizing flow variants compare favorably against planar flows and inverse autoregressive flows on four datasets.

  • Takeaways & Limitations

    The results highlight data-dependent flow parameters as important for flexible transformations and connect triangular Sylvester flows closely to mean-only IAF.

  • Takeaways & Limitations

    IAF transformations may require wide MADE or deep PixelCNN layers, resulting in transformations with many parameters.

Abstract

from arXiv · show

Variational inference relies on flexible approximate posterior distributions. Normalizing flows provide a general recipe to construct flexible variational posteriors. We introduce Sylvester normalizing flows, which can be seen as a generalization of planar flows. Sylvester normalizing flows remove the well-known single-unit bottleneck from planar flows, making a single transformation much more flexible. We compare the performance of Sylvester normalizing flows against planar flows and inverse autoregressive flows and demonstrate that they compare favorably on several datasets.

1 INTRODUCTION

The introduction motivates more flexible variational posteriors and presents Sylvester normalizing flows as a generalization of planar flows that removes their bottleneck. It positions SNFs against planar flows and IAFs, highlighting a relationship to IAF and the importance of data-dependent flow parameters.

  • Motivation: Variational inference can be insufficient because its parametric approximation recovers the true posterior exactly only when that posterior belongs to the chosen family.The introduction specifically identifies simple diagonal-covariance Gaussian families as likely insufficient.
  • Normalizing flows: Normalizing flows construct flexible variational distributions by transforming a base density through invertible parametric transformations with tractable Jacobians.Planar and radial flows were introduced as initial flow classes, but they can be difficult to train and may require many transformations.
  • Paper contribution: The paper introduces Sylvester normalizing flows using Sylvester’s determinant identity as a generalization of planar flows that removes their bottleneck.This construction is intended to make a single transformation more flexible.
  • Paper contribution: The authors compare multiple SNF variants against planar flows and IAFs and report that they compare favorably.One SNF variant is related to IAF, differing mainly in how flow parameters are amortized.
  • Paper contribution: The introduction emphasizes data-dependent flow parameters as important in addition to flexible transformations.This point follows from the paper’s comparison with IAF and its discussion of amortization strategy.

2 VARIATIONAL INFERENCE

Variational inference replaces intractable marginalization over latent variables with an approximate posterior optimized through the evidence lower bound (ELBO). Because simple diagonal-covariance Gaussian posteriors can yield loose bounds and biased parameter estimates, more flexible approximations such as normalizing flows are needed.

  • Variational inference introduces q(z|x) to approximate the posterior when integrating over latent variables is generally intractable.
  • The ELBO combines reconstruction error with KL divergence from the approximate posterior to the prior, which acts as a regularizer in VAEs.
  • A better variational approximation produces a tighter ELBO, while diagonal-covariance Gaussian posteriors are a widely used simple choice.
  • Loose ELBOs can bias maximum likelihood estimates of θ and harm generative performance, motivating more flexible approximate posterior distributions.
  • Normalizing flows construct flexible posteriors by transforming a simple base distribution through invertible transformations with easily computable Jacobians.
  • Planar flows compute Jacobian determinants in O(D) time but require many transformations because their update effectively acts as a single-neuron MLP.

3 SYLVESTER NORMALIZING FLOWS

Sylvester normalizing flows generalize planar flows with an M-unit residual transformation whose Jacobian determinant reduces to an M × M determinant. A structured triangular orthogonal case guarantees invertibility and yields three practical flow variants that preserve orthogonality in different ways.

  • Sylvester transformation: The proposed transformation uses A ∈ R^{D×M}, B ∈ R^{M×D}, b ∈ R^M, and M ≤ D in a residual MLP-like layer.It is described as a generalization of planar flows with M hidden units.
  • Jacobian determinant: When M < D, Sylvester’s determinant identity reduces the determinant computation from a D × D matrix to an M × M matrix.The resulting Jacobian determinant is expressed using det(IM + diag(h′(Bz + b))BA).
  • Invertibility: Invertibility is ensured when R and R̃ are upper triangular, R̃ is invertible, h has bounded positive derivative, and rii r̃ii > −1/∥h′∥∞.Under these conditions, the transformation is invertible; its determinant can be computed in O(M).
  • Flow variants: The flow variants preserve orthogonality using differentiable orthogonalization, Householder-reflection products, or alternating identity and reversal-permutation matrices.The variants are orthogonal Sylvester flows (O-SNF), Householder Sylvester flows (H-SNF), and triangular Sylvester flows (T-SNF).
  • Flow variants: The orthogonal variant permits M < D as a flexible bottleneck, whereas Householder constructions require M = D because they produce square matrices.Householder flows trade off parameter count and transformation generality through the number H of reflections.

4 RELATED WORK

Prior flow families offer tractable Jacobians but face scaling, sequential-sampling, parameterization, or inverse-tractability limitations. Sylvester flows relate to IAF and Householder flows while using input-dependent parameters to obtain flexible transformations with fewer parameters.

  • Planar and radial flows: Planar and radial flows are effective for small problems but difficult to scale to large latent spaces and often require many transformations.
  • Inverse Autoregressive Flows: IAF models autoregressive distributions, but sampling a vector requires D sequential steps because each transformation parameter depends on preceding variables.
  • Inverse Autoregressive Flows: Increasing IAF flexibility requires nonlinear autoregressive functions, with wide MADEs or deep PixelCNN layers producing transformations with many parameters.
  • Triangular Sylvester flows: T-SNF is strongly related to mean-only IAF, but directly amortizes R and R̃ as functions of x, equivalent to amortizing mean-only IAF's MADE parameters.
  • Householder Sylvester flows: Householder flows are a special case of H-SNF under h(z) = z, R as the identity matrix, and no residual connection.
  • Density-estimation flows: Density-estimation flows are generally not directly applicable to variational inference because both transformation directions need not be tractable.

5 NUMBER OF PARAMETERS

The section compares parameter counts for planar flows, IAF, and three Sylvester normalizing flows. Planar flows use the fewest parameters but generally perform worse, whereas SNF parameter counts scale quadratically with latent dimension and remain amortizable.

  • Parameter comparison: Planar flows require the fewest parameters but generally yield worse results; SNF counts are quadratic in latent dimension and can still be amortized despite being large.IAFs can also become parameter-heavy because good performance requires a large MADE width.

6 EXPERIMENTS

Experiments on four image datasets compare Sylvester flows with a factorized-Gaussian VAE, planar flows, and IAFs under shared model architectures. Sylvester flows generally outperform the alternatives, although planar flows perform best on the small Freyfaces dataset.

  • Experimental setup: The study evaluates Sylvester flows on static MNIST, Freyfaces, Omniglot, and Caltech 101 Silhouettes against a factorized-Gaussian VAE, planar flows, and IAFs.All models use the same base encoder and decoder architecture, with 64-dimensional latent spaces.
  • Static MNIST: On static MNIST, performance improves with more flows; planar flows approach the baseline with 4 flows but benefit clearly from additional transformations.The baseline–planar difference is very small at 4 flows.
  • Static MNIST: All three Sylvester flow variants outperform IAF and planar flows on static MNIST, and larger Orthogonal Sylvester bottlenecks improve performance.The comparison uses orthogonal-vector counts M = 16 and M = 32 for latent dimension D = 64; Householder results for H = 4 and H = 8 are nearly indistinguishable.
  • Other datasets: T-SNF’s advantage over IAF indicates that input-dependent flow parameters directly provide a more flexible transformation than a wide MADE with data-dependent context.T-SNF is closely related to mean-only IAF, but generates MADE parameters through an x-dependent hypernetwork.
  • Other datasets: On Freyfaces, all normalizing flows improve performance, with planar flows best and Triangular and Householder Sylvester flows closely behind.The paper attributes planar flows’ advantage to their lower sensitivity to overfitting on this small dataset of around 2000 faces.
  • Other datasets: On Omniglot and Caltech 101 Silhouettes, Sylvester flows perform best, with H-SNF and T-SNF outperforming O-SNF.The paper attributes O-SNF’s weaker result to its M = 32 bottleneck for latent dimension D = 64.

7 CONCLUSION

The paper introduces Sylvester normalizing flows, a family that generalizes planar flows while retaining efficiently computable Jacobian determinants and ensuring invertibility through structured parameter matrices.

  • 7 CONCLUSION: Sylvester normalizing flows generalize planar flows while preserving efficient Jacobian-determinant computation via Sylvester’s determinant identity.The flows ensure invertibility using orthogonal and triangular parameter matrices.
  • 7 CONCLUSION: The paper investigates three variants, including orthogonal Sylvester flows, which maintain parameter-matrix orthogonality iteratively.The supplied passage introduces the first variant but truncates the descriptions of the remaining two.

A Architecture

The architecture uses convolutional encoder and decoder layers with gated activations. The encoder ends in a fully connected layer for diagonal-Gaussian and flow parameters, while the decoder mirrors the encoder with transposed convolutions.

  • A Architecture: Convolutional layers in both encoder and decoder use a gated activation function.The activation specification defines layer inputs and outputs through convolutional weights, biases, and a sigmoid activation.
  • A Architecture: The encoder uses convolutional layers with the specified kernel, padding, and stride settings.Its highlighted sequence progresses from 1 to 32, then 64 channels, with stride-2 downsampling in the second and fourth layers.
  • A Architecture: The encoder concludes with Conv(in = 64, out = 256, k = 7, p = 0, s = 1), which acts as a fully-connected layer.Fully connected linear layers parameterize a diagonal Gaussian distribution and amortize flow parameters.
  • A Architecture: The decoder mirrors the encoder using transposed convolutional layers.The listed decoder begins with ConvT(in = 64, out = 64, k = 7, p = 0, s = 1) and ends with ConvT(in = 32, out = 1, k = 1, p = 0, s = 1).

A.1 Description of datasets

Experiments used four image datasets—static MNIST, OMNIGLOT, Caltech 101 Silhouettes, and Frey Faces—with mostly 28 × 28 images. The datasets varied in size, content, and split design, including validation sets and repeated experiments for Frey Faces.

  • The experiments used static MNIST, OMNIGLOT, Caltech 101 Silhouettes, and Frey Faces.
  • Most datasets contain 28 × 28 images, whereas Frey Faces contains 28 × 20 images.
  • MNIST has 60,000 training datapoints and 10,000 test sample points, with 10,000 training images reserved for model selection.
  • OMNIGLOT contains 1,623 handwritten characters from 50 alphabets, split into 24,345 training datapoints and 8,070 test images, with 1,345 training examples used for validation.Dynamic binarization was applied during training.
  • Caltech 101 Silhouettes has 4,100 training images, 2,264 validation datapoints, and 2,307 test examples across 101 object classes.Its small training sample size and many classes make learning challenging.
  • Frey Faces contains nearly 2,000 grayscale images, split into 1,565 training, 200 validation, and 200 test images, with the experiment repeated 3 times.

B MNIST experiments

The MNIST experiments report negative evidence lower bounds on the test set, with exact values provided in Table 3. All results use stochastic hidden units of size 64.

  • B MNIST experiments: Table 3 lists the exact negative evidence lower bounds for the MNIST test set.The values correspond to the evidence lower bound shown in Fig. 3.
  • B MNIST experiments: All MNIST results are obtained with stochastic hidden units of size 64.
Loading 1803.05649v2…