PaperlayerView exact revision on arXiv

Source-linked AI summary

Invertible Generative Modeling using Linear Rational Splines

Hadi M. Dolatabadi, Sarah Erfani, Christopher Leckie

arXiv:2001.05168v4stat.MLcs.LG

TL;DR

High-dimensional normalizing flows need tractable Jacobian determinants, while spline-based alternatives can require polynomial equation solving for inversion. This paper uses linear rational splines in coupling layers, reporting competitive performance with a straightforward inverse.

  • Problem

    Computing Jacobian determinants is a bottleneck for normalizing flows in high-dimensional settings, and piecewise-function alternatives may require solving polynomial equations to invert them.

    Source: §1 · p. 1, §3.1 · p. 4

  • Method

    The paper uses monotonic linear rational splines as coupling-layer transformations, with a straightforward inverse that avoids solving degree 2 or 3 polynomial equations.

    Source: §3.1 · p. 4, §5 · p. 8

  • Results

    The proposed method performs competitively with more complicated methods across experiments on synthetic and real datasets.

    Source: §4 · p. 6, §5 · p. 8

  • Takeaways & Limitations

    The same family of functions defines both forward and inverse transformations, making the models' mathematical properties more straightforward to investigate.

    Source: §5 · p. 8

    i-ResNets and Residual Flows lack analytical inverses and use fixed-point iteration, making inversion around 5–20x slower than inference.

    Source: §2.3 · p. 3

Abstract

from arXiv · show

Normalizing flows attempt to model an arbitrary probability distribution through a set of invertible mappings. These transformations are required to achieve a tractable Jacobian determinant that can be used in high-dimensional scenarios. The first normalizing flow designs used coupling layer mappings built upon affine transformations. The significant advantage of such models is their easy-to-compute inverse. Nevertheless, making use of affine transformations may limit the expressiveness of such models. Recently, invertible piecewise polynomial functions as a replacement for affine transformations have attracted attention. However, these methods require solving a polynomial equation to calculate their inverse. In this paper, we explore using linear rational splines as a replacement for affine transformations used in coupling layers. Besides having a straightforward inverse, inference and generation have similar cost and architecture in this method. Moreover, simulation results demonstrate the competitiveness of this approach's performance compared to existing methods.

1 INTRODUCTION

Normalizing flows represent data distributions through invertible transformations, but computing their Jacobian determinants is a bottleneck in high dimensions. This work introduces a transformation with an analytical inverse and reports competitive experimental performance.

1 INTRODUCTION

  • The method models an arbitrary data distribution by transforming a simple base random variable through invertible mappings, then uses the change-of-variables formula for likelihood estimation.The transformation is parameterized and optimized for maximum likelihood.Source: §1 · p. 1
  • High-dimensional normalizing flows face a Jacobian-determinant bottleneck, making tractable determinants a central design objective.For an arbitrary D×D matrix, determinant calculation has O complexity, as stated in the passage.Source: §1 · p. 1
  • The paper presents a new normalizing-flow transformation with an analytical inverse and reports competitive performance against existing state-of-the-art algorithms.The introduction describes experiments as indicating competitive performance.Source: §1 · p. 1

2 RELATED WORK

The related work organizes normalizing-flow approaches around coupling layers, autoregressive transformations, and other methods, comparing their architectures and computational trade-offs. It highlights challenges involving expressiveness, inversion, and Jacobian-determinant estimation.

2.1 Coupling Layer Transformations

  • Coupling layers split the input, leave one partition unchanged, and transform the other using parameters computed from the first, yielding a triangular Jacobian and requiring alternating masks.Alternating masks allow every dimension to be changed across consecutive transformations.Source: §2.1 · p. 2
  • NICE uses translation, Real NVP uses affine mappings, and Glow adds invertible 1 × 1 convolutions; affine-based flows may struggle with separated modes.Glow is reported to improve Real NVP for generative image modeling, while it may struggle with synthetic distributions having multiple separated modes.Source: §2.1 · p. 2

2.2 Autoregressive Transformations

  • Autoregressive flows transform each dimension conditioned on earlier dimensions, enabling efficient triangular-Jacobian computation; NAF and B-NAF use more expressive nonlinear transformations.B-NAFs are reported to match NAF performance with orders of magnitude fewer parameters, and universal density approximation is proved for both.Source: §2.2 · p. 2
  • NAF and B-NAF universal approximation establishes existence without convergence guarantees, and their inverses are not analytical; autoregressive inversion cannot be done in one pass.The paper therefore motivates coupling transformations that can compete with autoregressive flows while retaining easier inversion.Source: §2.2 · pp. 2–3

2.3 Other Methods

  • CNFs and FFJORD use ODE systems in place of the change-of-variables formula, while i-ResNets estimate the Jacobian determinant with a truncated power series that is biased.Sampling from CNF-based models requires solving an ODE system, which may slow generation.Source: §2.3 · p. 3
  • Residual Flows use a Russian roulette estimator for an unbiased Jacobian-determinant estimate and are reported to outperform i-ResNets, but residual-network inverses take around 5–20x longer than inference.The estimator randomizes the number of power-series terms and weights the terms used.Source: §2.3 · p. 3

3 PROPOSED APPROACH

Linear rational splines offer a more expressive alternative to affine coupling transformations while retaining an easy inverse. The approach constructs monotone splines, extends them beyond their finite interval, and uses them in coupling layers with a straightforward inverse.

3.1 Neural Spline Flows

  • Linear rational splines replace affine coupling transformations with piecewise functions that can be more expressive while avoiding polynomial-equation inversion.Earlier spline approaches used polynomial or rational quadratic functions whose inverses require solving degree-2 or degree-3 equations.Source: §3.1 · pp. 3–4

3.2 Monotonic Data Interpolation using Linear Rational Splines

  • Each bin is split at an intermediate point so two linear rational functions can satisfy endpoint value and derivative constraints while remaining continuous and monotone.Positive derivatives ensure monotonicity; continuity and derivative matching are enforced at the intermediate point.Source: §3.2 · p. 4

the Interval

  • A per-bin λ choice adds flexibility to the interpolated curve, while linear tails outside the finite spline interval cover unbounded data.The paper prefers linear tails to mapping data into [0, 1], which it notes can introduce numerical errors.Source: §3.2 · p. 4, the Interval · p. 5

3.3 Linear Rational Spline (LRS) Flows

  • The LRS flow predicts 4K − 1 parameters per dimension, applies the spline in coupling layers, and uses LU-decomposed 1 × 1 convolutions and multi-scale architecture for image generation.The inverse is analytical and has the same linear-rational-spline form as the forward transformation; unlike rational quadratic splines, LRS may use slightly more parameters, a difference that can be avoided by fixing λ.Source: §3.3 · pp. 5–6

4 SIMULATION RESULTS

Across synthetic density estimation, benchmark datasets, image generation, and VAE experiments, linear rational spline flows perform competitively, including against higher-order spline methods.

4 SIMULATION RESULTS

  • The simulations report competitive performance despite using a lower-order polynomial.This is the paper's overall characterization of the simulation results.Source: §4 · p. 6

4.1 Synthetic Density Estimation

  • For synthetic 2-d density estimation, LRS flows reconstruct the underlying distribution precisely and outperform Glow and i-ResNets; they also reconstruct a more complicated distribution accurately.Both synthetic experiments use two coupling layers constructed from linear rational splines.Source: §4.1 · p. 6, Figure 2 · p. 7, Figure 3 · p. 7

4.2 Density Estimation of Real-world Data

  • Linear rational spline flows perform competitively with rational quadratic Neural Spline Flows on benchmark density estimation despite using lower-degree polynomials.Table 1 reports test-set log-likelihoods for four UCI datasets and BSDS300.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Generative Modeling of Image Datasets

  • On benchmark image datasets, linear rational splines show competitive performance despite their simplicity, while residual flows perform better but sample more slowly.The reported sampling process is faster than residual flows by an order of magnitude; the linear rational spline inverse is analytic and has the same cost as the forward transformation.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 Variational Auto-encoders

  • In VAE simulations, the proposed model performs almost as well as Glow and rational quadratic splines.Table 3 reports results for MNIST and EMNIST.Source: §4.4 · p. 8, Table 3 · p. 8

5 CONCLUSION

The paper concludes that monotonic linear rational splines provide straightforward inversion and competitive performance on synthetic and real datasets. Its appendices describe forward and inverse derivative computations.

5 CONCLUSION

  • Monotonic linear rational splines invert straightforwardly without solving degree 2 or 3 polynomial equations and perform competitively with more complicated methods on synthetic and real datasets.Using the same function family for forward and inverse also makes their mathematical properties more straightforward to investigate.Source: §5 · p. 8

A.1 Derivative Computation

  • Dividing the derivative in Eq. (7) by δ(k) gives the derivative with respect to x, which never changes sign, even outside the interval.The passage identifies δ(k) as x(k+1) − x(k).Source: §A.1 · p. 11

A.2 Inverse Computation

  • Unlike rational quadratic splines, which require a degree-two polynomial root, linear rational splines have a closed-form inverse that is another linear rational spline with different parameters.The inverse first gives φ for each interval, then x = δ(k)φ + x(k) translates it into the interval.Source: §A.2 · p. 11

A.3 Inverse Derivative Computation

  • The inverse derivative describes the change with respect to φ in each interval; multiplying by δ(k) translates it to x, and its sign does not change outside 0 ≤ φ ≤ 1.Source: §A.3 · p. 11

B.1 Synthetic Density Estimation

The synthetic density-estimation experiments use linear rational spline coupling layers and include both finite-sample and infinite-generator settings.

B.1 Synthetic Density Estimation

  • For the Rings experiment, the model used 350,000 generated points, two coupling layers, 64 bins, and a tail bound of 5.Training used batches of 512, Adam at learning rate 0.0005, cosine annealing, and a 2-D standard normal base distribution.Source: §B.1 · p. 12
  • Figure 4 depicts density estimation on synthetic 2-D data from an infinite data generator after 50,000 training iterations.The generator supplies a different data set at each iteration.Source: §B.1 · p. 12, Figure 4 · p. 12
  • For the Figure 3 experiment, the setup used 1M samples, 1.5M iterations, and a uniform random variable as the base distribution.Source: §B.1 · p. 12

B.2 Density Estimation of Real-world Data

The real-data density-estimation experiments use configurations adapted from spline-flow benchmarks, with adjustments to optimization and regularization settings.

B.2 Density Estimation of Real-world Data

  • The real-data experiments use the configurations in Tables 4 and 5, with a residual network determining rational linear spline parameters.They use Adam and cosine annealing except for † cases, which use RAdam without annealing.Source: §B.2 · p. 12
  • The experiments vary the maximum gradient value from the fixed clipping used by Durkan et al. (2019b), and report that batch normalization and dropout were useful in some cases.Source: §B.2 · p. 12
  • Table 6 reports test-set log-likelihood in nats for four UCI datasets and BSDS300 under hyperparameters used for rational quadratic spline flows.Source: §B.2 · p. 12

B.3 Generative Modeling of Image Datasets

The appendix specifies configurations for image generative modeling and variational auto-encoders using linear rational spline transformations.

B.3 Generative Modeling of Image Datasets

  • Image generative modeling uses the multi-scale architecture of Dinh et al. (2017), with actnorm, a 1x1 convolution, and a coupling layer in each step.Linear rational splines form the coupling transformations, with a batch-normalized ResNet determining their parameters.Source: §B.3 · pp. 12–13
  • Tables 4 and 5 list hyperparameters for coupling and autoregressive linear rational spline transformations, while Table 7 lists configurations for Section 4.3 simulations.Source: Table 4 · p. 13, Table 5 · p. 13, Table 7 · p. 14
  • Table 6 gives test-set log-likelihood in nats for four UCI datasets and BSDS300 under rational quadratic spline hyperparameters.Source: Table 6 · p. 13

B.4 Variational Auto-Encoders

  • For the variational auto-encoders, a two-block ResNet parameterizes coupling or autoregressive splines; the latent dimension is 32 and the encoder computes 64 context features.The KL-divergence multiplier increases linearly from 0.5 to 1 over 10% of the training set; models train for 150k iterations with batches of 256.Source: §B.4 · pp. 13–14

C.1 Random Image Samples Generated by Models Trained for Section 4.3

The appendix shows random image samples from linear rational spline flows and randomly generated VAE samples using those flows as the prior and approximate posterior.

C.1 Random Image Samples Generated by Models Trained for Section 4.3

  • Figure 5 shows random image samples generated by trained linear rational spline flows.Source: Figure 5 · p. 15

C.2 Randomly Generated VAE Samples

  • Figure 6 shows randomly generated VAE samples with linear rational spline flows as the prior and approximate posterior.Source: Figure 6 · p. 16

D Further Simulation Results

On MNIST, linear rational splines outperform rational quadratic splines in log-likelihood despite using lower-degree polynomials; their result variability also declines with depth, with only a 0.23% parameter increase.

D Further Simulation Results

  • The MNIST experiments used eight seeds at each of five depths, selected the best flow model on validation data, and measured test-set log-likelihood in BPD.Hyperparameters are listed in Table 8.Source: D Further Simulation Results · p. 17, Table 8 · p. 17
  • Linear rational splines consistently outperform rational quadratic splines in MNIST test-set log-likelihood despite using lower-degree polynomials.Figure 7 compares the spline families for image generation on MNIST.Source: D Further Simulation Results · p. 17, Figure 7 · p. 18
  • Across depths, linear rational splines' standard deviation consistently decreases, while rational quadratic splines fluctuate and worsen at depth 32.The authors suggest higher-degree polynomials may require more numerical accuracy as depth increases; compositions of linear rational splines remain linear rational splines.Source: D Further Simulation Results · p. 17
  • The parameter increase for these simulations is 0.23%.Source: D Further Simulation Results · p. 17

Related papers

Loading 2001.05168v4…