PaperlayerView exact revision on arXiv

Source-linked AI summary

Unconstrained Monotonic Neural Networks

Antoine Wehenkel, Gilles Louppe

arXiv:1908.05164v3cs.LGcs.NEstat.ML

TL;DR

Existing monotonic architectures enable invertible flows but constrain transformation expressiveness. The paper introduces UMNNs, which enforce monotonicity through strictly positive derivatives and combine them into UMNN-MAF; experiments report competitive or state-of-the-art density-estimation and variational-inference performance.

  • Problem

    Monotonic invertible architectures constrain weights and activations, limiting the expressiveness of the resulting transformations.

  • Method

    UMNNs use free-form neural networks with strictly positive outputs to parameterize monotonic transformations, which are composed into the UMNN-MAF autoregressive flow.

  • Results

    UMNN-MAF achieves competitive or state-of-the-art results on normalizing-flow benchmarks and is applied to high-dimensional density estimation.

  • Takeaways & Limitations

    UMNNs provide invertible transformations without imposing constraints on the expressiveness of the hypothesis class and support density estimation and variational inference applications.

  • Takeaways & Limitations

    Static numerical integration can be inaccurate when the integrand is insufficiently smooth or too few integration steps are used.

Abstract

from arXiv · show

Monotonic neural networks have recently been proposed as a way to define invertible transformations. These transformations can be combined into powerful autoregressive flows that have been shown to be universal approximators of continuous probability distributions. Architectures that ensure monotonicity typically enforce constraints on weights and activation functions, which enables invertibility but leads to a cap on the expressiveness of the resulting transformations. In this work, we propose the Unconstrained Monotonic Neural Network (UMNN) architecture based on the insight that a function is monotonic as long as its derivative is strictly positive. In particular, this latter condition can be enforced with a free-form neural network whose only constraint is the positiveness of its output. We evaluate our new invertible building block within a new autoregressive flow (UMNN-MAF) and demonstrate its effectiveness on density estimation experiments. We also illustrate the ability of UMNNs to improve variational inference.

1 Introduction

The paper motivates unconstrained monotonic transformations by contrasting expressive limitations in constrained architectures with the need for effective reversible flows. It introduces UMNNs, which enforce monotonicity through a strictly positive derivative and support autoregressive flows.

  • Constrained weights and activations ensure invertibility but cap the expressiveness of monotonic transformations.
  • Autoregressive flows built from reversible transformations are effective for density estimation and posterior modeling, but the best transformation class remains unclear.
  • UMNNs enforce monotonicity by requiring a function's derivative to be strictly positive, using a free-form neural network whose output remains positive.
  • UMNN transformations are combined into UMNN-MAF, which achieves competitive or state-of-the-art normalizing-flow benchmark results and scales to high-dimensional density estimation.

2 Unconstrained monotonic neural networks

UMNNs parameterize strictly monotonic scalar functions through unconstrained positive derivative networks, evaluating them by numerical integration and inverting them with root-finding. Their backward formulation avoids memory growth with integration steps.

  • UMNNs learn strictly monotonic scalar functions by parameterizing their derivative with a strictly positive neural network output.
  • The positive derivative network can be arbitrarily complex because positivity is imposed only on its output using an ELU activation increased by 1.
  • Forward evaluation computes the integrated transformation numerically, with Clenshaw-Curtis quadrature providing an efficient approximation.
  • Gradients with respect to network parameters are obtained through the Leibniz integral rule rather than backpropagating through every integration step.
  • The backward pass has memory usage independent of the number of integration steps and therefore independent of the desired integration accuracy.
  • Because the learned function is strictly monotonic, its inverse is unique and can be computed efficiently with root-finding methods such as bisection.

3 UMNN autoregressive models

UMNN-MAF builds autoregressive invertible transformations by replacing constrained scalar mappings with UMNNs conditioned on preceding variables. Its triangular structure enables efficient Jacobian computation, while numerical integration and root-finding support evaluation and sampling.

  • Normalizing flows: Normalizing flows compose invertible transformations to map data to latent variables for density estimation and posterior modeling.The induced density is trained by maximizing data likelihood, and flows can also transform a base distribution into a variational posterior.
  • Autoregressive transformations: Autoregressive transformations make the Jacobian lower triangular, reducing determinant computation to O(d) and requiring each scalar component to be bijective.Each component depends on the current and preceding input variables, enabling tractable change-of-variables calculations.
  • UMNN-MAF construction: UMNN-MAF uses UMNN scalar transformations conditioned on embeddings of preceding variables, with masked autoregressive networks parameterizing the embeddings.The architecture applies a separate conditioning embedding h_i(x1:i−1) to each scalar transformation and combines all parameters into one autoregressive map.
  • Log-density: UMNN-MAF yields an exact Jacobian expression computable efficiently with a single forward pass, unlike FFJORD’s numerical Jacobian and transformation computation.The change-of-variables expression follows from the positive derivative of each UMNN component.
  • Evaluation and sampling: Forward evaluation uses numerical integration, while inversion uses common root-finding algorithms because arbitrary monotonic UMNN functions lack analytic inverses.The implementation uses Clenshaw-Curtis quadrature, and bisection was fast enough for inversion experiments.
  • Universality: UMNN-MAF is argued to be a universal density approximator because UMNNs can represent continuously differentiable monotonic functions and inverse cumulative distribution functions.The universality argument relies on the inverse sampling theorem and approximation of positive continuous derivatives.

4 Related work

UMNN-MAF is positioned among monotonic autoregressive flows, coupling-based flows, and neural ODE approaches. Its distinguishing design uses free-form neural networks with only positive outputs, while toy experiments show accurate density capture and numerical sampling.

  • Monotonic autoregressive flows: NAF and B-NAF use strictly monotonic transformations, whereas UMNN-MAF imposes only output positiveness on free-form neural networks.This design is presented as enabling arbitrary state-of-the-art neural architectures, simpler initialization, and lighter Jacobian evaluation.
  • Autoregressive flow context: Autoregressive flows extend masked-network approaches to real-valued density estimation and variational posterior representation, while Sum-of-Squares flows integrate autoregressive polynomial functions.These methods form the broader family of reversible transformations to which UMNN-MAF belongs.
  • Coupling-based flows: Coupling-layer flows split the input vector and use unconstrained functions, with RealNVP and Glow extending the approach through multiscale architectures and invertible 1x1 convolutions.UMNNs are used here in autoregressive architectures but could also replace linear transformations in related flow designs.
  • Neural ODE connection: UMNN transformations resemble structured neural ordinary differential equations whose separable vector-field dynamics can be solved by direct integration.This contrasts with NODE-based continuous flows such as FFJORD, which use a free-form Jacobian.
  • Toy experiments: UMNN-MAF precisely captures multimodal and discontinuous toy densities, and numerical inversion produces good samples despite the absence of analytic inversion.Figure 2 compares empirical samples, learned densities, and samples obtained by numerical inversion.

5 Experiments

UMNN-MAF is evaluated on toy distributions, tabular and image density estimation, and variational auto-encoders. It achieves competitive density-estimation results, scales to high-dimensional MNIST with fewer parameters than NAF, and slightly improves VAE performance on some datasets.

  • 5.1 2D toy problems: UMNN-MAF learns changes of variables that warp an isotropic Gaussian into multimodal or discontinuous two-dimensional distributions and produces good samples through numerical inversion.The flow uses a single transformation for these toy tasks.
  • 5.2 Density estimation: On MNIST, UMNN-MAF slightly outperforms other models without using a multi-scale architecture.MNIST has dimension d = 784, and NAF and B-NAF report no results for this benchmark, presumably because of memory explosion.
  • 5.2 Density estimation: In high-dimensional datasets, UMNN-MAF requires fewer parameters than NAF and a similar number to B-NAF while achieving similar or better performance.The parameter comparison is reported in Table 2.
  • 5.3 Variational auto-encoders: UMNN-MAF slightly outperforms classical VAE and planar flows on MNIST and Omniglot, while IAF, B-NAF, and UMNN-MAF achieve similar results on Freyfaces.FFJORD is reported as the best among the compared methods, with differences partly attributed to distinct amortization roles.

6 Discussion and summary

UMNN provides expressive invertible transformations with efficient evaluation and numerical inversion, while retaining limitations from quadrature accuracy and inversion cost. Composed into UMNN-MAF, it achieves competitive or state-of-the-art density-estimation and variational-inference results, including high-dimensional MNIST.

  • Limitations: Static quadrature can become inaccurate for insufficient integration steps or nonsmooth integrands, although controlled Lipschitz constants below 1000 generally permit fewer than 100 steps.Dynamic quadrature is proposed as an alternative, and further work also targets faster integration and broader architectures.
  • Limitations: UMNN inversion is numerical rather than analytic, although nested Clenshaw-Curtis evaluations can reuse previous evaluations efficiently.A learned inverse through distillation is identified as an alternative when numerical inversion remains costly.
  • Efficiency: The Jacobian requires a single forward neural-network evaluation, making UMNN-MAF particularly suitable for density estimation.Together with the Leibniz integral rule, the log-likelihood derivative has memory requirements comparable to ordinary supervised learning.
  • Summary: UMNN uses free-form neural networks to preserve expressiveness while guaranteeing monotonicity and invertibility.The transformation's monotonicity is enforced through a strictly positive derivative rather than restrictive weight or activation constraints.
  • Summary: UMNN-MAF composes these transformations into an autoregressive flow with competitive or state-of-the-art density-estimation and variational-inference results.This extends the scalar transformation into a flow suitable for the paper's benchmark evaluations.
  • Results: UMNN was successfully applied to high-dimensional MNIST density estimation and performed better than classical monotonic approaches.The paper presents this as the first successful monotonic transformation application to high-dimensional density estimation.

A.1 Density estimation and toy problems hyperparameters

The density-estimation and toy-problem experiments used configured architectures selected through preliminary comparisons across integrand-network depths and widths.

  • Training configurations: Training configurations covered 2D toy problems and five tabular datasets.The best tabular architecture was selected after preliminary experiments, whereas this selection was not needed for the 2D toy problems.
  • Architecture search: Integrand networks were tested with 3 or 4 hidden layers and widths of 50, 100, 150, or 200.Embedding networks used the best-performing MADE architecture from NAF.

A.2 Variational auto-encoders

The variational-autoencoder experiments used a fixed encoder and embedding-network setup, with a small search over integrand-network depth and width.

  • VAE configuration: The variational-autoencoder configurations used 320 encoder outputs, a 64-dimensional noise vector, and a 1280-unit single-hidden-layer embedding network.The encoder outputs and noise vector were supplied to the embedding network.
  • Architecture search: The integrand-network search varied hidden-layer counts between 3 and 4 and widths between 100 and 150.This was described as a small grid search over the integrand architecture.

B Clenshaw-Curtis module

The Clenshaw-Curtis module numerically integrates a differentiable scalar function over tensor-valued bounds and embeddings, then computes gradients through the same quadrature evaluations.

  • Forward pass: The forward module takes scalar upper integration bounds x, vector embeddings h, a differentiable function f, and an integration-step count N.It returns tensor-valued integral estimates for the batch.
  • Forward pass: Clenshaw-Curtis weights and evaluation points are computed before the module evaluates f and accumulates weighted function values.The quadrature loop runs over the N integration steps.
  • Backward pass: The backward module reuses quadrature evaluations to compute gradients with respect to inputs, embeddings, and function parameters.It receives the loss gradient with respect to the output integral and accumulates gradients across integration points and batch samples.
  • Backward pass: Gradients with respect to the superior integration bound are obtained by evaluating f at that bound and multiplying by the output gradient.The backward procedure returns gradients for x, the parameters, and h.

C Generated images from MNIST

UMNN-MAF generates class-conditional and unconditional MNIST samples across increasing sampling temperatures. The unconditional model requires sufficient noise to produce heterogeneous digit samples.

  • Sampling temperature increases from 0.1 in the top rows to 1.0 in the bottom rows.The figure shows both class-conditional and unconditional MNIST samples across this range.
  • At low noise, the unconditional model fails to incorporate digit structure.This behavior occurs at the smallest sampling temperatures in sub-figure (b).
  • With sufficient noise, the unconditional model generates random digits with high heterogeneity.Higher sampling temperatures correspond to greater noise during inversion.
  • The class-conditional model generates digit images organized by class columns across temperatures.Sub-figure (a) contains labeled samples, with columns corresponding to different classes.
Loading 1908.05164v3…