PaperlayerView exact revision on arXiv

Source-linked AI summary

Rational neural networks

Nicolas Boullé, Yuji Nakatsukasa, Alex Townsend

arXiv:2004.01902v2cs.NEcs.LGcs.NAmath.NAstat.ML

TL;DR

Choosing an activation function matters for neural-network performance and approximation, yet existing activations have limitations. This paper uses trainable rational activations and shows improved approximation over ReLU, with experiments suggesting practical promise.

  • Problem

    Choosing an activation function is important for network performance and approximation, while smooth and polynomial activations have identified limitations.

    Source: §1 · p. 1

  • Method

    The paper studies networks with trainable rational activations, composing low-degree rational functions to build high-degree functions.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Rational networks are theoretically shown to approximate smooth functions with fewer nodes and exponentially smaller depth than ReLU networks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present rational networks as an attractive alternative with smooth, trainable activations and simple implementation in existing architectures.

    Source: §5 · p. 8

    The authors do not claim type (3, 2) is best for every situation, since configurations may depend on the application.

    Source: §2 · p. 3

Abstract

from arXiv · show

We consider neural networks with rational activation functions. The choice of the nonlinear activation function in deep learning architectures is crucial and heavily impacts the performance of a neural network. We establish optimal bounds in terms of network complexity and prove that rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth. The flexibility and smoothness of rational activation functions make them an attractive alternative to ReLU, as we demonstrate with numerical experiments.

1 Introduction

Activation functions shape network approximation while node count and depth affect training cost. The introduction motivates rational networks as a theoretically grounded alternative to existing activations and ReLU.

1 Introduction

  • The activation choice matters because network training cost depends on node count and depth, while architecture design seeks fewer parameters without losing approximation power.Source: §1 · p. 1
  • Smooth activations can suffer vanishing gradients, while polynomial activations can be numerically unstable and approximate nonsmooth functions inefficiently.Source: §1 · p. 1
  • ReLU is fast and zero for many inputs, but its gradient is zero for negative inputs; proposed adaptations outperform it in image classification, and some have trainable parameters.Source: §1 · p. 1
  • Most exotic activation functions are motivated by empirical results rather than theoretical statements about improved approximation power over ReLU.Source: §1 · p. 2
  • Rational networks use trainable rational activations and are claimed to approximate smooth functions with fewer nodes and exponentially smaller depth than ReLU networks.The paper presents theoretical bounds and experiments motivated by the potential training cost of large networks.Source: §1 · p. 2

2 Rational neural networks

Rational networks compose low-degree trainable rational activations to obtain high-degree functions without a corresponding increase in trainable parameters. The paper motivates type (3, 2), while noting that activation choices may need to vary by application.

2 Rational neural networks

  • Prior rational-activation approaches face trade-offs: safe Padé Activation Units are nonsmooth with expensive gradients, while high-degree activations can increase parameter counts and training cost.The paper instead uses low-degree activations composed by the network.Source: §2 · p. 2
  • Composing low-degree rational activations builds high-degree rational functions while keeping training computationally efficient.For type (3, 2), overall degree grows as 3^#layers while trainable parameters grow linearly with network depth.Source: §2 · p. 2
  • Type (3, 2) is motivated by Zolotarev sign-function composition and allows nonconstant linear behavior at ±∞, unlike diagonal type (2, 2) or ReLU.Source: §2 · p. 2
  • The authors do not claim type (3, 2) is best for every situation; activation configurations may depend on the application.Source: §2 · p. 3
  • Initializing rational functions with a theory-motivated approximation to ReLU produces good performance and helps prevent arbitrarily large values.The paper says this initialization places the network near one with ReLU activations before rational functions adapt during training.Source: §2 · p. 3

3 Theoretical results on rational neural networks

The section gives complexity bounds for rational networks approximating ReLU networks and smooth functions. It shows that compositions of low-degree rational functions support efficient approximation with fewer parameters and smaller depth.

3.1 Approximation of ReLU networks by rational neural networks

  • Compositions of low-degree Zolotarev sign functions yield rational ReLU approximations with O(log(log(1/ϵ))) parameters.A composition of k type-(3, 2) functions has type (3k, 3k −1) but uses 7k parameters.Source: §3.1 · pp. 3–4
  • For a rational network with M layers and at most k nodes per layer, the construction uses O(kM log(log(M/ϵ))) parameters to approximate a ReLU network.The paper contrasts this with Telgarsky's O(kM log(M/ϵ)M) parameter construction.Source: §3.1 · p. 4
  • Rational networks approximate ReLU networks with size O(log(log(1/ϵ))), while ReLU networks approximate rational networks with size O(log(1/ϵ)^3).The rational size bound for ReLU approximation is optimal up to constants; a ReLU network also requires at least Ω(log(1/ϵ)) size to approximate x² on [−1, 1].Source: §3 · p. 3, §3.1 · pp. 4–5

3.2 Approximation of functions by rational networks

  • By Sobolev embedding [8], Fd,n contains Cn−1([0, 1]d) functions with Lipschitz first n −1 derivatives; Yarotsky [60, Theorem 1] constructed ReLU ϵ-approximations for Fd,n of size at most O(ϵ−d/n log(1/ϵ)) and depth smaller than O(log(1/ϵ)).Source: §3.2 · p. 5
  • Rational networks represent monomials with size independent of ϵ, whereas ReLU networks require O(log(1/ϵ)) parameters for them.ReLU networks exactly represent piecewise linear functions with constant parameters; rational networks approximate them with size at most O(log(log(1/ϵ))).Source: §3.2 · p. 5
  • O(ϵ^-d/n log(log(1/ϵ))) parameters for rational networks versus O(ϵ^-d/n log(1/ϵ)) for ReLU networks approximate functions in Fd,n.The rational construction also has depth O(log(log(1/ϵ))), exponentially smaller than the ReLU construction's depth.Source: §3.2 · pp. 5–6

4 Experiments using rational neural networks

The experiments use trainable rational activations, initialized near a rational approximation to ReLU, in function-approximation and GAN settings. On the KdV task, the rational network is more accurate than the ReLU and sinusoid networks; in the GAN experiment, it produces varied digits but also shows mode collapse.

4 Experiments using rational neural networks

  • The experiments use type (3, 2) rational activations, selected for empirical performance, and initialize them with the best rational approximation to ReLU.The coefficients are trainable; the initialization is intended to start near a ReLU-activation network and allow adaptation during training.Source: §4 · p. 6

4.1 Approximation of functions

  • The KdV comparison tracks validation loss across optimization steps for ReLU, sinusoid, rational, and polynomial activations.The rational network outperformed the sinusoid network despite the same asymptotic convergence rate, while the degree-3 polynomial network was harder to train.Source: §4.1 · p. 7, Figure 2 · p. 7
  • 5× greater accuracy than the sinusoid network and 20× greater accuracy than the ReLU network were reported for the rational network on the KdV solution task.The ReLU and rational networks had 8000 and 8035 trainable parameters, respectively.Source: §4.1 · p. 7

4.2 Generative adversarial networks

  • The rational GAN generated realistic digits with a broader range of features than the ReLU GAN, but its generated ones were identical, indicating mode collapse.The reference GAN used hyperparameters tuned for piecewise-linear activations, and the authors suggest exploring stabilization methods.Source: §4.2 · p. 8, Figure 3 · p. 8

5 Conclusions

The paper concludes that smooth, trainable rational activations have theoretical approximation advantages over ReLU networks and can be implemented in existing architectures. It identifies several application areas for future work, including PDE solving and discovery.

5 Conclusions

  • The authors conclude that rational networks have improved approximation power compared with ReLU networks.They also describe low-degree rational activations as a practical choice that can make training more computationally efficient.Source: §5 · p. 8
  • Rational networks are straightforward to implement in existing deep-learning architectures, including TensorFlow, with trainable activation functions.Source: §5 · p. 8
  • The authors identify image classification, time-series forecasting, GANs, and numerical PDE solution and discovery as future application areas.They suggest physics-informed neural networks could benefit in PDE-solver robustness and performance.Source: §5 · pp. 8–9

Broader Impact

The paper situates neural networks in societal applications and points to rational networks as a potential tool for PDE problems where mechanistic models are lacking.

Broader Impact

  • Neural networks are used in applications including facial recognition, credit-card fraud detection, speech recognition, and medical diagnosis.Source: Broader Impact · p. 9
  • The paper notes that growing understanding of neural-network approximation power adds theoretical justification to their use in societal applications.Source: Broader Impact · p. 9
  • The authors see potential for rational neural networks in discovering and solving PDEs where mechanistic models are lacking.Source: Broader Impact · p. 9

A.1 Deferred proofs of Section 3.1

The deferred proofs establish rational approximation bounds for ReLU and show how rational and ReLU networks can approximate one another. The bounds include a matching lower size requirement for rational networks approximating ReLU.

A.1 Deferred proofs of Section 3.1

  • The proof constructs a rational approximation to the absolute-value function using Zolotarev sign-function approximations, yielding square-root exponential convergence.The approximant is of rational type at most (k + 1, k).Source: §A.1 · pp. 11–12
  • A rational approximation to ReLU can be represented with size O(log(log(1/ε))), and any rational network approximating ReLU on [−1, 1] within ε requires size at least Ω(log(log(1/ε))).The lower bound is stated for rational neural networks measured by total number of nodes.Source: §A.1 · p. 13
  • A ReLU network approximates a rational function on [−1, 1] with size O(log(1/ε)^3).Source: §A.1 · p. 15
  • A rational network approximates a ReLU network with size O(Mk log(log(M/ε))) under the stated construction.The construction approximates ReLU activations to tolerance ε/M.Source: §A.1 · p. 15

A.2 Deferred proofs of Section 3.2

The deferred proofs establish rational-network representations for piecewise-linear functions and exact powers, then use these constructions in a local Taylor approximation of smooth functions. The construction combines partition-of-unity weights with rational approximations of its monomial and piecewise-linear terms.

A.2 Deferred proofs of Section 3.2

  • A piecewise-linear function with m breakpoints and Lipschitz constant L can be approximated by a rational network of size O(m log(log(L/ϵ))).The construction replaces the ReLU functions in an exact ReLU representation with rational approximations.Source: §A.2 · p. 16
  • A rational network represents x^n exactly with size at most 5⌊log_rP(n)⌋^2 + 1.The proposition allows rational activation functions of type (rP, rQ), where rP ≥ 2 and rQ ≥ 0.Source: §A.2 · p. 16
  • The smooth-function construction replaces local Taylor polynomials’ piecewise-linear functions and monomials with rational networks, using a partition of unity.The proof approximates the local Taylor construction with rational networks to obtain the stated approximation.Source: §A.2 · pp. 17–18
  • The local Taylor terms use monomials of degree n whose rational-network representation has size O(d log(n)^2), while the piecewise-linear weights are approximated to tolerance δ.The monomial factors are multiplied with rational networks; multiplication can be represented by one layer with three nodes.Source: §A.2 · pp. 17–18

A.3 Details of the approximation experiment

The approximation experiment trains a fully connected network to identify a KdV solution and compares activation functions. It also reports that rational networks of several types outperform ReLU networks in validation loss.

A.3 Details of the approximation experiment

  • The KdV solution was sampled into training and validation sets of 10^4 points each, with an approximately 10^5-point testing set.The testing data covered the stated space-time domain and were stored every 2,000 time steps.Source: §A.3 · p. 19
  • The KdV identifier used four hidden layers of 50 nodes, L-BFGS optimization, and 10,000 iterations to compare ReLU, sinusoid, rational, and degree-3 polynomial activations.The rational activations were type (3, 2) and initialized to approximate ReLU.Source: §A.3 · p. 19
  • Figure 5 presents approximation errors for networks using ReLU, sinusoid, and rational activation layers.The reported errors are between the different identification networks and the KdV solution.Source: Figure 5 · p. 19, §A.3 · p. 19
  • All tested rational activation types performed better than ReLU networks in the validation-loss comparison.The rational types were initialized to approximate ReLU; type (3, 2) is described as a trade-off between parameter count and approximation quality, not as best for every application.Source: §A.3 · p. 19

A.4 Details of the GAN experiment

The GAN experiment trains on MNIST with rational activations and compares generated images with ReLU-based outputs. Its figures document validation-loss curves for rational network types and generated samples.

A.4 Details of the GAN experiment

  • The GAN was trained on MNIST for 20 epochs with batch size 100 using Adam, with α = 0.0002 and β1 = 0.5.The experiment adapted the Keras example and used the cited hyperparameters.Source: §A.4 · p. 19
  • Figure 6 plots validation loss against epochs for rational networks of types (2, 2), (3, 2), (4, 3), and (5, 4).The caption identifies the activation types and the plotted quantities.Source: Figure 6 · p. 20
  • Figure 7 shows MNIST samples alongside images generated after 20 epochs by GANs with ReLU/Leaky ReLU and rational activation functions.The figure includes forty generated images and real images from MNIST.Source: §A.4 · p. 20, Figure 7 · p. 21
Loading 2004.01902v2…