Source-linked AI summary
Pad\'e Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks
Alejandro Molina, Patrick Schramowski, Kristian Kersting
TL;DR
Deep networks commonly rely on activation functions chosen before training, although the best choice depends on the architecture and dataset. The paper introduces trainable rational-function PAUs, reporting competitive performance across activation functions and architectures; it identifies robustness and other training directions for future work.
Problem
Activation functions are typically fixed by hand, and reliance on choosing fixed, hand-engineered activations is a concern addressed by the paper.
Source: §5 · p. 9Method
Results
Across activation functions and architectures, Padé networks are among the top-performing networks and are competitive with state-of-the-art networks using fixed and learned activations.
Source: §5 · p. 9Takeaways & Limitations
PAUs provide a compact learned-activation option; the experiments use 10 parameters per layer and report high predictive performance.
Source: §4.6 · p. 9Exploring the safe–unsafe PAU space, globally optimal training, and provable robustness are left for future work.
Source: §5 · p. 9
Abstract
from arXiv · showhide
The performance of deep network learning strongly depends on the choice of the non-linear activation function associated with each neuron. However, deciding on the best activation is non-trivial, and the choice depends on the architecture, hyper-parameters, and even on the dataset. Typically these activations are fixed by hand before training. Here, we demonstrate how to eliminate the reliance on first picking fixed activation functions by using flexible parametric rational functions instead. The resulting Pad\'e Activation Units (PAUs) can both approximate common activation functions and also learn new ones while providing compact representations. Our empirical evidence shows that end-to-end learning deep networks with PAUs can increase the predictive performance. Moreover, PAUs pave the way to approximations with provable robustness. https://github.com/ml-research/pau
1 INTRODUCTION
Activation functions affect deep-network training and predictive performance, yet choosing an effective one is difficult and often depends on the dataset and architecture. The paper proposes learnable rational-function activations as an alternative to hand-fixed choices.
1 INTRODUCTION
- Activation functions influence training time and predictive performance, while the best choice can depend on the dataset and architecture.Swish was identified empirically as a good candidate, but its behavior is not guaranteed for every dataset and its search algorithm is computationally demanding.Source: §1 · p. 1
- Existing learnable activations offer flexibility, but piecewise-linear methods can add many parameters and polynomial methods have limitations such as exploding values and oscillation.Hyperactivations can also be frustrating to implement for general activation networks.Source: §1 · p. 1
- PAUs use rational functions with trainable coefficients, enabling end-to-end learning and reported increases in deep networks’ predictive performance.Figure 1 illustrates approximations of ReLU, Sigmoid, Tanh, Swish, and Leaky ReLU using PAUs.Source: §1 · p. 2, Figure 1 · p. 2
2 PADÉ ACTIVATION UNITS (PAU)
The section motivates activation functions that support approximation, stable learning, and parsimony, then introduces trainable Padé rational functions as PAUs. It describes a safe denominator constraint and gradient-based learning for integrating PAUs into neural networks.
2 PADÉ ACTIVATION UNITS (PAU)
- The authors frame activation functions around five aims: universal approximation, improved gradients, stability, parameter parsimony, and high predictive performance.They report that rational functions meet the approximation and parsimony aims, while experiments support the other three.Source: §2 · p. 2, Table 1 · p. 2
2.1 PADÉ APPROXIMATION OF ACTIVATION FUNCTIONS
- Padé approximants represent functions as rational functions, and PAUs make their coefficients trainable so each layer can learn its needed activation end-to-end.The paper notes that rational functions can approximate typical activation functions well.Source: §2.1 · p. 3
- Safe PAUs constrain the denominator to remain at least 1, preventing poles and enabling safe computation over the real numbers.The authors use q = 1; they note that values q ∈ (0, 1) might offer gradient amplification but leave them for future work.Source: §2.1 · p. 3
2.2 LEARNING SAFE PADÉ APPROXIMATIONS USING BACKPROPAGATION
- PAUs are learned by backpropagation and stochastic gradient descent, with manually computed gradients available for a more efficient CUDA implementation.The implementation reuses expressions to reduce computation and defines sign handling to avoid division by zero in gradient calculations.Source: §2.2 · p. 3
3 PADÉ NETWORKS
Padé networks use learnable PAUs, sharing one PAU per layer to limit added parameters, and are universal approximators. The section also motivates sparse and randomized variants and reports PAU advantages over baseline activations on Fashion-MNIST with LeNet.
3 PADÉ NETWORKS
- Padé networks replace standard activations with PAUs; learning one PAU per layer adds φ = 10L parameters in the experiments.Padé networks may include convolutional and residual architectures with pooling layers.Source: §3 · pp. 3–4
3.1 PADÉ NETWORKS ARE UNIVERSAL FUNCTION APPROXIMATORS
- Theorem 1 states that networks using a PAU activation are dense in C(K) for compact K; the proof also applies to safe PAUs and gives density in C(K; R^m).Source: §3.1 · p. 4
3.2 SPARSE PADÉ NETWORKS AND RANDOMIZED PAUS
- Because Padé networks can ε-approximate ReLU networks, the authors expect lottery-ticket pruning to find smaller, well-performing Padé networks.They present this as an expectation, not an empirical result.Source: §3.2 · p. 4
- Since the virtual network inside a PAU is inaccessible to standard Dropout, the authors propose randomized PAUs that add coefficient noise during training to resist overfitting.Noise is added to each coefficient for each input.Source: §3.2 · pp. 4–5
- Over five Fashion-MNIST runs with LeNet, PAU outperforms all baseline activations in mean test accuracy and achieves lower mean train loss.The figure compares mean test accuracy and mean train loss.Source: Figure 2 · p. 5
4 EXPERIMENTAL EVALUATION
Across MNIST, Fashion-MNIST, CIFAR-10, and ImageNet, the experiments compare PAUs with fixed and learnable activation functions across several network architectures. The results report strong predictive performance, rapid learning, and a small parameter cost for PAUs.
4.1 EMPIRICAL RESULTS ON MNIST AND FASHION-MNIST BENCHMARKS
- PAU outperformed baseline activations on average across every tested network, matched or exceeded the best MNIST baseline, and used 50 parameters in VGG-8 and 40 in LeNet.On Fashion-MNIST, PAU provided the best results for both architectures; the authors also report lower training loss than all baselines and greater stability than SLAF.Source: §4.1 · pp. 5–6, Table 2 · p. 6
4.2 LEARNED ACTIVATION FUNCTIONS ON MNIST AND FASHION-MNIST
- Learned PAUs were flexible yet resembled standard activations; some appeared to be smoothed V-shaped Leaky ReLUs, consistent with their initialization.The authors leave alternative initialization and optimization settings for future work; SLAF produced undefined values in most reported runs.Source: §4.2 · p. 6, Figure 3 · p. 7
4.3 EMPIRICAL RESULTS ON CIFAR-10
- On CIFAR-10, learnable activations helped VGG-8 and MobileNetV2, while PAU and RPAU were often leading or close to the best; PAU gave the top ResNet101 performance.The authors note that activation flexibility had less impact on the larger ResNet101 and DenseNet121 models, though their performance remained competitive and stable.Source: §4.3 · pp. 6–7, Table 3 · p. 7
4.4 FINDING SPARSE PADÉ NETWORKS
- PAU remained above the compared activations under increased pruning pressure, and a ResNet101 subnetwork pruned by 30% outperformed all pruned and unpruned networks.The pruning experiments targeted convolutional layers; the authors discarded Maxout because its added parameters conflicted with the goal of pruning.Source: §4.4 · pp. 7–8, Figure 4 · p. 8
4.5 EMPIRICAL RESULTS ON IMAGENET
- On ImageNet with MobileNetV2, PAU led in top-1 accuracy and Swish in top-5, while both learned faster than the other tested activations.The authors also interpret PAU's rapid learning across experiments as evidence that it does not exhibit vanishing-gradient issues.Source: §4.5 · p. 8, Figure 5 · p. 8, Table 4 · p. 9
4.6 SUMMARIZED RESULTS
- Across all datasets and architectures, PAU and RPAU are competitive against the tested activation functions; the experiments aggregate counts of wins and losses rather than one pooled score.The summary reports 10 added parameters per layer and characterizes the networks as fast- and stably-learning, parsimonious, and high-performing.Source: §4.6 · p. 9, Table 5 · p. 9
5 CONCLUSIONS
PAUs represent activations as rational functions trained end-to-end, and image-classification results place Padé networks among the top performers across activation functions and architectures. The authors identify globally optimal training and provable robustness as future directions.
5 CONCLUSIONS
- Across activation functions and architectures, Padé networks are among the top-performing networks in image classification, competitive with state-of-the-art fixed and learned activations.The authors also report that PAUs learn new activation functions and novel networks.Source: §5 · p. 9
- PAUs encode activation functions as rational functions trainable end-to-end using backpropagation, making them replaceable within neural networks.Source: §5 · p. 9
- The results provide empirical evidence that rational functions can be used to design algorithms for training common deep architectures, answering an open question affirmatively.Source: §5 · p. 9
- Future work should explore safe and unsafe PAUs, globally optimal training, and provable robustness for Padé approximations of general deep networks.The authors note that Padé networks can be reduced to ReLU networks.Source: §5 · p. 9
A.1 INITIALIZATION COEFFICIENTS
The appendix describes how initial PAU coefficients approximate known activations, using different coefficient-construction procedures for different activation functions.
A.1 INITIALIZATION COEFFICIENTS
- For Sigmoid, Tanh, and Swish, the Padé approximants use predefined orders [5,4] and standard techniques.Source: §A.1 · p. 13
- For PRelu, LeakyRelu, and Relu, coefficients are optimized by least squares over [-3,3] in steps of 0.000001.Source: §A.1 · p. 13
- Table 6 lists initial coefficients for PAU approximations of different activation functions.Source: Table 6 · p. 13
A.2 LIST OF ACTIVATION FUNCTIONS
The appendix lists activation functions used for comparison, spanning fixed nonlinearities and learnable activation designs.
A.2 LIST OF ACTIVATION FUNCTIONS
- The appendix presents these functions and their respective parameters as the activation-function comparison set.Source: §A.2 · p. 13
- The comparison list includes fixed ReLU variants and nonlinearities, including ReLU6, Leaky ReLU, Random ReLU, ELU, CELU, and Swish.Source: §A.2 · p. 13
- Learnable alternatives include PReLU, Maxout, mixtures of activations, SLAF, APL, and SReLU.Source: §A.2 · p. 13
A.3.1 NETWORK ARCHITECTURES
The appendix describes VGG and LeNet architectures and reports their parameter counts, including the parameters used by PAU activations.
A.3.1 NETWORK ARCHITECTURES
- The appendix describes the VGG and LeNet architectures, with Table 7 captioned as the architecture of Simple Convolutional Neural Networks.Source: §A.3.1 · p. 13, Table 7 · p. 14
- VGG has 9,224,508 total parameters, including 50 for PAU.Source: §A.3.1 · p. 14
- LeNet has 61,746 total parameters, including 40 for PAU.Source: §A.3.1 · p. 14
A.3.2 LEARNING PARAMETERS
Across MNIST and Fashion-MNIST, PAUs match or outperform baseline activations, with reported gains in training loss and training time. The appendix also specifies training setups, network parameter counts, pruning procedures, and a CIFAR-10 accuracy-and-loss comparison.
A.3.2 LEARNING PARAMETERS
- For the reported experiments, networks trained for 100 epochs with Adam or SGD, batch size 256, and PAU coefficients initialized using Leaky ReLU constants.The authors report test accuracy and training loss after each epoch as means over five runs.Source: §A.3.2 · p. 14
A.3.3 PREDICTIVE PERFORMANCE
- On MNIST, PAUs match or outperform the best baseline test-accuracy performances and achieve lower training loss than all baselines.Figure 6 compares mean test accuracy and mean train loss over five runs using VGG and LeNet.Source: Figure 6 · p. 15
- On Fashion-MNIST, PAUs outperform baseline activations in performance and training time, especially with VGG.Figure 7 compares mean test accuracy and mean train loss over five runs using VGG and LeNet.Source: Figure 7 · p. 15
A.4.1 LEARNING PARAMETERS
- CIFAR-10 networks trained for 400 epochs with SGD and batch size 64; Randomized PAU noise was 1% for VGG8 and MobileNetV2 and 10% for ResNet101.PAU layers used constant learning rates and no weight decay, while the rest of the network used a decaying learning rate and weight decay.Source: §A.4.1 · p. 16
- For ImageNet, PAU and the rest of the network shared PyTorch’s default optimizer setup, with a 0.1 initial learning rate reduced by 10% after epochs 30, 60, and 90.Source: §A.4.1 · p. 16
A.4.2 NETWORK ARCHITECTURES
- PAUs add 50 parameters to VGG8, 360 to MobileNetV2, 1,000 to ResNet101, and 1,200 to DenseNet121, while Maxout substantially enlarges the networks.Source: §A.4.2 · p. 16
A.4.3 PRUNING EXPERIMENT
- The pruning experiment compares PAU networks with architecture-specific best-activation networks across pruning rates of 10%–60%, removing the lowest-weight-sum filters and retraining after each pruning step.Maxout is excluded because its parameter increase would defeat the purpose of pruning.Source: §A.4.3 · p. 16
A.4.4 PREDICTIVE PERFORMANCE CIFAR10
- Figure 8 compares accuracy and loss for PAU and baseline activation units on CIFAR-10.Accuracy is shown in the left column and loss in the right column.Source: Figure 8 · p. 17