Source-linked AI summary
Learning Activation Functions to Improve Deep Neural Networks
Forest Agostinelli, Matthew Hoffman, Peter Sadowski, Pierre Baldi
TL;DR
Fixed activation functions leave the broader space of useful nonlinearities underexplored. The paper learns a parametrized piecewise linear activation independently for each neuron with gradient descent, improving deep-network performance across benchmark tasks, including CIFAR-10 and CIFAR-100.
Problem
Activation functions strongly affect learning, but neural networks typically use pre-specified nonlinearities and the broader function space remains underexplored.
Method
The paper learns a parametrized piecewise linear activation independently for each neuron using gradient descent with the network weights.
Results
The adaptive activation improves deep-network performance across multiple benchmark tasks, including a 9.4% relative decrease in CIFAR-10 error rate.
Takeaways & Limitations
Neuron-specific learned activations can improve performance without significantly increasing parameter counts, while producing diverse activation functions.
Takeaways & Limitations
APL units cannot efficiently couple and summarize the outputs of multiple filters, unlike network-in-network architectures.
Abstract
from arXiv · showhide
Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.
1 INTRODUCTION
Deep neural networks commonly learn linear parameters while using pre-specified nonlinear activation functions. Because activation choice affects learning but the function space remains underexplored, the paper proposes independently learned adaptive activations.
- Common neural networks learn linear parameters while fixing nonlinearities such as logistic, tanh, rectified linear, or max-pooling functions.
- Rectified linear units accelerate deep-network training by avoiding sigmoidal saturation and alleviating weight-initialization and vanishing-gradient difficulties.
- Maxout computes the maximum of several linear functions, approximates any convex input function, and has achieved state-of-the-art performance on multiple benchmarks.
- Despite activation functions’ significant learning impact, the space of possible functions has hardly been explored.
- The paper introduces a parametrized piecewise linear activation learned independently for each neuron using gradient descent, representing convex and non-convex input functions.
2 ADAPTIVE PIECEWISE LINEAR UNITS
APL units learn a separate piecewise-linear activation for each neuron, combining hinge functions whose slopes and locations are optimized by gradient descent. They can represent non-convex functions and offer comparable expressive behavior to maxout and network-in-network with fewer parameters, while retaining different architectural trade-offs.
- APL activation units: APL activation functions are learned independently for each neuron as sums of hinge-shaped functions.The number of hinges S is fixed, while the slope and hinge-location parameters are learned during training.
- APL activation units: 2SM additional parameters are required for APL units across a network with M hidden units, and this is small relative to typical network weight counts.This parameter efficiency supports tuning different nonlinearities at individual feature-map locations.
- APL activation units: For large enough S, a single APL unit can approximate arbitrarily complex continuous functions, including non-convex functions.The representation assumes linear behavior in the far-right tail and constrained left-tail behavior.
- Comparison with other activation functions: Maxout can reproduce APL behavior by approximating its convex and concave components with separate maxout units, but standard maxout uses untied weight vectors.Implementing APL units with maxout would require O(SK) times as many parameters when K is the maxout input-vector size.
- Comparison with other activation functions: Network-in-Network can reproduce APL behavior only with aggressive weight tying, whereas it additionally couples and summarizes outputs from multiple filters.Replacing rectified linear units in an MLPConv layer with APL units is presented as a way to combine these capabilities.
3 EXPERIMENTS
Experiments evaluate adaptive piecewise linear activations on CIFAR-10, CIFAR-100, and Higgs boson decay benchmarks, including comparisons with fixed or alternative nonlinearities. Across these settings, learning activations improves performance and produces diverse functions across datasets and layers.
- 3.1 CIFAR: APL units improved the rectified-linear baseline by over 1% on CIFAR-10 and almost 3% on CIFAR-100.These corresponded to relative error-rate decreases of 9.4% and 7.5%, respectively.
- 3.1 CIFAR: APL units consistently outperformed leaky ReLU units across the tested CIFAR architectures and slope values.The comparison used multiple positive and negative k values, selected separately for each architecture and dataset.
- 3.2 HIGGS BOSON DECAY: A single APL network achieved state-of-the-art performance on the Higgs boson decay dataset, exceeding both a dropout-trained baseline and a five-network ensemble.The baseline used an eight-layer architecture, while the APL experiment used S = 2 and dropout in the top two hidden layers.
- 3.3 EFFECTS OF APL UNIT HYPERPARAMETERS: Learning activation functions produced better CIFAR-10 performance than freezing randomly initialized activations.The comparison was performed with S = 1 to distinguish learning from merely using complicated fixed activations.
- 3.4 VISUALIZATION AND ANALYSIS OF ADAPTIVE PIECEWISE LINEAR FUNCTIONS: Learned activations showed greater variance for CIFAR-100 than CIFAR-10 and greater variance for Higgs decay than CIFAR-100.For Higgs decay, activation variance decreased in higher layers.
4 CONCLUSION
The paper introduces neuron-specific piecewise linear activation functions learned by gradient descent, with learned activations differing across neurons and tasks.
- Each neuron computes an independent, piecewise linear activation function whose parameters are learned with the network weights.
- The learned activation functions form a diverse set, suggesting that using one activation function for every neuron may be suboptimal.
- CIFAR-100 sample plots contrast initialized activation functions with their final learned functions.
- Higgs→τ +τ − sample plots likewise contrast initialized activation functions with their final learned functions.