Source-linked AI summary
Continuously Differentiable Exponential Linear Units
Jonathan T. Barron
TL;DR
The paper addresses the discontinuity and potentially large gradients of the original ELU when α ≠ 1. It introduces CELU, an alternative parametrization that is continuously differentiable and interpolates between ReLU and a linear activation as α varies.
Problem
The original ELU derivative is discontinuous at x = 0 when α ≠ 1, and large α can produce large gradients for small negative x.
Method
CELU modifies ELU's negative branch so the derivative at x = 0 equals 1 for every α, with derivatives computed efficiently using precomputed exp.
Results
CELU is scale-similar in x and α and converges to ReLU as α approaches 0 from the right and to a linear activation as α approaches ∞.
Takeaways & Limitations
CELU provides an interpretation of α as a parameter that interpolates between ReLU and a linear function.
Abstract
from arXiv · showhide
Exponential Linear Units (ELUs) are a useful rectifier for constructing deep learning architectures, as they may speed up and otherwise improve learning by virtue of not have vanishing gradients and by having mean activations near zero. However, the ELU activation as parametrized in [1] is not continuously differentiable with respect to its input when the shape parameter alpha is not equal to 1. We present an alternative parametrization which is C1 continuous for all values of alpha, making the rectifier easier to reason about and making alpha easier to tune. This alternative parametrization has several other useful properties that the original parametrization of ELU does not: 1) its derivative with respect to x is bounded, 2) it contains both the linear transfer function and ReLU as special cases, and 3) it is scale-similar with respect to alpha.