Source-linked AI summary
Learned-Norm Pooling for Deep Feedforward and Recurrent Neural Networks
Caglar Gulcehre, Kyunghyun Cho, Razvan Pascanu, Yoshua Bengio
TL;DR
The paper addresses whether pooling-based nonlinear units can provide flexible representations beyond fixed pooling and maxout operations. It proposes learned-order Lp units, interprets their geometry, and reports competitive performance across feedforward and recurrent evaluations.
Problem
Fixed pooling operators use predefined forms, motivating a more flexible nonlinear unit whose order can be learned.
Method
The paper uses Lp units that compute normalized Lp norms over projected lower-layer signals and learn their orders within deep feedforward and recurrent networks.
Results
Across four feedforward benchmarks and polyphonic music prediction, experiments confirmed competitive generalization and dataset-dependent, non-degenerate distributions of learned orders.
Takeaways & Limitations
Combining curved boundaries from Lp units with different learned orders can efficiently model separating curves with non-stationary curvature.
Takeaways & Limitations
The geometrical analysis notes that superelliptic boundaries may be degenerate along some axes when the number of filters is smaller than the input dimension.
Abstract
from arXiv · showhide
In this paper we propose and investigate a novel nonlinear unit, called $L_p$ unit, for deep neural networks. The proposed $L_p$ unit receives signals from several projections of a subset of units in the layer below and computes a normalized $L_p$ norm. We notice two interesting interpretations of the $L_p$ unit. First, the proposed unit can be understood as a generalization of a number of conventional pooling operators such as average, root-mean-square and max pooling widely used in, for instance, convolutional neural networks (CNN), HMAX models and neocognitrons. Furthermore, the $L_p$ unit is, to a certain degree, similar to the recently proposed maxout unit (Goodfellow et al., 2013) which achieved the state-of-the-art object recognition results on a number of benchmark datasets. Secondly, we provide a geometrical interpretation of the activation function based on which we argue that the $L_p$ unit is more efficient at representing complex, nonlinear separating boundaries. Each $L_p$ unit defines a superelliptic boundary, with its exact shape defined by the order $p$. We claim that this makes it possible to model arbitrarily shaped, curved boundaries more efficiently by combining a few $L_p$ units of different orders. This insight justifies the need for learning different orders for each unit in the model. We empirically evaluate the proposed $L_p$ units on a number of datasets and show that multilayer perceptrons (MLP) consisting of the $L_p$ units achieve the state-of-the-art results on a number of benchmark datasets. Furthermore, we evaluate the proposed $L_p$ unit on the recently proposed deep recurrent neural networks (RNN).
1 Introduction
The paper motivates learned nonlinear activation functions by generalizing pooling and maxout, arguing that learned Lp orders can represent curved decision boundaries efficiently. It proposes evaluating Lp units in feedforward and recurrent neural networks.
- Pooling operators summarize groups of neural responses and can provide invariance to input variations across pooled filter outputs.
- Maxout treats pooling as a piecewise linear nonlinear activation and has achieved state-of-the-art results on several benchmark datasets.
- The proposed Lp unit generalizes pooling and maxout by computing a normalized Lp norm over filter outputs while learning the order p.
- Different learned orders can combine to model highly curved separating boundaries more efficiently than piecewise-linear rectifier boundaries.
- The paper evaluates Lp units in MLPs and recurrent neural networks across object-recognition and related benchmark tasks.
2 Background
The background frames an MLP as layered nonlinear neurons and explains how pooling can serve as a groupwise nonlinear activation rather than an element-wise operation.
- An MLP is a feedforward network composed of multiple layers of nonlinear neurons.
- Each conventional MLP neuron applies a nonlinear activation function to a weighted sum of incoming signals.
- Pooling reduces high-dimensional convolutional outputs and can provide local translation invariance when applied to spatially neighboring neurons.
- Viewed as an activation function, pooling operates on groups of hidden units and returns a scalar rather than applying element-wise.
- Maxout is a representative example that uses max pooling as a nonlinear activation function in this groupwise setting.
3 Lp Unit
The Lp unit replaces maxout’s maximum with a normalized Lp norm over projected lower-layer signals, while learning its order and related parameters. Its order recovers several pooling behaviors and distinguishes the unit from fixed-order alternatives.
- The Lp unit replaces the max operator in maxout with an Lp norm to create a more general pooling-rooted activation function.
- A normalized Lp norm is computed from a finite set of input signals, with the unit’s order allowed to differ across neurons.
- Each unit receives private groups of linear projections from the layer below, and its parameters, including the order, are estimated by backpropagation.
- For nonnegative inputs, p_j=1 yields average pooling and p_j=2 recovers root-of-mean-squared pooling.
- As p_j approaches infinity, the Lp norm becomes the maximum absolute input; with nonnegative inputs, this corresponds to maxout.
- The proposed unit learns orders rather than fixing them, with the authors conjecturing that order distributions vary across datasets and units.
4 Geometrical Interpretation
The paper interprets each Lp unit geometrically as defining a superelliptic boundary whose shape depends on p. Combining units with different learned orders can represent decision boundaries with changing curvature efficiently.
- Geometrical Interpretation: An Lp unit computes the p-th norm of an input projection onto a learned subspace.The projected vectors may be linearly dependent, producing a subspace of dimensionality k ≤ N.
- Geometrical Interpretation: Each Lp unit partitions input space into inside and outside regions bounded by a convex superellipse whose shape varies with p.For p ≥ 1, the projected shape remains convex; dependent bases can make the superellipse degenerate.
- Two Classes, Single Lp Unit: In the single-unit experiments, Lp superellipses correctly separated classes, including the p = ∞ rectangular case.The figures visualize class samples, learned separating curves, subspace axes, and superellipse shapes.
- Geometrical Interpretation: Different orders produce distinct geometric components, including an L2 superellipse and an L∞ rectangle, whose combination yields more non-trivial curvature.The experiments use mixtures of Gaussian classes and compare separating curves formed from fixed or different orders.
- Decision Boundary with Non-Stationary Curvature: Representational: Three Lp units achieved zero training error on all ten random runs, using only three units or six filters, on the non-stationary-curvature task.The Lp models outperformed L2, maxout, rectifier, and sigmoid models in representing this specific curve.
- Decision Boundary with Non-Stationary Curvature: Representational: The learned Lp model made 0 mistakes, whereas the illustrated four-unit rectifier model made 64 mistakes.The rectifier composed the boundary from linear segments, while the Lp units learned orders suited to the changing curvature.
5 Application to Recurrent Neural Networks
The paper embeds Lp units in deep-transition recurrent networks while retaining a saturating nonlinearity around the transition. This design permits non-saturating intermediate activations without losing bounded hidden states, and its expected benefit is evaluated in a DOT-RNN.
- Application to Recurrent Neural Networks: A saturating tanh nonlinearity bounds the hidden state, allowing the intermediate function f to use non-saturating activations such as Lp units.The paper places an Lp layer in the role of f within the deep-transition architecture.
- Application to Recurrent Neural Networks: A deep-transition RNN inserts one or more intermediate layers between consecutive hidden states.The transition maps the previous hidden state and current input to the next hidden state.
- Application to Recurrent Neural Networks: The paper evaluates this idea by training a deep-output, deep-transition RNN with the proposed Lp units.The motivation is that benefits for highly nonlinear hidden-state updates should transfer to deep recurrent networks.
6 Experiments
Experiments tested learned-order Lp units in densely connected MLPs across benchmark datasets and in deep recurrent networks. The results support dataset-dependent, heterogeneous orders and competitive generalization, including state-of-the-art outcomes on several tasks.
- Experimental setup: The experiments evaluated Lp units in densely connected feedforward networks and deep recurrent networks across four benchmark datasets and three music datasets.The feedforward datasets were MNIST, Pentomino, TFD, and Forest Covertype; recurrent evaluation used three polyphonic music datasets.
- Distributions of the Orders of Lp Units: Learned Lp orders depended strongly on the dataset and varied substantially within individual models, with Pentomino showing two distinct modes.These observations supported both claims that no universal order exists and that units can specialize with different orders.
- Distributions of the Orders of Lp Units: Fixed p = 2 on TFD produced a worse test error of 0.21 than the learned-order model.Orders initialized near 3 changed significantly during training and became different across datasets.
- Generalization Performance: 99.03% test accuracy on MNIST was comparable to the 99.06% accuracy of an MLP with maxout units.The model used two Lp layers, dropout, and a softmax output layer.
- Generalization Performance: 2.83% error on Forest Covertype improved on the previous 3.13% state-of-the-art rate and was comparable to an MLP with maxout units.On the three music datasets, Lp-based DOT-RNNs achieved state-of-the-art RNN-only results, though the authors called for further recurrent-network investigation.
7 Conclusion
The paper introduces Lp units as learned-order generalizations of pooling operators and analyzes their geometric role in modeling separating boundaries. Experiments across feedforward and recurrent networks support dataset-dependent orders and strong benchmark performance, while recurrent benefits remain provisional.
- Contribution: The proposed activation computes an Lp norm over projections of lower-layer units, generalizing max, average, and root-mean-squared pooling.The related maxout unit is closely related under non-negative input signals.
- Contribution: Unlike conventional pooling, each Lp unit learns its order, and the paper argues that optimal models use varied orders across units and datasets.The experiments found that estimated-order distributions depended strongly on the dataset and were far from a single concentrated value.
- Geometric analysis: The geometric analysis associates each Lp unit with an ellipsoidal boundary whose order-dependent combinations can model separating curves with non-stationary curvature more efficiently.This claim was supported by a small-scale experiment.
- Empirical evaluation: The approach was evaluated in deep feedforward networks on MNIST, TFD, Pentomino, and Forest Covertype, and in recurrent networks on polyphonic music prediction.The conclusion reports empirical verification across both network types.
- Empirical evaluation: Lp-based recurrent networks achieved state-of-the-art results on all three music datasets, but further investigation is needed before drawing more concrete conclusions about recurrent benefits.The recurrent result is therefore stronger as an empirical outcome than as a settled general conclusion.