PaperlayerView exact revision on arXiv

Source-linked AI summary

Group Sparse Regularization for Deep Neural Networks

Simone Scardapane, Danilo Comminiello, Amir Hussain, Aurelio Uncini

arXiv:1607.00485v1stat.MLcs.LG

TL;DR

The paper addresses the need to jointly optimize network weights, hidden-layer widths, and active input features rather than handling these tasks separately. It applies group and sparse group Lasso to outgoing-weight groups, and reports comparable accuracy with highly compact networks across digit and large-scale classification experiments.

  • Problem

    Pruning, feature selection, and network optimization are generally treated separately, motivating a principled formulation that performs all three tasks simultaneously.

  • Method

    The method groups each unit’s outgoing weights and applies group Lasso or sparse group Lasso to remove neurons, select inputs, and optionally sparsify retained groups.

  • Results

    Test accuracy is comparable across the three algorithms, while SG-L1-NN produces extremely sparse, more compact networks, with a negligible MNIST test-accuracy loss.

  • Takeaways & Limitations

    The sparse group Lasso formulation jointly enables pruning and feature selection with small implementation overhead and savings in storage requirements and embedded-device power consumption.

Abstract

from arXiv · show

In this paper, we consider the joint task of simultaneously optimizing (i) the weights of a deep neural network, (ii) the number of neurons for each hidden layer, and (iii) the subset of active input features (i.e., feature selection). While these problems are generally dealt with separately, we present a simple regularized formulation allowing to solve all three of them in parallel, using standard optimization routines. Specifically, we extend the group Lasso penalty (originated in the linear regression literature) in order to impose group-level sparsity on the network's connections, where each group is defined as the set of outgoing weights from a unit. Depending on the specific case, the weights can be related to an input variable, to a hidden neuron, or to a bias unit, thus performing simultaneously all the aforementioned tasks in order to obtain a compact network. We perform an extensive experimental evaluation, by comparing with classical weight decay and Lasso penalties. We show that a sparse version of the group Lasso penalty is able to achieve competitive performances, while at the same time resulting in extremely compact networks with a smaller number of input features. We evaluate both on a toy dataset for handwritten digit recognition, and on multiple realistic large-scale classification problems.

I. INTRODUCTION

Deep networks create efficiency and feature-selection challenges as their size and parameter count grow. The paper proposes group Lasso regularization to jointly obtain compact networks, prune neurons, and select input features.

  • Motivation: Large deep networks can overfit, burden distributed training with parameter exchange, and exceed the computational budgets of low-power devices.These concerns motivate reducing network size and parameter count.
  • Motivation: Feature selection seeks a small subset of input variables retaining most discriminative information, but is commonly handled through a separate workflow.That workflow may combine feature selection, network training, and weight-matrix compression.
  • Research question: The paper asks whether pruning, feature selection, and network optimization can be performed simultaneously by minimizing a principled cost function.It treats deleting an input feature as analogous to pruning a neuron in the input layer.
  • Approach: Group Lasso groups all outgoing weights from a neuron, allowing entire neurons to be removed; sparse group Lasso adds weight sparsity within retained groups.Groups can represent input neurons, hidden neurons, or bias units, yielding feature selection, pruning, or bias selection.
  • Evaluation: Experiments compare the proposed penalties with ℓ2 and ℓ1 regularization on handwritten-digit and realistic large-scale classification problems.The paper is organized around regularization methods, proposed group penalties, experiments, pruning comparisons, and conclusions.

II. WEIGHT-LEVEL REGULARIZATION FOR NEURAL

The paper formulates neural-network training as minimizing a task loss plus a regularization term. It contrasts weight decay and Lasso-style penalties, emphasizing their different sparsity behavior and optimization properties.

  • Network formulation: A deep network maps input x to output y through H hidden layers, with all adaptable weights and biases collected in a parameter vector w.Each layer applies its weights and bias through an element-wise activation function.
  • Network formulation: Training minimizes a regularized cost combining a loss function with a penalty weighted by λ.Squared error is given for regression, while cross-entropy is given for classification.
  • Weight decay: Weight decay uses an ℓ2 penalty that shrinks weights proportionally to their magnitude but does not directly produce sparsity.Sparsity requires thresholding small weights to zero.
  • Lasso: Lasso uses an ℓ1 penalty on absolute weight magnitudes and can promote weight-level sparsity during training.Its nondifferentiability at zero is handled with subgradients, while standard first-order procedures are commonly applied elsewhere.
  • Composite penalties: Elastic-net-style smoothing mixes ℓ2 and ℓ1 terms, but requires separate hyperparameters to weight them.Approximating the ℓ1 norm with a smooth convex term is another option.

III. NEURON-LEVEL REGULARIZATION WITH GROUP

The section defines group-level regularization that can simultaneously select input features, prune hidden neurons, and remove biases by zeroing structured connection groups.

  • Motivation: Standard ℓ1 and ℓ2 regularization prevent overfitting but do not directly pursue compact architectures, leaving equivalent-accuracy network designs dependent on initialization.A neuron can generally be removed only when all of its ingoing or outgoing connections are zero.
  • Group definitions: Input groups collect all outgoing connections from one input, hidden groups collect outgoing connections from one hidden neuron, and bias groups are scalar biases.Input groups correspond to rows of W1, while hidden groups correspond to transposed rows of later-layer weight matrices.
  • Structural effects: Zeroing an input group performs feature selection, zeroing a hidden group removes its neuron, and zeroing bias groups deletes corresponding biases.These group effects jointly produce feature selection and thinner hidden layers within the network regularization procedure.
  • Group penalties: Group Lasso forces all variables in a group to zero together, while sparse group Lasso additionally promotes sparsity within surviving groups.Both penalties are convex but non-differentiable; the sparse variant addresses remaining connection-level sparsity.
  • Connectivity patterns: Compared with Lasso, group Lasso removes all connections exiting a selected neuron, creating more structured connectivity even at a similar sparsity level.Sparse group Lasso combines neuron-level removal with additional removal of individual connections among remaining groups.

B. Graphical interpretation of group sparsity

The graphical interpretation views regularization as a constraint region whose geometry indicates which sparse solutions are favored, although neural-network loss remains non-convex.

  • Scope: Because hidden layers make neural-network optimization highly non-convex, the geometric interpretation is presented for visualization rather than as a direct convexity characterization.The construction is motivated by the convex-loss case even though the network model itself is non-convex.
  • Constrained formulation: For convex losses, the regularized problem can be reformulated as minimizing loss subject to R(w) ≤ µλ.The scalar µλ depends on λ, and the constrained formulation follows from duality when there is no duality gap.
  • Geometric view: At an active constraint, the equal-loss set is tangent to the feasible region B = {w : R(w) ≤ µλ}, enabling geometric comparison of penalties.The paper uses the shape of B to visualize the behavior of different regularization terms.
  • Penalty geometries: Figure 3 compares the sphere from squared ℓ2 regularization, the sparsity-favoring diamond from ℓ1 regularization, and the grouped ℓ2,1 geometry.The grouped geometry applies ℓ2,1 to groups {1, 2} and {3}, with µλ = 1 and without scaling factors.

A. Experimental setup

The experiments evaluate the proposed regularization on classification benchmarks using a consistent deep-learning training and data-processing pipeline.

  • Benchmarks: The evaluation starts with a handwritten-digit toy dataset and then examines more elaborate real-world classification datasets.All hidden layers use ReLU activations, while the output layer uses softmax with one-hot class encoding.
  • Optimization: Networks are initialized with the method of and trained with Adam, using its default parameters and mini-batch sizes adjusted to problem dimensionality.Adam is described as a stochastic-gradient-descent variant with adaptive step sizes and momentum.
  • Objective: Training minimizes the regularized objective with standard cross-entropy loss for classification.The loss is specified through the class targets and predicted softmax outputs.
  • Data processing: Each input column is normalized to [0, 1], 25% of each dataset is randomly held out for testing, and every experiment is repeated 25 times.The repeated runs are used to average out statistical variations.

B. Comparisons with the DIGITS dataset

On the DIGITS dataset, the proposed sparse group Lasso achieved accuracy comparable to the alternatives while producing more structured sparsity in connections, input features, and hidden neurons.

  • Experimental setup: The DIGITS experiment compared L2-NN, L1-NN, G-L1-NN, and SG-L1-NN while varying λ from 10^-1 to 10^-5.The dataset contains 1797 handwritten 8×8 grayscale digit images.
  • Accuracy: Test accuracy behaved similarly across the four penalties, rapidly approaching slightly below 100% for sufficiently small regularization.
  • Connection sparsity: SG-L1-NN produced sparsity equal or superior to the alternatives, whereas L2-NN ranged from 20% sparsity at best to 0% on average.Sparsity is measured as the percentage of zero weights among all connections.
  • Structured compactness: At λ = 10^-3, L1-NN and SG-L1-NN each removed four fifths of the connections, while SG-L1-NN more consistently removed input features and hidden neurons.
  • Selected features: Feature visualization showed that the proposed approach ignored pixels in the outermost left and right image regions.The visualization used cumulative input-to-first-hidden-layer weight intensity, with white indicating zero outgoing connections.

C. Comparisons with large-scale datasets

On three realistic datasets, the sparse group Lasso approach is evaluated against alternative penalties and produces highly sparse, compact networks with comparable test accuracy. The experiments report dataset characteristics, implementation settings, accuracy, timing, sparsity, and resulting layer sizes.

  • Datasets: The evaluation covers SSD, MNIST, and COVER, requiring deeper and larger networks than the earlier experiments.SSD uses 48 features and 58,508 examples; MNIST contains 70,000 28 × 28 images; COVER has roughly half a million training examples.
  • Experimental setup: The experiments compare penalties using dataset-specific architectures, regularization factors, and mini-batch sizes, while focusing on SG-L1-NN against competing penalties.The authors state that similar results arise from different architecture and regularization choices and that SG-L1-NN had outperformed simpler G-L1-NN.
  • Results: Test accuracy is comparable across the three algorithms, with a negligible MNIST loss for SG-L1-NN.Table III reports average training and test accuracy, training time, network sparsity, and final hidden-layer sizes.
  • Results: On MNIST, SG-L1-NN removes more than 200 input features on average and retains 146 hidden-layer neurons, versus 243 for L1-NN and 654 for L2-NN.The selected features can be visually inspected in Figure 6; white regions indicate low outgoing-weight strength.
  • Training time: On CPU, the SGL penalty computes faster than ℓ2 and ℓ1 norms, whereas CUDA training is slightly slower because each group requires two square-root operations.The authors note that GPU timing can be improved using faster mathematical computations at some precision cost.
  • Results: SG-L1-NN achieves high sparsity and low input- and hidden-layer neuron counts with no or negligible accuracy losses.The paper presents these results as evidence that sparse group Lasso produces compact networks across the evaluated datasets.

V. RELATED WORKS

The related work spans stochastic complexity reduction, post-training pruning, and low-rank compression. These approaches differ from the paper’s goal of directly learning a compact topology during network optimization.

  • Randomized complexity reduction: Stochastic complexity-reduction methods randomly remove connections, skip layers, or apply ℓ1 penalties to neuron activations during training.Examples include dropout, stochastic depths, and activation sparsity methods.
  • Pruning methods: Post-training pruning methods simplify network structure after training by removing connections or neurons according to saliency or error sensitivity.Optimal brain damage uses second-order cost derivatives, while other methods assess sensitivity to neuron removal.
  • Pruning methods: Han et al.’s two-step procedure first trains with ℓ2 regularization, then thresholds and fixes small weights at zero before retraining.The procedure can be repeated iteratively to further reduce network size.
  • Pruning methods: The reviewed pruning methods do not jointly satisfy direct neuron-level action, integrated optimization, and avoidance of heuristic assumptions.The passage contrasts these methods with the objective introduced earlier in the paper.
  • Low-rank approximation: Low-rank compression replaces each weight matrix with factors A and B of user-chosen rank r, trading compression against accuracy.Optimization then operates on the factor matrices rather than the original weight matrix.
  • Low-rank approximation: Low-rank approximation is not guaranteed to work efficiently and can produce substantially worse results when r is chosen poorly.

VI. CONCLUSIONS

The paper introduces sparse group Lasso regularization to jointly optimize neural-network weights, pruning, and feature selection. It reports compact networks with storage and power-consumption savings, while identifying unresolved regularization questions and extensions to other architectures.

  • Conclusions: Sparse group Lasso simultaneously performs pruning and feature selection while optimizing neural-network weights.The formulation is implemented efficiently with only a small overhead relative to standard ℓ2 or ℓ1 formulations.
  • Conclusions: Experimental comparisons report superior performance for obtaining highly compact networks, with savings in storage requirements and embedded-device power consumption.
  • Future work: Future work will study the interaction between sparse ℓ1 regularization and the paper’s non-convex cost, including non-convex regularizers and iterative convex approximations.The authors identify this interaction as an open problem in optimization literature.
  • Future work: The authors also plan to extend group Lasso formulations to convolutional and recurrent neural networks.They specifically mention reservoir computing architectures as an example of ongoing work.
Loading 1607.00485v1…