Source-linked AI summary
Learning the Number of Neurons in Deep Networks
Jose M Alvarez, Mathieu Salzmann
TL;DR
Deep networks are effective but their manually chosen, overcomplete architectures can be costly and difficult to deploy. The paper learns neuron counts and network parameters jointly with neuron-level group sparsity, reducing parameters by up to 80% while retaining or improving recognition accuracy. The resulting compact networks also save memory and computation at test time.
Problem
Deep-network layer and neuron counts are usually chosen manually, while very deep, redundant models impose high memory and computation costs that can limit constrained-platform deployment.
Method
The method uses a group sparsity regularizer whose parameter groups correspond to individual neurons, so zeroed groups remove neurons during joint learning.
Results
Up to 80% fewer parameters were obtained with no recognition-accuracy loss and typical accuracy improvements over the complete network.
Takeaways & Limitations
The approach jointly learns neuron counts and parameter values, producing more compact architectures that save memory and computation at test time.
Abstract
from arXiv · showhide
Nowadays, the number of layers and of neurons in each layer of a deep network are typically set manually. While very deep and wide networks have proven effective in general, they come at a high memory and computation cost, thus making them impractical for constrained platforms. These networks, however, are known to have many redundant parameters, and could thus, in principle, be replaced by more compact architectures. In this paper, we introduce an approach to automatically determining the number of neurons in each layer of a deep network during learning. To this end, we propose to make use of structured sparsity during learning. More precisely, we use a group sparsity regularizer on the parameters of the network, where each group is defined to act on a single neuron. Starting from an overcomplete network, we show that our approach can reduce the number of parameters by up to 80\% while retaining or even improving the network accuracy.
1 Introduction
Deep-network architecture design remains largely manual, while very deep models impose substantial memory, speed, and training costs. The paper addresses this by jointly selecting neurons during learning and reports compact networks with preserved or improved accuracy.
- Architecture design typically sets layer and neuron counts manually, determining model complexity through trial and error.
- Very deep networks can improve expressiveness but increase memory and speed costs, complicate learning, and hinder deployment on constrained platforms.
- Constructive approaches grow networks from shallow architectures, but shallow initialization may be less expressive and provide poor starting points for added layers.
- Post-processing methods train compact networks to mimic large ones, requiring successful training of an initial deep network.
- The proposed method applies neuron-level group sparsity while learning, jointly determining relevant neuron counts and parameter values without first training a redundant network.
- Up to 80% fewer parameters were achieved with no loss in recognition accuracy, often improving over the complete network.
2 Related work
Prior model-selection work grows or prunes networks and often treats compacting as post-processing, leaving architecture choices fixed during learning. This paper instead performs neuron selection directly through per-layer group sparsity and targets larger-scale settings.
- Model selection in deep learning has remained limited, with layer and neuron counts commonly tuned manually or avoided through very deep networks.
- Constructive methods incrementally add neurons or layers, whereas destructive methods begin with a deep network and remove parameters or neurons.
- Low-rank filter decompositions and parameter regularizers reduce model size but generally operate after training and do not determine layer or neuron counts.
- The proposed framework performs model selection and compact-network construction in one learning process using a regularizer, without pre-training.
- Unlike prior group-sparsity methods, it scales to datasets and architectures two orders of magnitude larger and applies regularization per layer rather than globally.
3 Deep Model Selection: Learning with Structured Sparsity
The paper selects neuron counts during network learning by applying structured sparsity to neuron-level parameter groups in an overcomplete deep network. Proximal optimization drives selected groups to zero, allowing the resulting neurons and dependent neurons to be removed.
- Framework: The framework jointly learns network parameters and neuron counts, rather than training an initial network as a preprocessing step.It is introduced for general deep networks composed of layered linear operations, nonlinearities, and potentially pooling.
- Framework: The learning objective combines a prediction loss with a parameter regularizer, such as logistic or square loss alongside weight decay or sparsity penalties.The regularizer acts on the network parameters and can encode different sparsity structures.
- Structured sparsity: Group sparsity drives entire neuron-specific parameter groups to zero, unlike standard penalties that typically shrink parameters individually.Each neuron is represented by a parameter group, so zeroing the group cancels that neuron's influence.
- Structured sparsity: The sparse group Lasso extension combines group sparsity with individual-parameter sparsity, with α controlling the relative influence of the two terms.The experiments use α = 0 and α = 0.5; α = 0 recovers the group-sparsity regularizer.
- Optimization: Proximal gradient descent alternates loss-based gradient steps with groupwise proximal updates, applied independently because the groups do not overlap.In stochastic training, the proximal operator is applied at the end of each epoch over mini-batch updates.
- Architecture reduction: Zeroed neuron parameters permit removing those neurons and, in fully connected layers, neurons that depend on zeroed outputs, yielding a compact architecture.The procedure starts from an overcomplete network and removes neurons after learning terminates.
4 Experiments
The experiments evaluate automatic neuron selection across three architectures and multiple image-recognition datasets, measuring accuracy, parameter reduction, generalization, runtime, and memory. The method reduces parameters substantially while generally maintaining or improving recognition performance and enabling faster, smaller test-time networks.
- Experimental setup: The study evaluates three architectures across ImageNet, Places2-401, and ICDAR, with particular focus on parameter reduction.The reported results emphasize ImageNet and ICDAR; Places2-401 results are provided as supplementary material.
- ImageNet: 27% reduction in total parameters is achieved for BNetC after reducing neurons by over 12% across layers while improving generalization ability.The largest neuron reduction occurs in the last layer, which also reduces the subsequent fully connected layer.
- ImageNet: More than 19% parameter reduction is achieved for Dec8-640 with improved generalization when the overcomplete network starts with 640 neurons per layer.Starting from 512 neurons yields only a small parameter reduction and minimal performance gain.
- Generalization: 48.2% of Dec8-768 parameters are zeroed out, while the training-validation accuracy gap decreases 28.5%, from 14% to 10%.The authors report a similar reduction in the training-validation gap for the other tested architectures.
- ICDAR: Up to 80% parameter reduction is obtained on ICDAR with Dec3 while further improving the original network's performance.The experiment compares group sparsity and structured group-lasso regularization against state-of-the-art baselines.
- Benefits at test time: Removing zeroed-out neurons after training produces speedups of up to 13% for BNetC and almost 50% for Dec3, alongside memory savings.Table 2 reports memory savings of up to 82% in parameters and up to 25% in computed features.
5 Conclusions
The paper presents group sparsity as a unified way to learn neuron counts and parameter values, producing compact networks while retaining practical benefits. Its main scope limitation is that the number of layers remains fixed.
- Group sparsity jointly learns the number of neurons and parameter values in a single framework.The regularizer enables automatic neuron-count estimation during network learning.
- The method produces more compact architectures than the initial overcomplete network, saving memory and computation at test time.
- Experiments demonstrate benefits across different network architectures, supporting the approach’s generalizability.
- The number of layers remains fixed, so the approach does not yet learn whether entire layers should be removed.The authors propose studying architectures in which layers can be bypassed entirely.
- Future evaluation will extend the approach to regression networks and autoencoders.