Source-linked AI summary
Diversity Networks: Neural Network Compression Using Determinantal Point Processes
Zelda Mariet, Suvrit Sra
TL;DR
Setting network architecture is costly, while networks are often trained with more parameters than necessary before pruning. Divnet uses a DPP to select diverse neurons and reweights connections to fuse the others into them; it outperforms previous neuron-pruning approaches and can shrink redundant networks without hurting performance.
Problem
Method
Divnet uses a DPP to select a diverse subset of neurons, then fuses the remaining neurons into the selected ones through reweighting.
Source: §1 · p. 1Results
Divnet consistently outperforms previous neuron-pruning approaches, and its reweighting strategy benefits other pruning approaches.
Source: §1 · p. 2Takeaways & Limitations
Divnet can shrink large, redundant networks without impacting performance or requiring further training.
Source: §4 · p. 8Divnet requires the user to select the final network size.
Source: §4 · p. 8
Abstract
from arXiv · showhide
We introduce Divnet, a flexible technique for learning networks with diverse neurons. Divnet models neuronal diversity by placing a Determinantal Point Process (DPP) over neurons in a given layer. It uses this DPP to select a subset of diverse neurons and subsequently fuses the redundant neurons into the selected ones. Compared with previous approaches, Divnet offers a more principled, flexible technique for capturing neuronal diversity and thus implicitly enforcing regularization. This enables effective auto-tuning of network architecture and leads to smaller network sizes without hurting performance. Moreover, through its focus on diversity and neuron fusing, Divnet remains compatible with other procedures that seek to reduce memory footprints of networks. We present experimental results to corroborate our claims: for pruning neural networks, Divnet is seen to be notably superior to competing approaches.
1 Introduction
Tuning network architecture is costly, and networks are often trained larger than necessary before being pruned. Divnet uses a DPP to select diverse neurons and fuses the remaining neurons into them, reducing network size while preserving performance and outperforming previous neuron-pruning approaches.
1 Introduction
- Architecture hyperparameters are difficult to tune, so networks are commonly trained with more parameters than necessary and pruned afterward.Too-small networks may learn poorly, while oversized ones can take longer to train and risk overfitting.Source: §1 · p. 1
- Divnet places a Determinantal Point Process over neurons to select a diverse subset, then fuses the remaining neurons into those selected.The fusion uses a reweighting procedure for the retained neurons’ connections.Source: §1 · pp. 1–2
- The approach is not limited to feed-forward networks; it also applies to architectures with fully connected layers, including Deep Belief Nets and CNNs.Source: §1 · p. 2
- Divnet consistently outperforms previous neuron-pruning approaches, and its reweighting strategy also benefits other pruning methods.Source: §1 · p. 2
- Divnet’s diversity focus is complementary to other compression techniques and can usually be combined with them.Parameter-removal methods and Divnet target different forms of redundancy.Source: §1 · pp. 2–3
2 Diversity and redundancy reduction
Divnet addresses the difficulty of identifying redundant neurons by sampling diverse activation patterns with a DPP, then reweighting retained neurons to preserve downstream inputs. Its method specifies a DPP kernel and fuses removed neurons’ contributions into the selected neurons.
2 Diversity and redundancy reduction
- Because activation vectors may be linearly independent even when neurons compute redundant information, Divnet samples diverse, nonsaturated activation patterns instead of simply retaining a maximal independent subset.The training set or number of batches can exceed the number of neurons, making maximal-subset selection likely to retain all neurons.Source: §2 · p. 3
2.1 Neuronal diversity via Determinantal Point Processes
- A DPP assigns subset probabilities according to determinants of kernel submatrices, tending to favor diverse subsets; its kernel is positive definite.For a subset Y, the relevant submatrix is the principal submatrix indexed by Y.Source: §2.1 · p. 3
2.1.1 Constructing the DPP kernel
- A Gaussian RBF kernel performed better than simple linear kernels; Divnet adds ε=0.01 and scales the kernel to target expected sample size k, using β=10/|T|.The fixed bandwidth was chosen to avoid the added training cost of cross-validation.Source: §2.1.1 · pp. 3–4
2.2 Fusing redundant neurons
- To preserve layer information, Divnet reweights retained neurons by least-squares fitting removed neurons’ contributions, minimizing changes to next-layer inputs.The procedure minimizes the difference between the next-layer inputs before and after DPP pruning.Source: §2.2 · p. 4
3 Experimental results
Experiments on MNIST, MNIST ROT, and CIFAR-10 compare DPP pruning and reweighting with other pruning procedures. Divnet’s results reflect both diverse-neuron selection and fusion of redundant neurons.
3 Experimental results
- The experiments cover MNIST, MNIST ROT, and CIFAR-10, with networks trained until a predefined training-error threshold is reached.The plots report standard deviations as error bars.Source: §3 · p. 5, Table 1 · p. 5
3.1 Pruning and reweighting analysis
- On MNIST, k-DPP sampling selects first-layer neurons with diverse activations, unlike the redundant activation patterns among the first 50 neurons.The first 50 include three neurons that saturate to 1 across all ten digit instances.Source: §3.1 · p. 5, Figure 1 · p. 6
- Shrinking the second layer produced similar results, with a much wider performance gap between Divnet and random pruning.These experiments kept the first layer intact.Source: §3.1 · p. 5
- For the same final network size, k-DPP-pruned networks have consistently better training and test errors than randomly pruned networks.Both pruning procedures significantly increase error, though retaining more neurons reduces the damage.Source: §3.1 · p. 5, Figure 2 · p. 7
- Reweighting considerably reduces training and test errors, even when 90% of a layer’s neurons are removed, and lowers result variability.The authors suggest the lower standard deviation may make reweighting a way to regularize networks.Source: §3.1 · p. 5, Figure 3 · p. 7
3.2 Performance analysis
- At heavy pruning, Divnet outperforms importance pruning; it also ultimately beats random pruning, although reweighting benefits random pruning too.Figure 5 compares importance pruning with Divnet, while Figure 4 compares Divnet with random pruning when both use reweighting.Source: §3.1 · p. 5, §3.2 · p. 6, Figure 4 · p. 7
3.3 Discussion and Remarks
- Sampling and reweighting ran several orders of magnitude faster than training, but reweighting took significantly longer than sampling.The authors note that using part of the training set could speed Divnet further, possibly at the cost of subsequent network performance.Source: §3.3 · p. 8
- DPP pruning during training did not significantly outperform random pruning, with or without reweighting.This contrasts with the reported benefits of pruning after training.Source: §3.3 · p. 8
4 Future work and conclusion
Divnet uses neuron-behavior similarities to merge redundancy, producing smaller networks without impacting performance or requiring further training. It is compatible with other pruning and memory-management methods, while users still choose the final network size.
4 Future work and conclusion
- Divnet merges redundant neurons based on their behavior, shrinking large networks without impacting performance or requiring further training.The authors expect a similar performance profile on larger datasets and networks, but identify those evaluations as future work.Source: §4 · p. 8
- Users choose the final network size, while sampling and reweighting precision can trade off accuracy, memory management, and computation time.The authors suggest exploring DPP set-size distributions to reduce the burden of choosing layer size.Source: §4 · pp. 8–9
- Divnet can be combined with other pruning or memory-management methods because it requires only neuron activation vectors.Its reweighting procedure is agnostic to how pruning is performed.Source: §4 · p. 9
- Applying Divnet to CNN filters and investigating alternative or learned kernels are proposed directions, not demonstrated results.The filter proposal views each filter's activations as a vector; kernel investigations are framed as future work.Source: §4 · p. 9
A Pruning the second layer
The appendix presents comparisons involving random pruning, k-DPP pruning, and Divnet when pruning the second layer, but the supplied passages do not state their outcomes.
A Pruning the second layer
- Figure 6 compares random pruning with k-DPP pruning for the second layer.The caption identifies the procedures but provides no result values or winner.Source: Figure 6 · p. 11
- Figure 7 compares Divnet with simple k-DPP pruning for the second layer.The supplied caption does not report which procedure performs better.Source: Figure 7 · p. 11
- Figure 8 compares random and k-DPP pruning when both are followed by reweighting.The supplied caption gives the comparison scope but no outcome.Source: Figure 8 · p. 11
B Influence of the β parameter on network size and error
The appendix examines how β relates to pruning outcomes on MNIST, with figures addressing the remaining neuron count and training error; the supplied passages do not report the direction or magnitude of those effects.
B Influence of the β parameter on network size and error
- Figure 9 examines β's influence on MNIST training error, with dotted lines marking minimum and maximum errors.The supplied passage gives no error values or trend.Source: Figure 9 · p. 12
- Figure 10 examines β's influence on the number of neurons remaining after non-parametric DPP pruning on MNIST.This procedure uses a DPP rather than a k-DPP.Source: Figure 10 · p. 12
C Comparison of Divnet to importance-based pruning and random pruning on the CIFAR-10 dataset
The appendix presents CIFAR-10 training- and test-error comparisons involving random pruning, importance pruning, and Divnet as network parameters are reduced; the supplied passages do not state the outcomes.
C Comparison of Divnet to importance-based pruning and random pruning on the CIFAR-10 dataset
- The supplied CIFAR-10 panel label identifies training error as an evaluation measure.No training-error values or method ranking are provided.Source: C Comparison of Divnet to… · p. 13
- The supplied CIFAR-10 panel label identifies test error as an evaluation measure.No test-error values or method ranking are provided.Source: C Comparison of Divnet to… · p. 13
- Figure 11 compares random pruning, importance pruning, and Divnet's effect on network performance as parameters are reduced.The caption does not report which method performs best.Source: Figure 11 · p. 13