Source-linked AI summary
Moonshine: Distilling with Cheap Convolutions
Elliot J. Crowley, Gavin Gray, Amos Storkey
TL;DR
The paper asks how to reduce neural-network memory for constrained deployment without a commensurate performance loss. It distills students formed by replacing teacher convolutions with cheaper blocks and using attention transfer, finding substantial savings with little accuracy loss and better performance than direct student training.
Problem
Memory-intensive neural networks impede deployment on embedded devices, motivating networks with substantially fewer parameters without a commensurate performance loss.
Method
Structural distillation replaces teacher convolutional blocks with cheaper grouped or bottleneck blocks while retaining the architecture and training with attention transfer.
Results
Across evaluated settings, cheap-block students achieve strong accuracy at much lower parameter counts, and attention transfer outperforms knowledge distillation and training from scratch.
Takeaways & Limitations
Compression can be applied quickly without redesign or additional engineering, using the original optimiser and hyperparameters for the cheaper student.
Takeaways & Limitations
The chosen cheap convolutions prioritize ease of implementation, leaving more complicated approximate operations and custom blocks for future work.
Abstract
from arXiv · showhide
Many engineers wish to deploy modern neural networks in memory-limited settings; but the development of flexible methods for reducing memory use is in its infancy, and there is little knowledge of the resulting cost-benefit. We propose structural model distillation for memory reduction using a strategy that produces a student architecture that is a simple transformation of the teacher architecture: no redesign is needed, and the same hyperparameters can be used. Using attention transfer, we provide Pareto curves/tables for distillation of residual networks with four benchmark datasets, indicating the memory versus accuracy payoff. We show that substantial memory savings are possible with very little loss of accuracy, and confirm that distillation provides student network performance that is better than training that student architecture directly on data.
1 Introduction
The paper targets memory-constrained deployment by distilling teacher networks into students that preserve the teacher architecture while replacing standard convolutions with cheaper blocks. It claims this enables greater compression than shrinking the architecture, with the same training setup and effective grouped replacements.
- 1 Introduction: The method addresses deployment barriers caused by resource constraints and memory-intensive networks in embedded devices.The dominant run-time memory cost is attributed to storing network parameters.
- 1 Introduction: Replacing standard convolutional blocks with cheaper blocks preserves the teacher architecture and outperforms smaller architectures with standard blocks at comparable parameter counts.The approach substitutes convolutional blocks rather than making networks thinner or shallower.
- 1 Introduction: Greater model compression is possible by replacing convolutional blocks through distillation rather than shrinking the network architecture.
- 1 Introduction: Grouped convolutional blocks, with or without bottleneck contraction, are effective replacement blocks for structural distillation.
- 1 Introduction: The substitution requires little design and training overhead because it uses the original optimiser and hyperparameters.The paper describes the replacement as cheap in design time and training complexity.
2 Related Work
Prior work reduces convolutional cost through grouped, separable, bottleneck, and other approximate operations, while distillation transfers information from a trained teacher to a smaller student. Attention transfer matches channel-averaged intermediate activations and is reported as effective for this paper's experiments.
- 2 Related Work: Depthwise separable convolutions apply channel-wise spatial kernels followed by pointwise mixing and have been adapted for mobile devices.
- 2 Related Work: Grouped convolutions restrict channel connectivity, while other approaches combine separability, topological subdivision, or bottlenecks to reduce model size while maintaining accuracy.
- 2 Related Work: The paper learns compressed students through distillation from a pre-trained teacher rather than only training compressed networks from scratch or retraining pruned models.
- 2 Related Work: Attention transfer matches activations after averaging over channels, providing an alternative to intermediate activation mapping for training smaller students.The paper reports attention transfer as effective in its experiments.
3 Compression with Cheap Convolutions
The paper compresses teacher networks by replacing standard convolutional blocks with cheaper structured blocks while preserving the teacher’s architecture, and trains the resulting students through distillation. It compares knowledge distillation and attention transfer, then introduces grouped and bottlenecked convolutions to reduce parameter costs.
- 3.1 Distillation: Knowledge distillation combines label cross-entropy with a temperature-scaled loss encouraging student outputs to match teacher outputs.Attention transfer instead aligns normalized spatial activation maps at selected corresponding layers.
- 3.2 Cheap Convolutions: Grouped convolution reduces a standard layer’s parameter cost by a factor of g by restricting channel mixing to g groups, followed by a pointwise convolution for cross-group mixing.The resulting G(g) operator replaces a k × k convolution with grouped and 1 × 1 convolutions.
- 3.2 Cheap Convolutions: Bottlenecked grouped blocks further reduce parameters by shrinking channels before grouped convolution and restoring them afterward.Varying bottleneck size and group count produces network parameter counts spanning two orders of magnitude.
- 3.2 Cheap Convolutions: The proposed grouped and bottlenecked blocks are structured alternatives to sparsity methods and are intended to provide substantial compression with minimal performance reduction when combined with distillation.They can be implemented directly in standard deep-learning frameworks without storing a separate sparsity structure.
4 CIFAR Experiments
The CIFAR experiments compare distilled students using cheaper convolutional blocks, reduced architectures, and different training objectives. Across CIFAR-10 and CIFAR-100, structural substitutions deliver strong compression and low error, while attention transfer outperforms knowledge distillation and training from scratch.
- 4.1 Network Descriptions: The experiments retain the WRN-40-2 teacher architecture while replacing standard blocks with S, B, G, and BG alternatives, including grouped and bottlenecked variants.Students also include thinner or shallower standard-block networks and dilated 2 × 2 convolutions; Table 1 specifies the block costs and components.
- 4.2 Analysis and Observations: Cheap convolutional blocks outperform reduced standard-block architectures at comparable parameter or mult-add budgets on CIFAR-10.Figure 2 compares test error against parameter count and mult-adds for attention-transferred students.
- 4.2 Analysis and Observations: BG(2, 2) achieves 5.57% error versus 5.66% for 16-2 while using 287K rather than 692K parameters.Several BG networks also outperform 16-1 with fewer parameters.
- 4.2 Analysis and Observations: G(N/8) reaches 5.06% error with just over one-fifth of the teacher’s parameters, while BG(2, M/8) uses less than one-tenth with a 1.15% error increase.These results demonstrate substantial compression with small accuracy losses.
- 4.2 Analysis and Observations: Grouped + pointwise blocks perform better with half or quarter as many groups than with depthwise-separable grouping, trading modest parameter increases for lower error.G(N/4) uses 363K parameters versus 294K for G(N) and has 1.26% lower error; large bottlenecks such as BG(4, M) increase error substantially.
- 4.2 Analysis and Observations: Attention transfer consistently outperforms knowledge distillation and training from scratch for the evaluated student networks on CIFAR-10 and CIFAR-100.The authors conclude that distillation is necessary for the observed high performance.
5 Additional Experiments
Additional experiments show that replacing convolutional blocks with cheaper grouped blocks preserves the teacher architecture while improving parameter-efficient performance on ImageNet and semantic segmentation.
- 5 Additional Experiments: These experiments extend convolution cheapening and distillation from CIFAR classification to ImageNet classification and Cityscapes semantic segmentation.The section tests whether the method remains effective for larger images, more classes, and a non-classification task.
- 5.1 ImageNet: The ImageNet study compares ResNet-34 students with G(N) and G(4) replacement blocks against ResNet-18 and channel-reduced Res18-0.5 students using attention transfer.The experiments use a pretrained 21.8M-parameter ResNet-34 teacher and evaluate top-1 and top-5 validation error.
- 5.1 ImageNet: 10.66% versus 15.02% top-5 error: distilled Res34-G(N) outperforms similarly sized Res18-0.5, while scratch Res34-G(N) reaches 12.26%.The comparison supports cheapening convolutions rather than shrinking the architecture; distillation helps Res34-G(N) but slightly worsens Res18-0.5.
- 5.1 ImageNet: 8.43% versus 8.57% top-5 error: distilled Res34-G(4) surpasses the ResNet-34 teacher despite using about 8.1M versus 21.8M parameters.Res34-G(4) also outperforms Res18 at roughly comparable cost, while its scratch performance is close to the teacher.
- 5.2 Semantic Segmentation: On Cityscapes, ERFNet-G(N) replaces ERFNet blocks while preserving appropriate dilations and uses attention transfer between selected feature maps.Performance is evaluated with validation-set class IoU accuracy.
- 5.2 Semantic Segmentation: One training run reduces ERFNet's parameter count to one quarter of the original for a modest performance drop.The student uses the original ERFNet optimisation schedule and adds an attention-transfer term.
6 Conclusion
The paper concludes that cheap convolutions combined with distillation substantially reduce parameters with only a small performance drop, without requiring additional engineering. The larger model's optimisation algorithm is sufficient for training the cheaper student, while future work could explore more complex approximations.
- 6 Conclusion: Cheapening convolutions with distillation reduces network parameters substantially for a small performance drop in classification and semantic segmentation.The strategy is described as fast to apply and not requiring additional engineering.
- 6 Conclusion: The larger model's optimisation algorithm is sufficient to train the cheaper student model.
- 6 Conclusion: The chosen cheap convolutions prioritise ease of implementation, leaving approximate operations, architecture search, and low-rank methods as future alternatives.The paper specifically points to more complicated approximate operations and custom or low-rank blocks.