Source-linked AI summary
From Deep to Shallow: Unconstrained and Efficient Layer Merging Strategy
Petro Shulzhenko, Gabriele Spadaro, Enzo Tartaglione
TL;DR
Deep-network depth compression is limited when convolution merging lacks a closed-form solution or increases kernel size, weakening practical speed-ups. The paper learns compact replacement layers under guidance from a linearized reference model, and experiments report lower latency with minimal or no accuracy degradation across architectures, datasets, and hardware platforms.
Problem
Depth compression must merge linearized layers, but padding can eliminate analytical solutions and larger merged kernels can limit inference speed-up.
Method
The method uses a loss-based strategy that learns merged layers from a linearized Reference model, combining classification, distillation, and activation-alignment losses.
Results
Across architectures, datasets, and hardware platforms, the Shrunk model reduces inference latency with minimal or no accuracy degradation and sometimes improves over the Reference model.
Takeaways & Limitations
The approach provides a practical solution for deploying compressed deep networks on resource-constrained devices.
Abstract
from arXiv · showhide
Although Deep Neural Networks have become foundational in many areas of Machine Learning, high computational demands limit their application in resource-constrained environments. To address this issue, depth compression methods have been proposed to identify and linearize redundant activation functions, thereby allowing for the merging of layers without intermediate non-linearities. However, these methods face two key challenges: they cannot be directly applied to convolutions with padding due to the absence of an analytical solution for merging these layers, and they typically increase the kernel size of merged layers, thus limiting speed-up gains. To overcome these limitations, we propose an efficient strategy that enables merging of layers without an existing analytical solution, and also without increasing kernel size. We validate our approach across multiple architectures and datasets, and measure inference speed-up gains on real embedded platforms. We publicly released the code at https://github.com/ShulzhenkoPetr/deep-to-shallow.
1 Introduction
The paper targets the computational cost of deep networks by making depth compression produce practical inference acceleration despite convolution-merging limitations. Its loss-based strategy supports broader configurations, avoids kernel growth, and preserves accuracy while reducing latency across devices.
- Deep networks’ depth and size increase latency, hardware costs, energy consumption, and environmental impact, limiting deployment in resource-constrained settings.
- Existing depth-compression methods linearize redundant activations, but padding can prevent analytical merging and larger equivalent kernels can reduce or cancel speed-up.
- The proposed loss-based strategy learns compact replacement layers from a linearized Reference model, supporting padding, stride, dilation, and varied kernel sizes without kernel growth.
- The method achieves performance close to analytical merging when available, including with smaller and more efficient kernels.
- Experiments span multiple architectures, datasets, and hardware platforms, with inference speed-ups on NVIDIA GeForce RTX 2080 Ti, NVIDIA Jetson Orin, and Raspberry Pi 5 with marginal or no accuracy loss.
2 Related Work
Related work reduces depth by linearizing activations and collapsing surrounding operations, but practical acceleration depends on how layers are merged. The proposed method learns flexible compact replacements under reference-model guidance to address unavailable closed-form solutions and kernel growth.
- Depth-compression methods replace redundant nonlinearities with identity mappings, allowing surrounding linear operations to be collapsed into shallower models.
- Existing methods differ mainly in how they identify removable activations, including trainable parameters, masks, entropy, and other criteria.
- Analytical convolution merging is restricted by padding configurations and generally increases effective kernel size, which may reduce or cancel expected speed-up.
- The proposed approach learns merged layers under reference-model guidance, enabling globally aware merging without closed-form solutions and allowing compact replacement configurations.
3 Method
The method learns compact replacement layers for consecutive linear operations separated by identity activations, including convolutional configurations that lack closed-form merging solutions. It guides training with both local activation alignment and global prediction objectives while allowing the replacement configuration to balance efficiency and accuracy.
- Method overview: The method merges consecutive linear or convolutional layers separated by activations replaced with identity functions.It is applied after an arbitrary layer-collapse algorithm has linearized selected activations.
- Method overview: Analytical convolution merging is restricted by padding and can increase the effective kernel size, whereas the proposed method learns the merged operation.When the second convolution uses non-zero padding, no input-independent closed-form solution is known.
- Training procedure: Training freezes the Reference and Shrunk models except for merged layers, reducing the number of optimized parameters.Batch-normalization layers remain in evaluation mode with fixed running statistics.
- Training objective: The objective combines classification, distillation, and activation-alignment losses to preserve task predictions and local feature behavior.The distillation term aligns final logits, while activation alignment targets local representations across merging blocks.
- Replacement layers: The Shrunk model replaces each compressed block with a single trainable layer whose size is selected according to the desired efficiency–accuracy trade-off.For convolutional blocks, the replacement uses the original block size or max(kA, kC), with stride and padding adjusted for architectural compatibility.
4 Experiments
Experiments across architectures, datasets, linearization methods, kernel sizes, and hardware show that loss-based layer merging preserves accuracy while reducing latency without kernel growth. Global objectives outperform local alignment alone, especially when compact merged kernels must recover task performance.
- Experimental setup: The method was evaluated on ResNet18, ResNet50, MobileNetv2, and Swin-T using CIFAR-10 and ImageNet-1k across GPU, edge, and CPU devices.Experiments used multiple architectures, datasets, and hardware platforms.
- Preliminary experiment: A loss-based 3 × 3 merge nearly matches the analytical 5 × 5 solution in the preliminary ResNet18 experiment while using a smaller kernel.The comparison uses CIFAR-10 accuracy under analytical and loss-based merging.
- Main results: On ImageNet, the proposed strategy preserves or slightly improves Reference accuracy while reducing latency on every tested device.ResNet50 reaches 75.08 from 74.93 with a 30 ms Raspberry Pi 5 reduction, while ResNet18 gains 1.1 points and saves 17 ms.
- Linearization methods: Merged models trained with the proposed strategy achieve accuracy in line with or above Reference models across all tested activation linearization methods.The comparison covers four linearization strategies on CIFAR-10 ResNet18 models with two merged convolutional blocks.
- Kernel size: 3 × 3 merged kernels provide 1.12–1.15× speed-ups on compute-bound Raspberry Pi 5 and Jetson Nano devices, whereas analytical 5 × 5 kernels provide none and can be slower.The effect is smaller on launch-bound Jetson Orin, showing why avoiding kernel growth matters for constrained compute-bound hardware.
- Training objective: Activation-map loss alone collapses four-block CIFAR-10 models to 10.00%, while composite objectives restore accuracy to within 0.4 points of the 92.32% Reference.Distillation helps when the Reference is strong, while cross-entropy helps compensate for weak Reference guidance.
5 Conclusion
The proposed layer-merging method supports analytically unavailable configurations and avoids kernel-size growth. Across architectures, datasets, and hardware, it reduces inference latency with minimal or no accuracy degradation, sometimes improving over the Reference model.
- Method: The method learns merged layers for depth-compressed networks when analytical merging is unavailable and avoids increasing kernel size.It is designed for practical layer merging after activation linearization.
- Evidence: Experiments across architectures, datasets, and hardware show reduced inference latency with minimal or no accuracy degradation.The conclusion reports the overall empirical pattern rather than a single benchmark.
- Outcome: In some cases, the Shrunk models improve accuracy over the Reference model.This outcome is included in the paper’s overall conclusion.