Source-linked AI summary
ESPNetv2: A Light-weight, Power Efficient, and General Purpose Convolutional Neural Network
Sachin Mehta, Mohammad Rastegari, Linda Shapiro, Hannaneh Hajishirzi
TL;DR
ESPNetv2 targets the high resource demands of CNNs used on edge devices for visual and sequential data. It combines group point-wise and depth-wise dilated separable convolutions to learn from a large effective receptive field with fewer FLOPs and parameters. Across classification, segmentation, detection, and language modeling, it reports strong performance and improved efficiency relative to prior methods.
Problem
CNNs used in real-world edge applications require substantial computation, memory, and power, despite operating under resource and latency constraints.
Method
ESPNetv2 is a lightweight general-purpose architecture using group point-wise and depth-wise dilated separable convolutions to model visual and sequential data efficiently.
Results
ESPNetv2 delivers state-of-the-art performance across object classification, detection, segmentation, and language modeling while using fewer FLOPs and being more power efficient.
Takeaways & Limitations
ESPNetv2 is designed as a broadly applicable, power-efficient network for deployment on edge devices across visual recognition and sequence-modeling tasks.
Takeaways & Limitations
ERU performance can be further improved through rigorous hyper-parameter search and advanced dropout methods.
Abstract
from arXiv · showhide
We introduce a light-weight, power efficient, and general purpose convolutional neural network, ESPNetv2, for modeling visual and sequential data. Our network uses group point-wise and depth-wise dilated separable convolutions to learn representations from a large effective receptive field with fewer FLOPs and parameters. The performance of our network is evaluated on four different tasks: (1) object classification, (2) semantic segmentation, (3) object detection, and (4) language modeling. Experiments on these tasks, including image classification on the ImageNet and language modeling on the PenTree bank dataset, demonstrate the superior performance of our method over the state-of-the-art methods. Our network outperforms ESPNet by 4-5% and has 2-4x fewer FLOPs on the PASCAL VOC and the Cityscapes dataset. Compared to YOLOv2 on the MS-COCO object detection, ESPNetv2 delivers 4.4% higher accuracy with 6x fewer FLOPs. Our experiments show that ESPNetv2 is much more power efficient than existing state-of-the-art efficient methods including ShuffleNets and MobileNets. Our code is open-source and available at https://github.com/sacmehta/ESPNetv2
1. Introduction
ESPNetv2 addresses the mismatch between resource-intensive CNNs and the low-latency, limited-resource requirements of edge devices. It introduces a lightweight architecture intended for efficient visual and sequential modeling across multiple tasks.
- Motivation: Edge applications such as self-driving cars and robots require low-latency online processing on resource-constrained devices.These applications motivate efficient CNN designs for practical deployment.
- Motivation: ResNet-50 uses 25.56 million parameters, 98 MB of memory, and 2.8 billion high precision operations per image.The resource demands increase further for deeper CNNs such as ResNet-101.
- Prior Approaches: Existing efficiency strategies include network compression, low-bit representations, and lightweight architectural designs.Compression prunes redundancies, while low-bit methods reduce weight precision without changing network structure.
- Contribution: ESPNetv2 is introduced as a lightweight architecture that can be deployed on edge devices and model visual and sequential data.Its performance is demonstrated across tasks ranging from object classification to language modeling.
- Contribution: ESPNetv2 uses depth-wise dilated separable convolutions and reports improved efficiency and accuracy across visual recognition tasks.The introduction reports 2% higher accuracy than MobileNetv2 at 28 MFLOPs, 4–5% improvement over ESPNet with 2–4× fewer FLOPs on PASCAL VOC and Cityscapes, and further object-detection gains over YOLOv2.
2. Related Work
Prior efficient-network research reduces computation through factorized or grouped convolutions, channel operations, architecture search, pruning, and quantization. These approaches are described as complementary to ESPNetv2.
- Efficient CNN Architectures: Depth-wise separable and group convolutions reduce computational complexity in efficient CNN architectures.Depth-wise convolution filters each input channel separately, while point-wise convolution learns channel combinations.
- Efficient CNN Architectures: Channel shuffle and channel split can further improve network efficiency and accuracy, but are described as orthogonal to ESPNetv2.They complement convolutional factorization rather than replacing it.
- Neural Architecture Search: Neural architecture search explores network spaces containing different layers, units, parameters, and filter sizes.The cited search-based methods improved MobileNetv2 and are considered complementary to ESPNetv2.
- Network Compression: Network compression improves inference by pruning connections or channels, exploiting redundant CNN weights and parameter sparsity.The passage notes that sparse implementations can be difficult to execute efficiently on CPUs because of lookup and data-migration costs.
- Low-Bit Representation: Low-bit representation improves inference by quantizing weights with fewer bits than 32-bit high-precision floating-point values.Like compression methods, low-bit representations are presented as complementary to ESPNetv2.
3. ESPNetv2
ESPNetv2 builds an efficient EESP unit from group point-wise and depth-wise dilated separable convolutions, preserving large receptive fields while reducing computation. The architecture also uses hierarchical fusion and long-range shortcuts to support efficient multi-scale representation learning.
- Depth-wise dilated separable convolution: Depth-wise dilated separable convolutions factor standard convolution into per-channel dilated filtering and point-wise channel mixing.The dilation rate expands the effective receptive field to n_r × n_r, where n_r = (n−1)·r + 1.
- EESP unit: EESP replaces point-wise convolutions with group point-wise convolutions and standard dilated convolutions with depth-wise dilated separable convolutions.Hierarchical feature fusion additively combines branch outputs to remove gridding artifacts.
- EESP unit: 7× fewer parameters are learned by EESP than ESP when M=240, g=K=4, and d=M.The comparison follows the parameter expression involving group count, branch count, and branch width.
- EESP unit: Group point-wise convolution has the same complexity as independently computing K point-wise convolutions but uses one convolutional kernel instead of K.This implementation reduces kernel-launch overhead while preserving the stated complexity.
- Strided EESP unit: Strided EESP units replace dilated convolutions with strided versions, add average pooling, and concatenate outputs for efficient down-sampling.The average-pooled input is matched to feature-map dimensions, while concatenation expands feature-map dimensions efficiently.
- Strided EESP unit: A long-range shortcut connects the input image to each down-sampling unit by down-sampling it and applying standard 3 × 3 and point-wise convolutions.The first convolution learns spatial representations, and the second learns channel-wise linear combinations before projection to a high-dimensional space.
4. Experiments
ESPNetv2 is evaluated across classification, segmentation, detection, multi-label classification, power efficiency, and language modeling. Across these experiments, it targets competitive accuracy with reduced computation, latency, parameters, or power consumption.
- ESPNetv2 is evaluated on object classification, semantic segmentation, object detection, and language modeling.
- Image classification: Cyclic learning with warm restarts is used for ImageNet training, with ηmin = 0.1, ηmax = 0.5, and T = 5.Networks are trained for 300 epochs with batch size 512 and scheduled learning-rate decays.
- Image classification: At 28 million FLOPs, ESPNetv2 outperforms MobileNetv1 by 10% and MobileNetv2 by 2%.
- Image classification: At about 300 million FLOPs, ESPNetv2 is 1.1% more accurate than CondenseNet and delivers competitive performance against other efficient networks.
- Efficiency analysis: ESPNetv2 has slightly lower inference speed than ShuffleNetv2 but is much more power efficient while maintaining similar ImageNet accuracy.The comparison uses NVIDIA GTX 1080 Ti and Jetson TX2 devices.
- Semantic segmentation: On semantic segmentation, ESPNetv2 is 2–3% less accurate than ICNet, ERFNet, and ContextNet but uses 9–12× fewer FLOPs.Under similar computational constraints, it outperforms ENet and ESPNet by a large margin.
- Object detection: For object detection, ESPNetv2 matches YOLOv2 performance while using 25× fewer FLOPs.It also provides competitive performance relative to SSD.
- Language modeling: ERUs achieve similar or better language-modeling performance than state-of-the-art recurrent networks while learning fewer parameters.ERU replaces LSTM input linear transforms with the EESP unit.
5. Ablation Studies on the ImageNet Dataset
The ablations show that depth-wise dilated separable convolutions, hierarchical feature fusion, long-range shortcuts, and cyclic or longer training improve ESPNetv2 performance with limited complexity costs.
- Impact of different convolutions: Depth-wise dilated separable convolutions are more effective than dilated and depth-wise convolutions.
- Impact of hierarchical feature fusion (HFF): HFF improves classification performance by about 1.5% without affecting network complexity.It also removes gridding artifacts and shares information between EESP branches.
- Impact of long-range shortcut connections with the input: Long-range shortcut connections improve performance by about 1% with little or negligible impact on network complexity.
- Fixed vs cyclic learning schedule: Cyclic learning produces about 1% higher top-1 validation accuracy than a fixed schedule on ImageNet.
- Fixed vs cyclic learning schedule: Training ESPNetv2 for 300 epochs improves performance by about 4% under the stated learning schedule.
6. Conclusion
ESPNetv2 is presented as a light-weight, power-efficient general-purpose network that encodes spatial information using a large effective receptive field and supports visual and sequential tasks.
- ESPNetv2 encodes image spatial information using representations learned from a large effective receptive field.
- The network generalizes across object classification, detection, segmentation, and language modeling while remaining power efficient.