An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection

Youngwan Lee; Joong-won Hwang; Sangrok Lee; Yuseok Bae; Jongyoul Park

Source-linked AI summary

An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection

Youngwan Lee, Joong-won Hwang, Sangrok Lee, Yuseok Bae, Jongyoul Park

arXiv:1904.09730v1Apr 22, 2019cs.CV

TL;DR

DenseNet offers strong multi-receptive-field features but can be slow and energy-inefficient because dense connections increase memory access and computation costs. The paper proposes VoVNet with One-Shot Aggregation, which aggregates features once, and reports faster, more energy-efficient detectors than DenseNet and ResNet.

Problem
DenseNet’s dense connections preserve diverse features with few parameters and FLOPs but produce slow, energy-inefficient object detectors through increasing memory access and computation costs.
Method
The paper proposes VoVNet, whose One-Shot Aggregation aggregates intermediate features once in the final feature map while retaining concatenative feature representation.
Results
VoVNet-based detectors outperform DenseNet- and ResNet-based detectors with faster speed and better energy efficiency across the evaluated detection settings.
Takeaways & Limitations
VoVNet provides an efficient backbone for real-time object detection while preserving diversified feature representation from multiple receptive fields.
Takeaways & Limitations
The OSA design relies on the hypothesis that dense connections induce correlated, redundant features and that intermediate-layer aggregation strength is negatively related to final-layer aggregation.

Abstract

from arXiv · show

As DenseNet conserves intermediate features with diverse receptive fields by aggregating them with dense connection, it shows good performance on the object detection task. Although feature reuse enables DenseNet to produce strong features with a small number of model parameters and FLOPs, the detector with DenseNet backbone shows rather slow speed and low energy efficiency. We find the linearly increasing input channel by dense connection leads to heavy memory access cost, which causes computation overhead and more energy consumption. To solve the inefficiency of DenseNet, we propose an energy and computation efficient architecture called VoVNet comprised of One-Shot Aggregation (OSA). The OSA not only adopts the strength of DenseNet that represents diversified features with multi receptive fields but also overcomes the inefficiency of dense connection by aggregating all features only once in the last feature maps. To validate the effectiveness of VoVNet as a backbone network, we design both lightweight and large-scale VoVNet and apply them to one-stage and two-stage object detectors. Our VoVNet based detectors outperform DenseNet based ones with 2x faster speed and the energy consumptions are reduced by 1.6x - 4.1x. In addition to DenseNet, VoVNet also outperforms widely used ResNet backbone with faster speed and better energy efficiency. In particular, the small object detection performance has been significantly improved over DenseNet and ResNet.

1. Introduction

The paper seeks to preserve DenseNet’s diversified feature representation while improving its efficiency for object detection. It identifies dense connections as a source of memory and GPU-computation inefficiency and proposes One-Shot Aggregation within VoVNet.

DenseNet preserves information from multiple receptive fields through concatenation, which is especially useful for detecting objects at varied scales.Concatenation preserves original feature information from different layers, whose receptive fields capture visual information at various scales.
DenseNet’s dense connections increase memory access cost quadratically with network depth, creating computation overhead and higher energy consumption.All previous feature maps feed subsequent layers, increasing intermediate-feature memory access as depth grows.
Linearly increasing input channels force DenseNet to use 1×1 bottlenecks, adding smaller sequential layers that reduce GPU-computation efficiency.The bottleneck limits input dimensions and FLOPs but fragments computation into smaller operations.
One-Shot Aggregation aggregates intermediate features once in the final feature map, targeting DenseNet’s efficiency problems while preserving concatenative aggregation.The paper applies OSA in VoVNet backbones across DSOD, RefineDet, and Mask R-CNN.
VoVNet-based detectors outperform DenseNet- or ResNet-based detectors with better speed and energy efficiency.The introduction states this conclusion across the evaluated detector families.

2. Factors of Efficient Network Design

The paper argues that efficient network design must account for memory access and GPU utilization, not only FLOPs and model size. It introduces MAC and FLOP/s as measures related to these deployment costs.

Reducing FLOPs and model size does not necessarily reduce GPU inference time or real energy consumption.The paper cites cases where models with similar or fewer FLOPs differ in speed and energy-related efficiency.
Memory access cost includes accesses involving filter parameters and intermediate feature maps, whose footprint can raise energy and time consumption.Intermediate activation memory can increase access cost even when model parameters remain unchanged.
MAC for a convolutional layer is modeled as hw(c_i + c_o) + k^2c_ic_o.The notation passage defines k as kernel size, h and w as input height and width, and c_i and c_o as input and output channels.
GPU-computation efficiency depends on how well an architecture utilizes parallel processing, with larger operand tensors generally improving utilization.Fragmenting computation into smaller operations can leave fewer computations running in parallel.
The paper measures GPU efficiency using FLOPs per Second, calculated by dividing total FLOPs by actual GPU inference time.Higher FLOP/s indicates more efficient execution of the same nominal computation.

3. Proposed Method

DenseNet’s dense connections preserve diverse features but create memory-access and GPU-computation inefficiencies. VoVNet’s One-Shot Aggregation (OSA) aggregates intermediate features only at each block’s final layer, reducing these costs while retaining strong feature representation.

Rethinking Dense Connection: DenseNet’s linearly increasing input channels create imbalanced layer dimensions, raising memory access cost and energy consumption.Under fixed computation or parameter count, memory access is minimized when input and output channel sizes are balanced.
Rethinking Dense Connection: DenseNet’s bottleneck layers reduce FLOPs and parameters but split computation into smaller tensors, limiting GPU parallel efficiency.The bottleneck structure compensates for computation that otherwise grows quadratically with network depth.
Rethinking Dense Connection: Trained DenseNet connectivity shows active intermediate aggregation in Dense Block3, whereas Dense Block1 relies more heavily on its transition layer.These patterns motivate examining whether intermediate dense connections provide sufficient benefit for their cost.
Rethinking Dense Connection: The authors hypothesize that dense intermediate connections produce correlated, redundant features, weakening the influence of earlier intermediate layers on the final layer.This motivates aggregating intermediate features only once at the final layer of each block.
One-Shot Aggregation: OSA connects each convolution layer to the next layer while aggregating features once into the final output feature map.Its intermediate layers retain constant input and output sizes, avoiding an additional 1×1 bottleneck and supporting efficient GPU computation.
One-Shot Aggregation: Replacing DenseNet-40’s dense block with a 5-layer, 43-channel OSA module reduces estimated MAC from 3.7M to 2.5M.The reduction results from equal intermediate input and output channel sizes, while the reported CIFAR-10 error rate remains similar to DenseNet-40: 5.44% versus 5.24%.

4. Experiments

Experiments evaluate VoVNet backbones across lightweight and large-scale one-stage and two-stage detectors, measuring accuracy, speed, GPU-computation efficiency, energy efficiency, and memory footprint. VoVNet generally achieves stronger accuracy–efficiency trade-offs than DenseNet and ResNet, while an ablation shows that 1×1 bottlenecks can reduce FLOPs but harm inference efficiency.

Lightweight models: VoVNet-27-slim based DSOD300 achieves 74.87% accuracy and runs two times faster than the comparable DenseNet-67 counterpart.It also outperforms Pelee by 3.97% at much faster speed.
Lightweight models: 400 GFLOP/s makes VoVNet-27-slim the highest GPU-computation-efficiency model among the compared lightweight detectors.The results indicate that FLOPs alone do not sufficiently reflect GPU inference time.
Lightweight models: 0.9J per image makes VoVNet-based DSOD 4.1× less energy-consuming than the compared DenseNet-based detector.Energy efficiency is evaluated in Joule/frame, where lower values are better.
Lightweight models: With two times better energy efficiency than MobileNet and Pelee, VoVNet exceeds their accuracy by 6.87% and 3.97%, respectively.Figure 3(d) places VoVNet in the most efficient region for combined GPU-computation and energy efficiency.
Large-scale models: VoVNet-39 achieves 33.5 AP at about twice DenseNet-161’s speed and exceeds ResNet-50 by 3.3% AP at comparable speed.It uses fewer parameters and less memory than DenseNet-161.
Large-scale models: VoVNet-39/57 achieve higher computation efficiency and consume 4.8J and 5.9J per image, respectively, with higher or comparable accuracy.Compared with DenseNet161, VoVNet-39 uses two times less energy at comparable accuracy.
Large-scale models: VoVNet improves small-object AP by 1.9% and 1.2% over DenseNet121 and DenseNet161, respectively.VoVNet and DenseNet obtain higher small- and medium-object AP than ResNet.
Mask R-CNN: In Mask R-CNN, VoVNet-39 improves detection AP by 2.2% and 0.9% over ResNet-50 and ResNet-101, and segmentation AP by 1.6% and 0.4%, respectively.VoVNet-57 also achieves state-of-the-art performance against ResNet-101 at faster inference speed.

5. Conclusion

VoVNet is proposed as an efficient backbone that preserves diversified multi-receptive-field features while improving DenseNet’s inefficiency. One-Shot Aggregation reduces memory access cost through a constant input size, enabling faster GPU computation.

VoVNet preserves diversified feature representations with multiple receptive fields while improving DenseNet’s inefficiency.
One-Shot Aggregation aggregates all features once in the final feature map instead of using linearly increasing dense-connection inputs.
Constant input size reduces memory access cost and makes GPU computation more efficient.
VoVNet-based detectors outperform DenseNet-based detectors at much faster speed across lightweight and large-scale models.

8. Appendix B: Qualitative comparisons

Figure 5 compares RefineDet320 detectors using VoVNet-57, DenseNet-161, and ResNet-101 on COCO minival images. VoVNet-57 detects small objects better than its counterparts.

Figure 5 compares RefineDet320 detectors with VoVNet-57, DenseNet-161, and ResNet-101 on the COCO minival dataset.The displayed boxes represent detections with confidence scores over 0.6.
VoVNet detectors outperform their DenseNet and ResNet counterparts in the qualitative comparison.
VoVNet-57 is especially strong at detecting small objects.The figure description specifically notes better small-object detection than the counterpart models.