PaperlayerView exact revision on arXiv

Source-linked AI summary

ThunderNet: Towards Real-time Generic Object Detection

Zheng Qin, Zeming Li, Zhaoning Zhang, Yiping Bao, Gang Yu, Yuxing Peng, Jian Sun

arXiv:1903.11752v3cs.CV

TL;DR

Real-time generic object detection on mobile devices is difficult because existing CNN detectors demand substantial computation, while lightweight one-stage detectors leave an accuracy gap. ThunderNet investigates efficient two-stage detection through a lightweight backbone, compressed detection components, and feature-enhancement modules, achieving stronger accuracy-efficiency trade-offs and real-time ARM inference.

  • Problem

    Mobile object detection requires real-time inference under strict computational constraints, but existing CNN detectors are resource-hungry and lightweight one-stage detectors do not provide an ideal accuracy-speed trade-off.

  • Method

    ThunderNet is a lightweight two-stage detector combining an object-detection-specific SNet backbone, compressed RPN and detection head, and Context Enhancement and Spatial Attention Modules.

  • Results

    ThunderNet surpasses prior lightweight one-stage detectors on PASCAL VOC and COCO with significantly less computational cost, reaching 24.1 fps on ARM.

  • Takeaways & Limitations

    The results support the effectiveness of two-stage detectors for real-time generic object detection on mobile-oriented platforms.

Abstract

from arXiv · show

Real-time generic object detection on mobile platforms is a crucial but challenging computer vision task. However, previous CNN-based detectors suffer from enormous computational cost, which hinders them from real-time inference in computation-constrained scenarios. In this paper, we investigate the effectiveness of two-stage detectors in real-time generic detection and propose a lightweight two-stage detector named ThunderNet. In the backbone part, we analyze the drawbacks in previous lightweight backbones and present a lightweight backbone designed for object detection. In the detection part, we exploit an extremely efficient RPN and detection head design. To generate more discriminative feature representation, we design two efficient architecture blocks, Context Enhancement Module and Spatial Attention Module. At last, we investigate the balance between the input resolution, the backbone, and the detection head. Compared with lightweight one-stage detectors, ThunderNet achieves superior performance with only 40% of the computational cost on PASCAL VOC and COCO benchmarks. Without bells and whistles, our model runs at 24.1 fps on an ARM-based device. To the best of our knowledge, this is the first real-time detector reported on ARM platforms. Our code and models are available at \url{https://github.com/qinzheng93/ThunderNet}.

1. Introduction

Real-time object detection on mobile devices is constrained by detector computation, while lightweight one-stage models face accuracy limitations. ThunderNet addresses this gap with an efficient two-stage design and reports stronger accuracy-efficiency trade-offs, including real-time ARM inference.

  • Motivation: Mobile devices impose strict computational limits, while CNN detectors require substantial computation for high detection accuracy.These constraints hinder real-time inference in mobile scenarios.
  • Motivation: Two-stage detectors use RPN-generated regions followed by a detection head, but their detection parts are typically too expensive for mobile devices.Heavy detection parts can exceed 10 GFLOPs; lightweight heads may remain mismatched with small backbones.
  • Motivation: Lightweight one-stage detectors are computationally attractive but produce coarser results and have not achieved an ideal accuracy-speed trade-off on mobile devices.This motivates testing whether efficient two-stage detectors can surpass them in real-time detection.
  • ThunderNet design: ThunderNet combines an object-detection-specific SNet backbone with compressed RPN and detection-head components.Its design targets computationally expensive structures in existing two-stage detectors.
  • ThunderNet design: Context Enhancement and Spatial Attention Modules improve feature representation by aggregating multi-scale context and refining feature distribution with RPN information.The modules address performance degradation associated with small backbones and feature maps.
  • Results: 24.1 fps on ARM and 47.3 fps on x86 were reported with MobileNet-SSD level accuracy, while ThunderNet outperformed Tiny-DSOD with 42% of its computational cost.The paper reports these results on PASCAL VOC and COCO benchmarks.

2. Related Work

Prior work primarily developed CNN detectors as either two-stage or one-stage systems, with different accuracy and efficiency profiles. ThunderNet extends this literature by focusing on an efficient two-stage detector for real-time detection.

  • CNN-based object detectors: CNN-based object detectors are commonly classified into two-stage and one-stage detectors.Two-stage systems include R-CNN, Faster R-CNN, and R-FCN, while one-stage systems include SSD and YOLO.
  • CNN-based object detectors: Faster R-CNN introduced an RPN for generating region proposals, and R-FCN shared computation across the entire image.These designs improved detection efficiency or architecture sharing within two-stage detection.
  • CNN-based object detectors: One-stage detectors such as SSD and YOLO achieved real-time GPU inference with competitive accuracy, while RetinaNet used focal loss to address foreground-background imbalance.The related work establishes one-stage detection as an important efficiency-oriented direction.
  • Real-time generic object detection: ThunderNet presents an efficiency-focused two-stage detector rather than another one-stage design.Its stated focus is reducing the computational cost of two-stage detection.

3. ThunderNet

ThunderNet is a lightweight two-stage detector designed for efficient mobile inference. Its SNet backbone and compressed detection components are complemented by CEM and SAM to recover context and feature quality under lightweight constraints.

  • Input Resolution: ThunderNet uses a 320×320 input resolution to improve inference speed.The design also matches input resolution to backbone capability rather than pairing a small backbone with large inputs or vice versa.
  • Backbone Networks: SNet replaces 3×3 depthwise convolutions with 5×5 convolutions, enlarging the theoretical receptive field from 121 to 193 pixels.SNet49 targets faster inference, SNet535 targets better accuracy, and SNet146 targets a speed/accuracy trade-off.
  • Detection Head: ThunderNet compresses RPN and the detection head with depthwise convolution, reduced channels, PSRoI align, and a narrower pre-RoI feature map.The thin feature map uses α = 5 instead of α = 10, while the R-CNN subnet uses a 1024-d fully connected layer.
  • Context Enhancement Module: CEM aggregates local and global context from three scales using two 1×1 convolutions and a fully connected layer.It merges C4, C5, and globally pooled C5 features to enlarge the receptive field and refine thin-feature representations with lower computation than prior FPN structures.
  • Spatial Attention Module: SAM re-weights CEM features using intermediate RPN features before RoI warping.Its inputs are the RPN feature map and CEM thin feature map; a 1×1 convolution matches their channel dimensions and sigmoid constrains re-weighting values to [0, 1].

4. Experiments

The experiments evaluate ThunderNet on PASCAL VOC and COCO, followed by ablation studies of its design.

  • ThunderNet is evaluated on the PASCAL VOC and COCO benchmarks.
  • The experiments include ablation studies to evaluate ThunderNet’s design.
  • The evaluation covers both benchmark performance and component effectiveness.

4.1. Implementation Details

ThunderNet is trained end-to-end with synchronized SGD and efficiency-oriented settings, including small input resolutions and multi-scale training.

  • ThunderNet is trained end-to-end on 4 GPUs with synchronized SGD, weight decay 0.0001, and momentum 0.9.The batch size is 16 images per GPU.
  • 320×320-pixel inputs replace the 600× or 800× resolutions common in large two-stage detectors.Multi-scale training uses 240, 320, and 480 pixels, with heavy data augmentation for the small input resolution.
  • The networks are trained for 62.5K iterations on VOC and 375K iterations on COCO.The learning rate starts at 0.01 and decays by 0.1 at 50% and 75% of total iterations.

4.2. Results on PASCAL VOC

On PASCAL VOC, ThunderNet outperforms prior lightweight one-stage detectors and achieves results superior to several large detectors while using substantially less computation.

  • ThunderNet with SNet49 outperforms MobileNet-SSD using merely 21% of its FLOPs.
  • 2.9 mAP higher performance than Tiny-DSOD is achieved by the SNet146-based model with about 43% of the FLOPs.
  • ThunderNet with SNet146 performs 6.5 mAP better than Tiny-DSOD under similar computational cost.
  • ThunderNet exceeds YOLOv2, SSD300*, SSD321, and R-FCN, matches DSSD321, and reduces computational cost by orders of magnitude.The paper characterizes this as a better trade-off between accuracy and efficiency despite ThunderNet’s weaker and smaller backbone.

4.3. Results on MS COCO

On COCO test-dev, ThunderNet matches or exceeds competing detectors while using substantially less computation. Its stronger AP75 results indicate more accurate localization than several alternatives.

  • Less than 40% of the computational cost lets ThunderNet with SNet146 surpass MobileNet-SSD, MobileNet-SSDLite, and Pelee.Its considerably better AP75 suggests stronger localization.
  • 42% of the FLOPs gives ThunderNet better AP but worse AP50 than Tiny-DSOD.The authors conjecture that Tiny-DSOD’s deep supervision and feature pyramid improve classification accuracy, while ThunderNet remains better in localization.
  • At comparable computational cost, ThunderNet with SNet535 achieves significantly better detection accuracy than other one-stage counterparts.It surpasses them by at least 4.8 AP, 5.8 AP50, and 6.7 AP75.
  • ThunderNet with SNet146 surpasses YOLOv2 with 37× fewer FLOPs, while SNet535 significantly outperforms YOLOv2 and SSD300 and rivals SSD321.

4.4. Ablation Experiments

The ablations show that accuracy and efficiency depend on balancing input resolution, backbone capacity, and detection-head complexity. CEM, SAM, and compact detection components improve the lightweight two-stage design.

  • 4.4.1 Input Resolution: Large backbones with small images and small backbones with large images are both nonoptimal, revealing a trade-off between representation capacity and feature-map resolution.The backbone and input images should be matched for balance.
  • 4.4.2 Backbone Networks: 5×5 depthwise convolutions reduce detection performance by 0.9 AP when replaced with 3×3 convolutions at unchanged computational cost.The authors attribute the benefit to larger receptive fields.
  • 4.4.2 Backbone Networks: Adding a wide Conv5 improves classification slightly but reduces detection AP by 0.4, indicating that detection also benefits from early-stage detail features.
  • 4.4.2 Backbone Networks: Removing Conv5 from SNet49 severely degrades classification and detection because the backbone output channels are cut by half.
  • 4.4.2 Backbone Networks: SNet146 surpasses Xception, MobileNetV2, and ShuffleNet variants on object detection under similar FLOPs.
  • 4.4.3 Detection Part: Replacing the RPN convolution with depthwise and pointwise convolutions cuts computational cost by 28% without harming accuracy, while halving the R-CNN fully connected layer adds 13% compression with a 0.2 AP decrease.
  • 4.4.3 Detection Part: CEM improves AP by 1.7, AP50 by 2.5, and AP75 by 1.8 with negligible FLOPs increase by combining multi-scale feature maps.
  • 4.4.3 Detection Part: SAM improves AP by 1.4 with only 5% extra computational cost and refines features by enhancing foreground regions while weakening background regions.

4.5. Inference Speed

ThunderNet achieves real-time inference on ARM and CPU with SNet49, while all evaluated models exceed 200 fps on GPU. SNet146 remains real-time on CPU but not ARM.

  • 24.1 fps on ARM and 47.3 fps on CPU make ThunderNet with SNet49 real-time on both platforms.The evaluation used single-thread inference on ARM and CPU.
  • 13.8 fps on ARM and 32.3 fps on CPU are achieved by ThunderNet with SNet146.All three evaluated models run at over 200 fps on GPU.

5. Conclusion

ThunderNet is presented as a lightweight two-stage detector designed for real-time generic object detection. Its design combines an object-detection-oriented backbone, efficient detection components, and feature-enhancement modules.

  • ThunderNet investigates two-stage detectors for real-time generic object detection and introduces a lightweight detector for this setting.
  • SNet is designed for object detection, while the RPN and detection head are compressed for efficiency.
  • CEM and SAM improve feature representation, and the paper studies balance among input resolution, backbone, and detection head.
  • ThunderNet achieves higher detection accuracy than prior one-stage detectors with significantly less computational cost and reports real-time single-thread inference on ARM.
Loading 1903.11752v3…