PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards High Performance Video Object Detection for Mobiles

Xizhou Zhu, Jifeng Dai, Xingchi Zhu, Yichen Wei, Lu Yuan

arXiv:1804.05830v1cs.CV

TL;DR

Mobile video detection faces real-time compute demands and video appearances that can challenge recognition, while desktop-oriented architectures are too heavy for mobile use. The paper combines sparse feature propagation, Light Flow, and flow-guided GRU aggregation in a lightweight end-to-end system, reporting 60.2% mAP at 25.6 fps on Huawei Mate 8.

  • Problem

    Mobile video detection must operate in real time on limited compute, but per-frame deep detection is costly and video appearances can challenge recognition.

    Source: §1 · p. 1

  • Method

    A lightweight end-to-end detector uses Light Flow for correspondence, propagates features on non-key frames, and aggregates features with a flow-guided GRU on sparse key frames.

    Source: §1 · p. 2

  • Results

    60.2% mAP at 25.6 fps on Huawei Mate 8; the authors report one order faster than the best previous fast-detection effort with on-par accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper reports that sparse feature propagation and multi-frame feature aggregation also hold under very limited computational overhead.

    Source: §6 · p. 15

    Online streaming may incur short latency because key-frame recognition is not yet fast enough.

    Source: §6 · p. 15

Abstract

from arXiv · show

Despite the recent success of video object detection on Desktop GPUs, its architecture is still far too heavy for mobiles. It is also unclear whether the key principles of sparse feature propagation and multi-frame feature aggregation apply at very limited computational resources. In this paper, we present a light weight network architecture for video object detection on mobiles. Light weight image object detector is applied on sparse key frames. A very small network, Light Flow, is designed for establishing correspondence across frames. A flow-guided GRU module is designed to effectively aggregate features on key frames. For non-key frames, sparse feature propagation is performed. The whole network can be trained end-to-end. The proposed system achieves 60.2% mAP score at speed of 25.6 fps on mobiles (e.g., HuaWei Mate 8).

1 Introduction

Mobile video detection must balance real-time operation on computationally limited devices with accuracy challenges from video-specific appearances. The paper adapts sparse feature propagation and feature aggregation for mobiles and reports 60.2% mAP at 25.6 fps.

1 Introduction

  • Video detection on computationally limited platforms must meet real-time needs, while applying deep networks on every frame incurs unaffordable cost and video appearances can impair recognition.Examples of challenging appearances include motion blur, defocus, and rare poses.Source: §1 · p. 1
  • Mobile deployment remains difficult because desktop-oriented video architectures use flow estimation too costly for real-time mobile processing and long-term aggregation restricted by runtime memory.The paper frames these as mobile-specific limits on existing approaches built around temporal information.Source: §1 · p. 2
  • The proposed mobile architecture propagates features on most non-key frames, aggregates features on sparse key frames with a flow-guided GRU, and uses Light Flow for correspondence.A lightweight image detector computes key-frame features; the techniques are unified in an end-to-end learning system.Source: §1 · p. 2
  • 60.2% mAP at 25.6 fps on Huawei Mate 8; the system is reported as one order faster than the best previous fast-detection effort, with on-par accuracy.The authors describe this as real-time mobile video object detection with reasonably good accuracy.Source: §1 · p. 2

2 Revisiting Video Object Detection Baseline

Video detection inherits the speed and appearance challenges of applying image detectors to every frame. Prior approaches address these with sparse feature propagation and multi-frame aggregation, while mobile use requires cheaper flow estimation and designs that fit limited compute and memory.

2 Revisiting Video Object Detection Baseline

  • Applying a single-image detector to every video frame is inefficient because its backbone is deep and slow, while motion blur, defocus, and rare poses can hurt detection accuracy.Image detectors extract convolutional feature maps with a backbone and then classify regions and regress boxes.Source: §2 · p. 3

Sparse Feature Propagation

  • Sparse feature propagation computes expensive features on sparse key frames and propagates them to most other frames using a flow network cheaper than the feature network.The cited Deep Feature Flow approach reports 5× speedup with minor accuracy drop; the passage describes propagation from a preceding key frame using bilinear warping.Source: Sparse Feature Propagation · p. 3

Multi-frame Feature Aggregation

  • Multi-frame aggregation aligns nearby-frame feature maps with flow and forms a weighted average, with weights based on similarity to the frame-i features.Prior work also applies recursive aggregation only on sparse key frames to retain feature quality while reducing computational cost.Source: Multi-frame Feature Aggregation · pp. 3–4

2.1 Practice for Mobiles

  • Mobile practice combines feature extraction and aggregation on sparse key frames with lightweight propagation on most other frames, but requires cheaper flow estimation.FlowNet uses 11.8× the FLOPs of MobileNet at the same input resolutions, and even FlowNet Inception uses 1.6× more FLOPs.Source: §2.1 · p. 4
  • Mobile aggregation should use flow-aligned features to avoid severe errors from large object displacements, while long-term dependencies can fuse more temporal information.The backbone should also be as small as possible because it computes features on sparse key frames.Source: §2.1 · p. 4

3 Model Architecture for Mobiles

The mobile architecture computes and aggregates features on sparse key frames, then propagates features to non-key frames. It combines a lightweight flow network with flow-guided GRU aggregation and a lightweight key-frame detector.

3 Model Architecture for Mobiles

  • Key frames are detected from flow-guided aggregated features, while non-key-frame detections use features propagated from the preceding key frame.The key-frame path aggregates its fresh features with the preceding aggregated features; the non-key-frame path warps the aggregated key-frame features before detection.Source: §3 · pp. 4–5

3.1 Light Flow

  • Light Flow is nearly 65× faster theoretically than FlowNet, with a 15% increase in end-point error.It replaces encoder convolutions with depthwise separable convolutions and decoder deconvolutions with upsampling followed by depthwise separable convolutions; averaged multi-resolution predictions reduce end-point error by nearly 10%.Source: §3.1 · pp. 5–6

Architecture and Implementation

  • Light Flow runs on half-resolution images with output stride 4, and propagation also covers intermediate detection-network features on non-key frames.The flow is downsampled to match the feature network’s stride 16; propagation is applied to intermediate 256-d RPN and 490-d Light-Head R-CNN features.Source: Architecture and Implementation · p. 6

3.2 Flow-guided GRU based Feature Aggregation

  • Flow-guided convolutional GRU aggregation models long-term dependencies across sparse key frames, aligning features to account for object displacement.The GRU uses 3 × 3 convolutions and ReLU; its key-frame inputs may be ten frames apart.Source: §3.2 · p. 8

3.3 Lightweight Key-frame Object Detector

  • The key-frame detector uses MobileNet features with RPN and Light-Head R-CNN for detection.MobileNet is pre-trained on ImageNet classification.Source: §3.3 · p. 8

Feature Network

  • The feature network removes MobileNet’s pooling and fully connected layers, then combines upsampled stride-32 features with stride-16 features.A 3 × 3 convolution reduces feature dimensions to 128 before nearest-neighbor upsampling; a 1 × 1 convolution processes the stride-16 maps.Source: Feature Network · p. 9

Detection Network

  • The detection network applies RPN and Light-Head R-CNN to shared 128-d features, using 256-d intermediate RPN maps.Light-Head R-CNN uses a 1 × 1 convolution followed by 7 × 7 position-sensitive RoI warping and sibling classification and regression layers.Source: Detection Network · p. 9

3.4 End-to-end Training

  • The feature, detection, and flow modules are jointly trained end-to-end using sampled nearby frames and a key-frame duration of 10.The experiments use eight key-frame samples; all operations are differentiable.Source: §3.4 · p. 9

4 Experiments

Experiments evaluate the mobile video detector through flow-network and feature-aggregation ablations, then assess its speed–accuracy trade-off across computational budgets.

4 Experiments

  • ImageNet VID evaluation uses 3,862 training and 555 validation snippets across 30 categories, with inference timed on a Huawei Mate 8 Cortex-A72 processor.Training also uses ImageNet DET; VID video frames and DET images are sampled at a 1:1 ratio.Source: §4 · pp. 9–10

Ablation on flow networks

  • Light Flow provides 65.2× theoretical speedup and 14.9× fewer parameters than FlowNet with a 15% relative EPE increase; detection reaches 61.2% versus 61.5% and runs one order faster.Table 2 covers optical-flow prediction on Flying Chairs and video detection on ImageNet VID.Source: Ablation on flow networks · p. 10, Table 2 · p. 11

Ablation on feature aggregation

  • Flow guidance surpasses no-flow guidance, with mAP gaps of 1.5% at l = 10 and 2.9% at l = 20.Accuracy also drops when flow is removed from sparse feature propagation on non-key frames.Source: Figure 3 · p. 11, Ablation on feature aggregation · p. 11
  • Training on longer sequences is beneficial but gains saturate at length 8; untrimmed inference is on par in accuracy with trimmed inference.The default trains on sequences of length 8 and applies the model to untrimmed sequences.Source: Ablation on feature aggregation · p. 11, Table 3 · p. 12
  • The [21] aggregation approach achieves 58.4% mAP, comparable to the single-frame baseline, but trails flow-guided GRU by 2.8% mAP at close computational overhead.Table 4 compares feature aggregation approaches.Source: Ablation on feature aggregation · pp. 11–12, Table 4 · p. 12
  • ReLU in the flow-guided GRU yields 3.9% higher mAP than tanh; widening from 128-d to 256-d adds 1.2%, while stacking 2 or 3 layers lowers accuracy.The authors report faster convergence with ReLU; increasing width is preferred over stacking layers when computation allows.Source: Ablation on feature aggregation · p. 12

Accurate Realtime Video Object Detection on Mobile

  • At l = 20, accuracy remains on par with the single-frame baseline at 10.6× greater computational efficiency; α = 1.0 favors adequate compute, while α = 0.5 suits limited compute.Figure 4 varies key-frame duration from 1 to 20 and compares networks of different complexity.Source: Accurate Realtime Video Object… · p. 12, Figure 4 · p. 13
  • 60.2% accuracy at 25.6 frames per second, and 51.2% accuracy at 50Hz, are reported for the proposed system on the mobile test platform.The 60.2% result uses α = 1.0, β = 0.5, l = 10; the 51.2% result uses α = 0.5, β = 0.5, l = 10.Source: Accurate Realtime Video Object… · pp. 12–13, Table 5 · p. 13

5 In Context of Previous Work on Mobile

This section reviews lightweight image detectors and mobile video-detection approaches, then compares detection systems on ImageNet VID. It describes the selected image detector and contrasts prior mobile systems with the proposed system.

5.1 Lightweight Image Object Detector

  • YOLO, SSD, and Light-head R-CNN [23] are highlighted for speed-accuracy trade-offs; this system integrates Light-head R-CNN, while other lightweight detectors may also apply.Light-head R-CNN is a two-stage detector that uses position-sensitive feature maps to reduce detector burden and is reported to outperform single-stage detectors on speed-accuracy.Source: §5.1 · pp. 13–14

5.2 Mobile Video Object Detector

  • Fast YOLO [51] copies key-frame boxes to other frames without feature aggregation, flow-guided warping, or end-to-end training; [44] aggregates dense-frame features but lacks sparse key frames and feature alignment.The cited works' performance is difficult to compare with the proposed system because their reported evaluation settings or results are not directly comparable.Source: §5.2 · pp. 14–15

5.3 Comparison on ImageNet VID

  • The comparison varies image input sizes for the systems and evaluates them at the detection-system level; Fast YOLO [51] is excluded because it reports no accuracy and has no public code.The reproduced SSDLite, Tiny YOLO, and YOLOv2 models were trained on ImageNet VID and ImageNet DET training sets.Source: §5.3 · p. 15
  • 60.2% accuracy at 25.6 fps; YOLOv2, SSDLite, and Tiny YOLO reach 58.7% at 0.3 fps, 57.1% at 3.8 fps, and 44.1% at 2.2 fps, respectively.The paper reports the comparison on ImageNet VID validation and describes its system as surpassing the existing systems by a clear margin.Source: §5.3 · p. 15

6 Discussion

The discussion reports that sparse feature propagation and multi-frame feature aggregation also hold under very limited computational overhead. It identifies online-streaming latency and the flow network's reduced input resolution as relevant implementation details.

6 Discussion

  • The proposed lightweight mobile detector combines Light Flow with flow-guided GRU aggregation and verifies sparse propagation and multi-frame aggregation under limited computational overhead.The flow network input resolution is half that of the image recognition network.Source: §6 · p. 15
  • Online streaming may incur short latency because key-frame recognition is not yet fast enough.Source: §6 · p. 15

Related papers

Loading 1804.05830v1…