Source-linked AI summary
Towards High Performance Video Object Detection
Xizhou Zhu, Jifeng Dai, Lu Yuan, Yichen Wei
TL;DR
Video object detection remains challenging because processing every frame is computationally expensive and video appearances can deteriorate. The paper unifies multi-frame end-to-end feature and motion learning with three adaptive techniques, achieving 77.8% mAP at 15.22 frames per second and establishing a new state of the art.
Problem
Video detection has received limited attention despite the computational cost of processing all frames and accuracy challenges from blur, defocus, and rare poses.
Method
The paper unifies prior multi-frame approaches with sparsely recursive aggregation, spatially-adaptive partial feature updating, and temporally-adaptive key-frame scheduling.
Results
77.8% mAP at 15.22 frames per second establishes a new state of the art.
Takeaways & Limitations
The three techniques steadily advance the video-detection speed-accuracy trade-off toward high-performance detection.
Takeaways & Limitations
Comparison fairness is limited because methods use different recognition networks, object detectors, and post-processing techniques.
Abstract
from arXiv · showhide
There has been significant progresses for image object detection in recent years. Nevertheless, video object detection has received little attention, although it is more challenging and more important in practical scenarios. Built upon the recent works, this work proposes a unified approach based on the principle of multi-frame end-to-end learning of features and cross-frame motion. Our approach extends prior works with three new techniques and steadily pushes forward the performance envelope (speed-accuracy tradeoff), towards high performance video object detection.
1. Introduction
Video object detection must address both the computational cost of processing every frame and appearance deterioration absent from many still images. Building on multi-frame end-to-end learning, the paper introduces three techniques and reports a new speed-accuracy state of the art.
- Video detectors face unaffordable computation when deep networks run on every frame and reduced accuracy under blur, defocus, and rare poses.
- Prior work exploits redundancy between adjacent frames for speed through sparse feature propagation and improves recognition through dense temporal feature aggregation.These approaches motivated the ImageNet Video Object Detection Challenge 2017 winner.
- Sparse propagation saves computation but uses approximate, error-prone features, whereas dense aggregation improves quality but is slowed by repeated motion estimation, propagation, and aggregation.
- Sparsely recursive feature aggregation operates on sparse key frames to retain aggregation quality while reducing computational cost.The technique combines the stated merits of the two prior approaches.
- Spatially-adaptive partial feature updating recomputes non-key-frame features where propagated features are poor, while temporally-adaptive scheduling selects key frames according to predicted feature quality.The updating technique further improves recognition accuracy, and scheduling makes key-frame usage more efficient.
- 77.8% mAP at 15.22 frames per second establishes a new state of the art for the reported speed-accuracy trade-off.
2. From Image to Video Object Detection
Video detection extends image detection's feature-extraction and detection pipeline with motion-based propagation or temporal aggregation. Sparse propagation reduces computation, while dense aggregation improves features but incurs substantial runtime costs.
- Image detectors extract convolutional feature maps with a deep, slow feature network and generate detections with a shallow, fast detection network.The networks are denoted Nfeat(I) = F and Ndet(F) = y, respectively.
- Sparse feature propagation computes features only on sparse key frames and warps them to non-key frames for detection.This avoids computing the expensive feature network on every frame.
- A lightweight flow network estimates the motion field, and end-to-end training of all modules improves accuracy while sparse propagation can be up to 10× faster than single-frame detection.
- Dense feature aggregation evaluates the feature network on every frame, warps nearby-frame features into each target frame, and combines them using a weighted average.Every frame is treated as a key frame within the temporal window.
- Dense aggregation improves detection accuracy by about 3 mAP points overall and about 6 mAP points for fast-moving objects, but runs about 3 times slower.
3. High Performance Video Object Detection
The paper unifies motion-based video detection methods around efficient feature computation and feature-quality preservation. Three techniques combine sparse recursive aggregation, spatially selective updating, and adaptive key-frame scheduling to improve the runtime–accuracy trade-off.
- Unified viewpoint: The baseline methods are complementary: Sparse Feature Propagation reduces computation through approximation, whereas Dense Feature Aggregation improves feature quality at higher computational cost.Both methods use motion estimation and end-to-end multi-frame learning, but trade accuracy against speed differently.
- Sparsely Recursive Feature Aggregation: Sparsely Recursive Feature Aggregation computes features and recursively aggregates them only on sparse key frames.Aggregated features retain information from historical key frames before propagation to the next key frame.
- Spatially-adaptive Partial Feature Updating: Spatially-adaptive Partial Feature Updating recomputes non-key-frame features where propagated features are predicted to be inconsistent with real features.A learned temporal-consistency indicator identifies locations whose propagated features are poor approximations.
- Temporally-adaptive Key Frame Scheduling: Temporally-adaptive key-frame scheduling marks a frame as key when the area with poor propagation quality exceeds a fraction γ of all locations.This adapts feature-network evaluation to changing temporal dynamics instead of using a fixed key-frame interval.
- Scheduling analysis: 80.9% mAP at 22.8 fps is achieved by an oracle key-frame schedule, indicating that scheduling remains an important direction for improvement.The oracle uses ground-truth information to choose whether each frame should be key.
- Unified viewpoint: The unified method integrates all three techniques, while alternative settings degenerate to the two baselines or the per-frame detector.The integrated configuration is reported as the best-performing version in the unified inference framework.
4. Related Work
Video object detection methods incorporate temporal information either at the box level or feature level, with feature-level methods offering principled temporal modeling but relying on optical flow. This work belongs to the feature-level family and targets inaccurate feature propagation.
- Speed/accuracy trade-offs: The proposed method computes higher-quality features faster using temporal information without targeting a specific feature or detection network.Consequently, speed-oriented techniques such as changing networks or critical detection parameters can also be applied to it.
- Video object detection: Video object detection methods are divided into box-level and feature-level approaches based on where temporal information is incorporated.Feature-level methods include DFF and FGFA, while box-level methods improve consistency among tracked detections.
- Box-level methods: Box-level methods improve accuracy by propagating and rescoring bounding boxes or constructing sequences across consecutive frames.T-CNN uses optical flow and tracking to form tubelets, whereas Seq-NMS links nearby high-confidence boxes.
- Feature-level methods: Feature-level methods use optical flow to establish pixel-to-pixel correspondence among nearby frames before feature propagation or aggregation.These methods can additionally be combined with box-level methods.
- Feature-level methods: Feature-level methods are principled but suffer from inaccurate optical flow, motivating this work’s spatially-adaptive partial feature updating.The method is positioned as a feature-level approach that addresses errors introduced during feature propagation.
5. Experiments
Experiments evaluate the unified approach on ImageNet VID using speed–accuracy trade-off curves, ablations, and comparisons with prior methods. The proposed variants improve accuracy and efficiency through recursive aggregation, partial updating, and adaptive key-frame scheduling.
- Experimental Setup: ImageNet VID training and validation use 3,862 and 555 fully annotated snippets, respectively, across 30 object categories.Training also uses ImageNet DET images from the same 30 categories.
- Unified Evaluation: Sparse Feature Propagation achieves 5× speedup with accuracy loss within 1% when key-frame duration l varies from 1 to 10.This is reported as a baseline trade-off on Figure 3.
- Unified Evaluation: Dense Feature Aggregation improves mAP by 2.9% but is 3× slower than the per-frame baseline when aggregating windows from ±1 to ±10 frames.The result reflects the accuracy–efficiency cost of dense temporal aggregation.
- Unified Evaluation: Method (c1) provides 1% higher accuracy and 3× faster speed than the per-frame baseline through recursive aggregation on sparse key frames.It also delivers 10× speedup with 2% accuracy loss compared with Dense Feature Aggregation.
- Unified Evaluation: Method (c2) reaches 1.8% higher accuracy with 3× speedup, or 1.4% higher accuracy with 4× speedup, than the per-frame baseline.Spatially adaptive partial updating further improves accuracy with negligible computation burden.
- Unified Evaluation: Method (c3) improves accuracy by 0.5%–1% over c2 at high runtime speed and achieves 1% higher accuracy with 4.75× speedup versus the per-frame baseline.Its temporally adaptive key-frame scheduling replaces a fixed key-frame duration.
- Ablation Study: Recursive aggregation surpasses non-recursive aggregation at almost all runtime speeds, avoiding the accuracy decline caused by aggregating more than two key frames.The aggregated feature theoretically retains historical information from previous key frames.
- Ablation Study: Key-frame duration l = 10 gives the best speed–accuracy trade-off for c2, while small or large l values are less helpful because of redundancy or feature diversity.The partially updating coefficient λ and key-frame duration l control the trade-off.
mAP (%)
The figures present speed–accuracy trade-off curves for the proposed variants and flow-network choices, alongside comparisons with reported state-of-the-art systems. The reported comparison reaches 77.8% mAP at approximately 15.2 fps, but fairness is limited by differing system components.
- Aggregation Variants: Figure 4 compares c1 with non-recursive aggregation variants across speed–accuracy trade-offs.The corresponding ablation reports recursive aggregation surpassing non-recursive aggregation at almost all runtime speeds.
- C2 Ablations: Figure 5 compares c2 curves sharing a fixed key-frame duration l, while Figure 6 compares c2 curves sharing a fixed partially updating coefficient λ.These parameters control the speed–accuracy trade-off in the c2 ablation.
- Flow Networks: FlowNetS achieves the best speed–accuracy trade-off among the evaluated flow networks, including FlowNetC and FlowNet2.The paper attributes this result to FlowNetS's fast inference and reports improved results with joint training.
- State-of-the-Art Comparison: 77.8% mAP at 15.2 fps is reported for the proposed method with Deformable R-FCN, compared with 76.8% mAP at 15.4 fps for the ImageNet VID 2017 winner.The comparison uses reported results on ImageNet VID validation.
- State-of-the-Art Comparison: Fair comparison is difficult because different approaches use different recognition networks, object detectors, and post-processing techniques.The paper explicitly identifies this as a limitation of the reported comparison.
mAP (%)
The proposed method achieves high mAP across multiple speed settings and reports better accuracy–speed performance than prior state-of-the-art methods.
- 78.6% mAP is achieved at 13.0 / 8.6 fps on Titan X / K40.
- 77.8% mAP is achieved at 22.9 / 15.2 fps on Titan X / K40.
- The method achieves better performance in both accuracy and speed than the 2017 ImageNet VID challenge winner.The winning entry reports 76.8% mAP at 15.4 fps on Titan X.