Source-linked AI summary
Flow-Guided Feature Aggregation for Video Object Detection
Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, Yichen Wei
TL;DR
Video object detection is difficult because object appearances degrade across frames, while prior temporal methods operate mainly on boxes without end-to-end training. FGFA aggregates motion-aligned features before detection through an end-to-end framework, significantly improving ImageNet VID performance, especially for fast-moving objects.
Problem
Video detection accuracy suffers from degraded appearances such as motion blur, defocus, and rare poses, while existing temporal methods use box-level post-processing without end-to-end training.
Method
FGFA warps nearby-frame features along estimated motion paths, adaptively aggregates them at a reference frame, and trains feature extraction, flow, aggregation, and detection end-to-end.
Results
FGFA significantly improves strong single-frame baselines on ImageNet VID, with the largest gains for fast-moving objects and accuracy on par with leading engineered systems.
Takeaways & Limitations
Feature-level temporal aggregation provides a principled complement to box-level video detection methods.
Takeaways & Limitations
The method slows inference somewhat, and the authors suggest lighter flow networks as a possible way to speed it up.
Abstract
from arXiv · showhide
Extending state-of-the-art object detectors from image to video is challenging. The accuracy of detection suffers from degenerated object appearances in videos, e.g., motion blur, video defocus, rare poses, etc. Existing work attempts to exploit temporal information on box level, but such methods are not trained end-to-end. We present flow-guided feature aggregation, an accurate and end-to-end learning framework for video object detection. It leverages temporal coherence on feature level instead. It improves the per-frame features by aggregation of nearby features along the motion paths, and thus improves the video recognition accuracy. Our method significantly improves upon strong single-frame baselines in ImageNet VID, especially for more challenging fast moving objects. Our framework is principled, and on par with the best engineered systems winning the ImageNet VID challenges 2016, without additional bells-and-whistles. The proposed method, together with Deep Feature Flow, powered the winning entry of ImageNet VID challenges 2017. The code is available at https://github.com/msracver/Flow-Guided-Feature-Aggregation.
1. Introduction
Video detection is challenged by degraded appearances and by methods that use temporal information only after detection. FGFA instead aggregates motion-aligned features end-to-end, improving recognition, particularly for fast-moving objects.
- Motivation: Video object detection suffers when motion blur, defocus, and rare poses degrade object appearance relative to still images.A strong still-image detector deteriorates remarkably for fast-moving objects.
- Prior approaches: Existing video methods detect individual frames and then assemble boxes using optical flow and hand-crafted tracking or association rules.These operations occur in a dedicated post-processing step rather than end-to-end training.
- Proposed framework: FGFA improves per-frame features by warping nearby-frame features to a reference frame and aggregating them with adaptive weights.The feature extractor, flow estimator, aggregation module, and detector are trained end-to-end.
- Results: FGFA is reported as complementary to box-level methods and on par with leading engineered ImageNet VID systems without extra bells-and-whistles.The framework, together with Deep Feature Flow, powered the winning ImageNet VID 2017 entry.
2. Related Work
Prior video methods commonly use temporal information at the box level, whereas this work uses temporal information to improve convolutional features before detection. The approach is designed to handle motion and complement later box-level refinement.
- Image detection: Still-image detectors use deep CNN feature maps followed by a shallow detection-specific network to generate results.The related work contrasts this image-detection structure with the paper’s video-focused feature aggregation.
- Video detection: Many video detection systems exploit temporal information during final bounding-box post-processing rather than feature computation.Examples propagate boxes with optical flow, form tubelets with tracking, and re-score boxes using tubelet classification.
- Video detection: Sequential stage dependencies make it difficult for such pipelines to correct errors produced by earlier stages.The cited limitation concerns reliance of each stage on previous-stage results.
- FGFA: FGFA incorporates temporal information at the feature level, trains the entire system end-to-end, and can also use bounding-box post-processing.Its feature-level processing is presented as complementary to box-level techniques.
- FGFA: The method associates appearance information across consecutive frames using flow-based feature warping to improve feature representation and video recognition accuracy.This design addresses motion while focusing on feature aggregation rather than only box assembly.
3. Flow Guided Feature Aggregation
FGFA enhances a reference frame by motion-guided warping and adaptive aggregation of nearby feature maps before detection. It uses a sliding temporal buffer and differentiable, end-to-end modules to combine information across frames.
- Model overview: A baseline applies an off-the-shelf object detector independently to each video frame, while FGFA propagates nearby information to improve poor reference-frame appearances.Nearby features can enhance responses when motion blur makes single-frame detection fail.
- Flow-guided warping: FGFA estimates motion between a reference frame and neighbors, then warps neighbor feature maps into the reference frame.The flow field is produced by a flow network, and bilinear warping is applied across feature-map locations and channels.
- Feature aggregation: The aggregation module fuses the reference and warped neighbor features using spatially varying weights shared across feature channels.The default aggregation range is K = 10, and the formulation resembles attention over a feature memory buffer.
- Detection: The aggregated feature map is passed to the detection sub-network to produce the reference-frame result.This places temporal aggregation before final detection rather than after bounding-box prediction.
- Adaptive weighting: Adaptive weights increase when warped neighbor features resemble reference features at the same spatial location, using cosine similarity after embedding.Weights are normalized across nearby frames for each spatial location.
- Training and inference: Inference processes frames with a sliding feature buffer, while the fully differentiable architecture includes feature extraction, flow estimation, aggregation, and detection.Training uses a smaller default K = 2 because of memory limits, with randomly sampled neighbors from the larger inference range.
- Complexity: The proposed method’s extra computation mainly comes from flow estimation and is generally lower than the feature network’s complexity.The reported approximation is r ≈ 1 + (2K+1)·O(F)/O(Nfeat).
4. Experiments
Experiments on ImageNet VID show that fast motion is a major detection challenge and that flow-guided, end-to-end feature aggregation improves accuracy over strong single-frame baselines. The method remains competitive without additional bells and whistles, while runtime and flow-computation trade-offs are documented.
- Experiment Setup: ImageNet VID experiments use 3,862 training snippets, 555 validation snippets, 30 object categories, and standard mAP plus motion-group mAP.Objects are grouped by motion IoU into slow (>0.9), medium ([0.7, 0.9]), and fast (<0.7).
- Ablation Study: Naive aggregation lowers overall mAP to 72.0% from the 73.4% single-frame baseline, with fast-motion mAP falling from 51.4% to 44.6%.The variant omits flow motion and adaptive weighting while retaining end-to-end training.
- Ablation Study: Adaptive weighting raises mAP to 74.3% and fast-motion mAP from 44.6% to 52.3%, while flow-guided aggregation raises overall mAP to 76.3% and fast-motion mAP to 57.6%.Compared with the single-frame baseline, FGFA improves overall mAP by 2.9% and fast-motion mAP by 6.2%.
- Ablation Study: Without end-to-end training, the FGFA variant is clearly worse than the proposed method, indicating that joint training is important.The comparison uses fixed single-frame feature and detection subnetworks, a pretrained FlowNet, and a learned embedding subnetwork.
- Efficiency and Temporal Sampling: FGFA processes one frame in 733ms versus 288ms for the single-frame baseline; composited adjacent flows reduce runtime to 356ms with approximately 1% lower accuracy.The default inference setting aggregates over 21 frames, while training uses 2 frames; accuracy saturates at 21 inference frames.
5. Conclusion and Future Work
The framework is accurate, end-to-end, and principled, while remaining complementary to box-level methods. Future improvements target runtime, fast object motion, flow estimation, annotation data, and aggregation memory.
- FGFA provides an accurate, end-to-end, and principled framework for video object detection.
- The feature-quality focus makes FGFA complementary to existing box-level frameworks for improving video-frame accuracy.
- The method slows down detection somewhat, motivating lighter flow networks for future speed improvements.
- Further progress is needed for fast object motion, potentially through more annotation data and more precise flow estimation.
- Future work could replace the attention-based aggregation with a better adaptive memory scheme.