Source-linked AI summary
Memory Enhanced Global-Local Aggregation for Video Object Detection
Yihong Chen, Yue Cao, Han Hu, Liwei Wang
TL;DR
Video object detection is challenged by degraded individual frames and requires both global semantic and local localization information. MEGA combines these sources and adds Long Range Memory so key frames can access substantially more content. It achieves 85.4% mAP on ImageNet VID, the best reported result in the paper’s evaluation.
Problem
Degraded single frames can make objects difficult to identify, while existing approaches separately emphasize global semantic or local localization information and aggregate insufficient content.
Method
MEGA uses a multi-stage global-local aggregation architecture with a Long Range Memory module that reuses precomputed features to provide key frames more content.
Results
85.4% mAP on ImageNet VID is reported for MEGA, described as the best reported result.
Takeaways & Limitations
MEGA provides a joint global-local approach to video object detection and uses memory to aggregate information from longer content.
Abstract
from arXiv · showhide
How do humans recognize an object in a piece of video? Due to the deteriorated quality of single frame, it may be hard for people to identify an occluded object in this frame by just utilizing information within one image. We argue that there are two important cues for humans to recognize objects in videos: the global semantic information and the local localization information. Recently, plenty of methods adopt the self-attention mechanisms to enhance the features in key frame with either global semantic information or local localization information. In this paper we introduce memory enhanced global-local aggregation (MEGA) network, which is among the first trials that takes full consideration of both global and local information. Furthermore, empowered by a novel and carefully-designed Long Range Memory (LRM) module, our proposed MEGA could enable the key frame to get access to much more content than any previous methods. Enhanced by these two sources of information, our method achieves state-of-the-art performance on ImageNet VID dataset. Code is available at \url{https://github.com/Scalsol/mega.pytorch}.
1. Introduction
Video object detection must use temporal information because individual frames can suffer motion blur, occlusion, or defocus. MEGA combines global and local aggregation with Long Range Memory to access more content and achieves 85.4% mAP on ImageNet VID.
- Motion blur, occlusion, and out-of-focus frames make single-frame video object detection difficult.
- Global semantic information helps identify similar objects across frames, while local localization information helps locate objects in the key frame.
- Using the entire video is infeasible because it contains a huge number of candidate boxes, motivating approximations that balance efficiency and accuracy.
- Previous state-of-the-art methods aggregate only 20-30 reference frames spanning 1-2 seconds, which inadequately approximates global or local influence.
- MEGA augments key-frame candidate box features by aggregating both global and local information in a multi-stage structure.
- 85.4% mAP on ImageNet VID is reported for MEGA, described as the best result to date.
2. Related Work
Video object detection methods exploit temporal information through local or global aggregation. MEGA combines both approaches and uses memory to access longer-range content.
- Video object detection methods improve per-frame detection by exploiting temporal information to address motion blur, occlusion, and defocus.
- Local aggregation methods use nearby frames through optical-flow propagation, learned alignment, or box-level relation modeling.
- Global aggregation methods enhance pixel or box features with semantic information beyond fixed temporal windows, but lack locality information for localization.
- MEGA combines global and local aggregation rather than treating them separately, while memory enables use of longer content.
- Prior work also explores aggregation beyond a small local range, including methods combining global and local information or using memory for video segmentation.
3. Method
MEGA aggregates global and local information for key-frame detection, then uses Long Range Memory to reuse intermediate features and expand the accessible temporal context.
- Long Range Memory: Long Range Memory caches intermediate features from previous frames and recurrently incorporates them into local aggregation without recomputing everything from the start.MEGA thereby models longer-term global and local dependencies while reusing precomputed features.
- Global-Local Aggregation: MEGA first forms local and global pools from candidate boxes in adjacent and randomly selected video frames.Candidate boxes are generated for each frame; adjacent boxes form L, while boxes from shuffled frame indices form G.
- Relation Module: Each candidate box is represented by semantic and localization features encoding appearance, spatial properties, and frame number.Localization features include height, width, center location, and temporal information.
- Relation Module: Relation modules enhance boxes through weighted sums of other boxes’ semantic features, with location-free and location-based variants.Location-free modules focus on semantic relations, while location-based modules also incorporate localization features and temporal information.
- Global-Local Aggregation: The base model first aggregates global features into local features, then applies stacked location-based relation modules to mine spatial-temporal information.The resulting key-frame box features are passed to an RCNN head for classification and regression.
- Long Range Memory: With Nl enhanced local aggregation stages, the key frame accesses Nl × Tm + Tl local and Nl × Tm + Tg global reference frames.The figure’s example uses Tl = 4, Tg = 4, Nl = 2, and Tm = 3, yielding 10 local and 10 global reference frames.
4. Experiments
MEGA is evaluated on ImageNet VID using Faster R-CNN with ResNet-101 and ResNeXt-101 backbones, including comparisons, runtime analysis, and ablations. The experiments show that combining global features, local features, and long-range memory improves detection performance while adding little computation overhead.
- Implementation Details: MEGA is evaluated on the ImageNet VID validation set using mAP, with training performed jointly on ImageNet VID and DET data.The dataset contains 3,862 training videos, 555 validation videos, and 30 object categories.
- Implementation Details: The detection system uses Faster R-CNN with ResNet-101 or ResNeXt-101 feature extractors and generates 300 candidate boxes per frame.MEGA uses a local temporal window of 25 frames, with Ng = 1 global and Nl = 3 local relation modules.
- Main Results: 84.1% mAP is achieved with ResNeXt-101 without post-processing, while ResNet-101 reaches 82.9% mAP and improves 1.1% over RDN.With BLR post-processing, MEGA reaches 84.5% mAP using ResNet-101 and 85.4% using ResNeXt-101.
- Ablation Study: A 1.5% mAP gap separates MEGA from the base model, and a 1% gap remains after enlarging the base model’s local span at substantially slower speed.The authors attribute the improvement to LRM, which provides efficient access to longer global and local content.
- Ablation Study: Removing global features reduces performance by 1.6% mAP, while removing local features produces a result 1.1% below the full model.These ablations support the reported necessity of both global and local information.
- Ablation Study: The full combination of local features, global features, and memory outperforms using any of the three components alone.MEGA also improves performance over its base model with little computation overhead.
5. Conclusion
MEGA jointly aggregates global and local information for video object detection, using a two-stage design that addresses ineffective and insufficient aggregation. Experiments on ImageNet VID validate the method’s effectiveness.
- MEGA jointly aggregates global and local information to solve video object detection.
- The first stage aggregates global features into local features to address ineffective aggregation.
- A novel Long Range Memory module addresses insufficient aggregation by extending the content available to key frames.
- Experiments on the ImageNet VID dataset validate MEGA’s effectiveness.