Source-linked AI summary
Seq-NMS for Video Object Detection
Wei Han, Pooya Khorrami, Tom Le Paine, Prajit Ramachandran, Mohammad Babaeizadeh, Honghui Shi, Jianan Li, Shuicheng Yan, Thomas S. Huang
TL;DR
Video object detection remains difficult because detections can vary across frames due to scale changes, occlusion, and motion blur. Seq-NMS incorporates temporal information during post-processing to associate detections across adjacent frames, boost weaker scores, and ranked 3rd in the ILSVRC2015 VID task.
Problem
Video object detection remains challenging because objects detected easily in one frame may be difficult to detect in another due to scale changes, occlusion, and motion blur.
Method
Seq-NMS modifies post-processing by linking bounding boxes across adjacent frames and using high-scoring sequences to rescore weaker detections.
Results
Seq-NMS ranked 3rd in the ILSVRC2015 VID task, and the best submission achieved 48.7% mAP on the test set.
Takeaways & Limitations
Using temporal information in post-processing improved video detection performance over traditional frame-wise NMS and helped recover detections affected by common video perturbations.
Takeaways & Limitations
Seq-NMS may drift between similar nearby objects or accumulate spurious detections because its sequence-score objective does not penalize adding detections.
Abstract
from arXiv · showhide
Video object detection is challenging because objects that are easily detected in one frame may be difficult to detect in another frame within the same clip. Recently, there have been major advances for doing object detection in a single image. These methods typically contain three phases: (i) object proposal generation (ii) object classification and (iii) post-processing. We propose a modification of the post-processing phase that uses high-scoring object detections from nearby frames to boost scores of weaker detections within the same clip. We show that our method obtains superior results to state-of-the-art single image object detection techniques. Our method placed 3rd in the video object detection (VID) task of the ImageNet Large Scale Visual Recognition Challenge 2015 (ILSVRC2015).
1 Introduction
Video object detection remains difficult because detections vary across frames under scale changes, occlusion, and blur. Seq-NMS addresses this by adding temporal information to post-processing and uses nearby high-scoring detections to improve weaker ones.
- Motivation: Video object detection remains an open problem because objects can be easy to detect in one frame but difficult in another.The cited difficulties include drastic scale changes, occlusion, and motion blur.
- Motivation: Single-image detection pipelines use region proposal generation, object classification, and post-processing, but ignore temporal information.The paper identifies post-processing as the phase modified by Seq-NMS.
- Contribution: Seq-NMS modifies post-processing by using high-scoring detections from nearby frames to boost weaker detections within the same video clip.The method associates detections across frames, suppresses overlaps, and rescored weaker detections.
- Evaluation: Seq-NMS was evaluated on ImageNet VID and reported to outperform state-of-the-art single-image-based methods, including difficult scale, occlusion, and blur cases.The paper also presents specific examples where the method improves performance.
- Evaluation: Seq-NMS placed 3rd in the ILSVRC2015 video object detection task.This result is reported for the ImageNet Large Scale Visual Recognition Challenge 2015 VID task.
2 Our Approach
Seq-NMS extends frame-wise post-processing by selecting temporally linked box sequences, rescoring them, and suppressing overlapping alternatives. Sequence selection maximizes summed detection scores subject to adjacent-frame linkage constraints, and the process repeats iteratively.
- Seq-NMS: Seq-NMS repeats sequence selection, sequence re-scoring, and suppression until no sequences remain.The method is designed as a heuristic re-ranking procedure for bounding boxes in video sequences.
- Input: The method processes all proposal boxes and scores from a video clip rather than proposals from a single image.The clip contains frame-level proposal sets whose sizes can vary by frame.
- Sequence Selection: Sequence selection links boxes in adjacent frames when their IoU exceeds a threshold and searches for a maximum-score linked sequence.The selected sequence contains at most one box per frame.
- Sequence Selection: The linkage constraint uses IoU > 0.5 between adjacent boxes.The optimization restricts sequences to linked boxes across consecutive frames.
- Sequence Selection: Dynamic programming efficiently returns the selected boxes Bseq and their associated scores Sseq′.The algorithm maintains the maximum score sequence found so far at each box.
- Sequence Re-scoring: Re-scoring applies either the average or the maximum of the selected sequence scores to produce new frame-level scores.The function maps Sseq′ to Sseq.
- Suppression: Suppression removes selected sequence boxes from future linking and removes same-frame candidates whose overlap with selected boxes exceeds a threshold.This prevents overlapping candidates from remaining in the candidate set.
3 The Dataset
The ImageNet VID dataset organizes annotated video frames into clips or snippets for video object detection. It contains 30 object categories, which are a subset of ImageNet’s 200 detection categories.
- Dataset Structure: ImageNet VID groups frames from the same videos into clips or snippets rather than treating them as independent images.Every visible object in every frame is annotated with a class label and bounding box.
- Dataset Categories: The VID dataset contains 30 object categories that are a subset of the 200 categories in the ImageNet detection task.
4 Results
Seq-NMS improves video detection by adding temporal information to post-processing, with gains on validation and test sets but known failure cases. Its strongest reported results include 52.2% validation mAP and a third-place test ranking.
- Quantitative Results: Seq-NMS validation experiments compared single-image NMS with average, maximum, and best-sequence rescoring on ImageNet VID.The best variant selects the strongest of the three techniques per class and averages the results.
- Quantitative Results: 53.6% mAP was achieved on the initial validation set by combining all three NMS techniques.Seq-NMS (avg) achieved 51.5% on the same set.
- Quantitative Results: 52.2% mAP was achieved on the full validation set with Seq-NMS (best).Seq-NMS (avg) achieved 51.4% on the full validation set.
- Quantitative Results: 3rd place was achieved on the ImageNet VID test set, with the best submission reaching 48.7% mAP.The results used VGG net models and improved over frame-wise NMS post-processing.
- Quantitative Results: 7.3% versus 6.7% was the improvement from temporal information for Seq-NMS compared with the challenge’s top-performing method.The comparison isolates temporal-information techniques rather than the complete systems.
- Qualitative Results: Seq-NMS improves weak detections caused by occlusion, scale changes, or blur by associating them with stronger detections in nearby frames.Average rescoring raises the confidence of lower-scoring detections in the illustrated clips.
- Qualitative Results: Seq-NMS can drift between similar nearby objects or accumulate false positives because its sequence-score objective does not penalize adding detections.These behaviors produce missed detections, incorrect score assignments, and additional false positives.
5 Related Work
The paper situates Seq-NMS among video detection approaches that use temporal information, contrasting its post-processing strategy with tracking-based methods and illustrating its behavior on difficult clips.
- Related Work: Earlier video object detection work often framed the task as multiple-object tracking.A popular subclass applied detection independently to each frame and associated detections across frames into object trajectories.
- Related Work: Tracking-by-detection methods associate framewise detections into trajectories, using motion or object appearance as primary cues.Classical association methods included Kalman filters with Hungarian matching, while particle filters handled multiple hypotheses.
- Qualitative Examples: Seq-NMS examples show temporal rescoring helping when objects become occluded, very small, or blurred.The selected boxes are subsampled to display both high- and low-scoring detections.
6 Conclusion
The submission combines a strong Faster R-CNN baseline with additional temporal information and achieved a top result in the ImageNet video detection challenge. The authors identify further training, suppression, and analysis work as future directions.
- Conclusion: Seq-NMS uses additional temporal information with a strong Faster R-CNN baseline for video object detection.The authors describe the approach as one of the top performers in the ImageNet Object Detection from Video challenge.
- Conclusion: Future work includes training on the entire VID dataset, experimenting with neural-network suppression, and analyzing model weaknesses more deeply.These directions are presented as improvements to the submission.