Source-linked AI summary
Object Detection in Videos with Tubelet Proposal Networks
Kai Kang, Hongsheng Li, Tong Xiao, Wanli Ouyang, Junjie Yan, Xihui Liu, Xiaogang Wang
TL;DR
Video object detection needs temporal information, while existing tubelet-generation approaches trade off tubelet length, quality, or computational efficiency. The paper combines an efficient Tubelet Proposal Network with an LSTM that learns temporal features from tubelets, and experiments demonstrate effectiveness on ImageNet VID and YTO.
Problem
Existing video detection methods have limitations in generating high-quality tubelets efficiently, while video detection requires temporal information.
Method
The framework uses a Tubelet Proposal Network to generate diverse spatiotemporal tubelets and an LSTM to classify them using visual and temporal features.
Results
15.6% improvement over the second-best results is reported, while MoveTubelets+LSTM reaches 0.678 Mean AP and the system is 12× faster than the KCF comparison.
Takeaways & Limitations
Experiments demonstrate the framework’s effectiveness for video object detection on ImageNet VID and object localization on YTO.
Takeaways & Limitations
The approach assumes static proposals can serve as spatial anchors whose covered object portions remain consistent across later frames.
Abstract
from arXiv · showhide
Object detection in videos has drawn increasing attention recently with the introduction of the large-scale ImageNet VID dataset. Different from object detection in static images, temporal information in videos is vital for object detection. To fully utilize temporal information, state-of-the-art methods are based on spatiotemporal tubelets, which are essentially sequences of associated bounding boxes across time. However, the existing methods have major limitations in generating tubelets in terms of quality and efficiency. Motion-based methods are able to obtain dense tubelets efficiently, but the lengths are generally only several frames, which is not optimal for incorporating long-term temporal information. Appearance-based methods, usually involving generic object tracking, could generate long tubelets, but are usually computationally expensive. In this work, we propose a framework for object detection in videos, which consists of a novel tubelet proposal network to efficiently generate spatiotemporal proposals, and a Long Short-term Memory (LSTM) network that incorporates temporal information from tubelet proposals for achieving high object detection accuracy in videos. Experiments on the large-scale ImageNet VID dataset demonstrate the effectiveness of the proposed framework for object detection in videos.
1. Introduction
Video object detection must handle temporal changes and use temporal information effectively, but existing tubelet generation methods face quality and efficiency challenges. The paper proposes a Tubelet Proposal Network with LSTM-based temporal classification to address these issues.
- Video detection is difficult because objects undergo appearance and scale changes, occlusions, motion blur, and a mismatch between static-image and video data.
- Tubelets are sequences of bounding-box proposals, but generating them from frame-by-frame detections can be extremely time consuming.A cited tracking algorithm requires 0.5 second per detection box, limiting the number of tubelet proposals produced in videos with many frames and boxes.
- The proposed Tubelet Proposal Network simultaneously generates hundreds of diverse tubelets from static proposals through efficient feature-map pooling and multi-frame regression.
- An LSTM estimates object confidences from tubelet features and captures long-term temporal dependencies for proposal classification.
- The framework combines tubelet proposal generation with temporal classification using visual-temporal features.
2. Related work
Related work extends static-image detection, sequence-based score processing, and video localization, but these approaches address different aspects of video understanding.
- Object detection in static images: Static-image detection evolved from multi-stage R-CNN systems toward Fast R-CNN methods that accelerate feature extraction for region proposals.
- Object detection in videos: Video detection methods commonly post-process static-detector class scores to enforce temporal consistency across frames.
- Object localization in videos: Video localization datasets often simplify the task by assuming one known or unknown class per video and annotating only one object in each frame.
3. Tubelet proposal networks
The Tubelet Proposal Network generates spatiotemporal tubelet proposals from static spatial anchors by pooling multi-frame features and regressing temporal displacements. It uses relative movement prediction and block initialization to support efficient, multi-frame proposal generation.
- Existing tubelet methods are computationally expensive or prone to tracking failures, limiting dense proposal generation for video detection.A 100-fps single-object tracker could require about 56 GPU days to generate tubelets for ImageNet VID.
- TPN extracts visual features across time at static proposal locations and regresses temporal displacements to generate tubelet proposals.Large CNN receptive fields allow same-location pooling to capture possible object movements.
- Static proposals serve as spatial anchors for estimating later object movements while preserving high object recall when movement is robustly predicted.Static proposals can exceed 90% recall at individual frames, providing an upper bound for detection performance.
- The regression layer maps concatenated multi-frame pooled features to 4w movement values, from which tubelet box locations are inferred.The predicted relative movements are defined with respect to the anchor at the first frame, whose movement target is zero.
- Learning movements relative to the first-frame spatial anchor can avoid cumulative errors associated with conventional tracking.The regression objectives normalize movement targets using their mean and standard deviation and optimize a smoothed L1 loss.
- 3.4. Initialization for multi-frame regression layer: Block initialization uses learned parameters from a 2-frame regression model to initialize larger temporal-window models because their parameter matrices grow quadratically with window size.Random initialization becomes difficult when the temporal window is large.
4. Overall detection framework with tubelet generation and tubelet classification
The framework combines efficient Tubelet Proposal Networks for generating long, diverse tubelets with a CNN-LSTM classifier that uses visual-temporal features for detection.
- Overall framework: The framework uses a TPN to generate candidate object tubelets and a CNN-LSTM network to classify their bounding boxes over time.The TPN generates tubelets from static proposals, while the classification network predicts object categories using temporal information.
- Tubelet generation: TPN estimates object movement within temporal windows and iteratively extends tubelets until they cover the desired track length.The last estimated locations become spatial anchors for subsequent iterations, enabling tubelets of arbitrary lengths.
- Tubelet generation: Multiple static anchors share one CNN feature-map computation, allowing hundreds of tubelet proposals to be generated efficiently in parallel.The Fast R-CNN network receives multiple anchors in a frame through a single forward pass.
- Tubelet classification: The classification sub-network ROI-pools features from each tubelet across time and feeds them into an LSTM to learn spatiotemporal representations.Each LSTM memory unit conveys visual information across time for tubelet classification.
- Tubelet classification: The encoder-decoder LSTM encodes an entire tubelet before decoding in reverse order, using future as well as past information and avoiding weak early predictions from zero initialization.Its hidden states support class-confidence prediction and box regression through fully connected layers.
5. Experiments
Experiments evaluate tubelet generation, temporal classification, and video detection on ImageNet VID, with additional localization evaluation on YouTubeObjects. The proposed tubelets and LSTM-based classifier improve accuracy while substantially reducing tubelet-generation cost.
- Datasets and setup: The framework is evaluated on ImageNet VID and YouTubeObjects, using video detection and object localization metrics respectively.ImageNet VID contains 30 classes and reports validation-set results; YouTubeObjects uses CorLoc with IoU above 0.5.
- TPN training and evaluation: TPN training evaluates tubelet accuracy with mean absolute pixel difference, mean relative pixel difference, and mean intersection-over-union.The 2-frame baseline records MAD 15.50, MRD 0.0730, and Mean IOU 0.7966.
- TPN training and evaluation: Transformation initialization improves the 5-frame TPN over random initialization, while larger temporal windows can reduce accuracy as motion becomes harder to recover.The parameter size grows quadratically with the temporal window, and large displacements may exceed the receptive field.
- ImageNet VID results: MoveTubelets+Fast RCNN achieves Mean AP 0.623, versus 0.567 for KCF+Fast RCNN, while the proposed method is 12× faster with 300 proposals per frame.KCF runs at 50 fps for single-object tracking but takes 6 seconds per frame with 300 proposals.
- ImageNet VID results: MoveTubelets+LSTM reaches Mean AP 0.678, improving 5.5% over tubelet results and 4.8% over the static baseline.The improvement is attributed to learning appearance and temporal features from tubelet proposals; whale AP improves by over 25%.
- ImageNet VID results: The encoder-decoder LSTM reaches Mean AP 0.684, a 0.6% gain over vanilla LSTM, while the tubelet proposal model improves 2.5% over its regression baseline.The encoder-decoder model improves on over half of the classes and is at least as strong as the tubelet baseline across classes.
6. Conclusion
The paper concludes with a video object-detection system combining efficient tubelet proposal generation with an encoder-decoder CNN-LSTM for temporal feature learning. Evaluations on ImageNet VID and YouTubeObjects demonstrate the framework’s effectiveness.
- 6. Conclusion: The proposed system combines a Tubelet Proposal Network with an encoder-decoder CNN-LSTM model to learn temporal features from tubelets.The TPN generates tubelet proposals, while the CNN-LSTM classifies them.
- 6. Conclusion: Experiments evaluate the system for video object detection on ImageNet VID and video object localization on YouTubeObjects.The paper reports that these experiments demonstrate the effectiveness of the proposed framework.