Source-linked AI summary
Object Detection from Video Tubelets with Convolutional Neural Networks
Kai Kang, Wanli Ouyang, Hongsheng Li, Xiaogang Wang
TL;DR
The paper addresses general object detection in videos, where still-image detection and tracking alone do not robustly handle temporal variation. It combines CNN detection, generic tracking, tubelet processing, and temporal convolution, achieving 47.5% performance with far fewer boxes than still-image detection.
Problem
General video object detection requires robustly detecting unknown object classes across frames despite appearance changes and temporal score fluctuations.
Method
A multi-stage framework combines still-image CNN detection, generic object tracking, tubelet perturbation and max-pooling, and temporal convolutional re-scoring.
Results
47.5% mean AP is achieved on tubelet proposals, a 2.2-point increase over the still-image detection framework with 1/38 as many boxes.
Takeaways & Limitations
Combining detection and tracking provides tubelet proposals while temporal convolution incorporates consistency into video detection scores.
Takeaways & Limitations
The perturbation scheme depends on a sampling ratio hyperparameter controlling confidence in original versus tubelet boxes.
Abstract
from arXiv · showhide
Deep Convolution Neural Networks (CNNs) have shown impressive performance in various vision tasks such as image classification, object detection and semantic segmentation. For object detection, particularly in still images, the performance has been significantly increased last year thanks to powerful deep networks (e.g. GoogleNet) and detection frameworks (e.g. Regions with CNN features (R-CNN)). The lately introduced ImageNet task on object detection from video (VID) brings the object detection task into the video domain, in which objects' locations at each frame are required to be annotated with bounding boxes. In this work, we introduce a complete framework for the VID task based on still-image object detection and general object tracking. Their relations and contributions in the VID task are thoroughly studied and evaluated. In addition, a temporal convolution network is proposed to incorporate temporal information to regularize the detection results and shows its effectiveness for the task.
1. Introduction
Video object detection extends still-image detection to general objects across frames, where appearance changes and temporal variation require combining detection, tracking, and temporal modeling. The proposed framework studies these components jointly and reports improved tubelet-based detection.
- The ImageNet VID task requires detecting general object classes and localizing their bounding boxes in every video frame.
- Still-image detectors can fail under video-specific appearance changes, such as occlusion, pose variation, blur, and unseen object views.
- Tubelet box perturbation and max-pooling raise performance from 37.4% to 45.2% with only 1/38 as many boxes as image proposals.
- The framework combines detection-generated anchors, tracking-based tubelets, spatial max-pooling, and temporal convolution to regularize video detection.
2. Related Works
Related work spans deep CNN-based still-image detection, video localization, action localization, and object tracking. The VID task differs by requiring detection of multiple unknown object instances and classes in every frame.
- R-CNN introduced a multi-stage CNN pipeline that classifies region proposals for general object detection.
- Video object localization methods typically localize one known or unknown class in each video, unlike VID's multiple instances and classes per frame.
- Action localization methods generate or classify proposals for human actions using appearance, motion, or super-voxel cues.
- Deep CNN trackers include object-specific and multi-domain approaches designed to learn robust representations for tracking.
3. Method
The method combines image proposals, proposal scoring, bidirectional tracking, tubelet perturbation, spatial max-pooling, and temporal convolution. These stages address sparse proposals, localization sensitivity, tracking drift, and fluctuating detection scores.
- Framework: The framework has spatio-temporal tubelet proposal and tubelet classification and re-scoring modules.
- Task setting: VID evaluates frame, class, confidence, and bounding-box annotations across 30 classes using mean average precision.
- Tubelet proposal: Image proposals are scored with 30 DET-trained SVM models, whose high-confidence detections provide anchors for bidirectional tracking.
- Tubelet proposal: Tracking generates tubelet proposals, but drift, sparse coverage, location sensitivity, and low-confidence positive boxes limit naive tubelet classification.
- Tubelet classification and rescoring: Detection scores fluctuate even on ground-truth tubelets, motivating temporal modeling for more consistent scores.
- Tubelet classification and rescoring: Tubelet box perturbation and max-pooling replace boxes with higher-confidence alternatives to improve spatial detection robustness.
- Tubelet classification and rescoring: The TCN uses detection scores, tracking scores, and anchor offsets to produce temporally dense predictions for each tubelet box.
4. Experiments
The experiments evaluate the framework on ImageNet VID and YTO, while testing proposal generation, tubelet perturbation, and TCN settings. They compare perturbation schemes, overlap thresholds, and the four-layer TCN configuration.
- Datasets: The overall pipeline is evaluated on the ImageNet VID validation set, while YTO provides an additional object-localization evaluation.ImageNet VID contains fully labeled video snippets; YTO is weakly annotated and contains 10 object classes.
- Proposal generation: Image proposals are generated with Selective Search in fast mode after resizing inputs to 500 pixels wide.Generated proposals are mapped back to the original image coordinates.
- Tracking and proposal settings: The implementation uses early stopping at tracking confidence 0.1, at most 20 anchors per class, and a track-detection suppression overlap of 0.3.The minimum detection score for a new tracking anchor is 0, and tracking ends when no detections exceed that threshold.
- Tubelet perturbation: R(20, 0.1), R(20, 0.2), and O(0.5) are selected for later framework components after evaluating perturbation ratios, sample counts, and overlap thresholds.O(0.1) through O(0.9) are evaluated for adding original proposals.
- Temporal convolutional network: The TCN uses four convolutional layers, with raw detection scores, tracking scores, and normalized absolute anchor offsets as input features.Optimization and initialization settings are manually adjusted on one class and then held fixed across all 30 classes.
5. Results
The framework improves video detection through tubelet perturbation, spatial max-pooling, and temporal convolution, achieving strong mean AP with sparse proposals. Qualitative results show temporally consistent re-scoring, while high-confidence tracking reduces false positives and inference time.
- Quantitative results: 37.4% mean AP is obtained by directly scoring tubelet proposals, below the 45.3% mean AP of still-image detection.
- Tubelet perturbation and spatial max-pooling: 44.5% is the best result for adding original proposals with O(0.5), compared with 41.7% for random sampling with R(20, 0.2).Both schemes generally improve over the baseline, but overly large or small perturbations provide smaller gains.
- Tubelet perturbation and spatial max-pooling: 45.2% mean AP results from combining the best random-sampling and original-proposal perturbation schemes, with about 1:38 as many proposals per image as still-image detection.The combination doubles the number of tubelets relative to either perturbation scheme alone.
- Qualitative results: Tubelet proposals aggregate around objects while remaining sparse, and starting tracks only from high-confidence anchors reduces false positives and later inference time.Tracking suppression preserves coverage of many objects while limiting proposal density.
- Temporal convolution: Temporal convolution uses detection scores, tracking scores, and anchor offsets to produce probability curves that are more temporally consistent and better aligned with ground-truth overlaps.Detection scores vary substantially across frames, whereas the re-scored outputs smooth these variations.
- Generalization to YTO: On YTO, the full framework improves over its baseline by around 2% and is reported to outperform prior methods by a large margin.The authors attribute generalization to rich ImageNet CLS, DET, and VID supervision.
6. Conclusion
The paper presents a complete multi-stage video object-detection pipeline combining still-image detection and generic tracking for tubelet proposals, then evaluates perturbation and scoring strategies. Its temporal convolution network incorporates temporal consistency and consistently improves over still-image detection.
- The complete pipeline combines still-image object detection with generic object tracking for tubelet proposal generation.
- The paper studies the relationship and contributions of still-image detection and tracking, and evaluates tubelet perturbation and scoring schemes.
- A temporal convolutional network incorporates temporal consistency into video object detection and shows consistent improvement over still-image detections.
- A more advanced tubelet-based framework developed from this work won the ILSVRC2015 ImageNet VID challenge with provided data.