PaperlayerView exact revision on arXiv

Source-linked AI summary

T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos

Kai Kang, Hongsheng Li, Junjie Yan, Xingyu Zeng, Bin Yang, Tong Xiao, Cong Zhang, Zhe Wang, Ruohui Wang, Xiaogang Wang, Wanli Ouyang

arXiv:1604.02532v4cs.CV

TL;DR

Still-image detectors are not designed to use videos’ temporal consistency and richer contextual information. T-CNN extends these detectors with tubelet-based temporal and contextual processing, achieving state-of-the-art VID performance and winning the provided-data ILSVRC2015 challenge.

  • Problem

    Still-image object detectors produce temporally inconsistent detections and cannot fully exploit video context, including information that can identify false positives.

  • Method

    T-CNN extends R-CNN and Faster R-CNN-style detectors by incorporating tubelet-based temporal propagation, confidence revision, and video-level contextual suppression.

  • Results

    T-CNN ranked 1st on the ILSVRC2015 provided-data track, winning 28 of 30 classes, while its components improved detection results by up to 6.7 percentage points.

  • Takeaways & Limitations

    Incorporating temporal and contextual information improved still-image detection baselines for video object detection and produced state-of-the-art performance on the ImageNet VID task.

Abstract

from arXiv · show

The state-of-the-art performance for object detection has been significantly improved over the past two years. Besides the introduction of powerful deep neural networks such as GoogleNet and VGG, novel object detection frameworks such as R-CNN and its successors, Fast R-CNN and Faster R-CNN, play an essential role in improving the state-of-the-art. Despite their effectiveness on still images, those frameworks are not specifically designed for object detection from videos. Temporal and contextual information of videos are not fully investigated and utilized. In this work, we propose a deep learning framework that incorporates temporal and contextual information from tubelets obtained in videos, which dramatically improves the baseline performance of existing still-image detection frameworks when they are applied to videos. It is called T-CNN, i.e. tubelets with convolutional neueral networks. The proposed framework won the recently introduced object-detection-from-video (VID) task with provided data in the ImageNet Large-Scale Visual Recognition Challenge 2015 (ILSVRC2015).

I. INTRODUCTION

Still-image detectors are effective on images but do not adequately use videos’ temporal consistency and contextual information. T-CNN extends these detectors with tubelet-based temporal and video-level contextual processing, improving video detection performance.

  • Motivation: Video detections from still-image frameworks fluctuate across adjacent and long-term frames because temporal consistency is not incorporated.The instability affects both bounding-box locations and detection confidences.
  • Motivation: Motion-guided propagation can transfer detections to neighboring frames, reducing missed detections when objects persist with similar motion and appearance.Duplicate propagated boxes can be removed with non-maximum suppression.
  • Motivation: Tubelets impose long-term constraints by increasing low-confidence scores when most boxes in the tracked sequence have high confidence.Low scores may arise from blur, unusual poses, or insufficient training examples for particular poses.
  • Motivation: Video-level context identifies low-confidence classes as outliers against the majority of high-confidence detections and suppresses their scores.This helps distinguish false positives that single-frame context cannot resolve.
  • Contribution: T-CNN extends R-CNN and Faster R-CNN with temporal and contextual information from tubelets for general object detection in videos.The framework won the ILSVRC2015 VID provided-data task and placed second with external data.

II. RELATED WORK

Related work spans deep CNN-based still-image detection, video detection methods that exploit temporal sequences, video localization, image classification, and visual tracking. The VID task differs from localization by requiring every target-class object in every frame and evaluating mean average precision.

  • Still-image object detection: R-CNN and Faster R-CNN are deep CNN-based detection frameworks that classify region proposals, with Faster R-CNN generating proposals through an integrated Region Proposal Network.ResNet further enabled very deep detection networks.
  • Object detection in videos: Video detection work includes sequence NMS and feedback-based frameworks that associate or reuse detections across frames.These methods address temporal structure in video detection.
  • Object localization in videos: VID requires annotating every object from all target classes in each frame, unlike localization tasks that seek one ground-truth object of a known or unknown class.VID uses Mean AP, whereas localization uses CorLoc with IOU > 0.5.
  • Object localization in videos: Localization datasets may annotate only a few frames, while VID provides annotations for different classes throughout training videos.This annotation difference contributes to the distinct task requirements described by the paper.
  • Image classification: Image classification supplies large-scale datasets and pretrained deep neural networks commonly used to initialize object-detection models.Batch normalization is described as reducing mini-batch statistical variation and accelerating training.
  • Visual tracking: Visual tracking research provides CNN-based methods for learning object-specific or generic representations used to track objects.These tracking methods support temporal processing in video analysis.

III. METHODS

T-CNN combines still-image detection with contextual suppression, motion-guided propagation, tubelet re-scoring, and model combination. These components address false positives, missed detections, temporal inconsistency, and proposal integration in VID.

  • Framework overview: T-CNN comprises still-image detection, multi-context suppression with motion-guided propagation, temporal tubelet re-scoring, and model combination.The VID task detects 30 classes and evaluates annotations using mean average precision.
  • Still-image detection: Still-image detectors apply DeepID-Net and CRAFT independently to video frames, producing region proposals and scored detections.DeepID-Net extends R-CNN, while CRAFT extends Faster R-CNN.
  • Multi-context suppression: Multi-context suppression ranks detections within a video and suppresses scores for low-confidence classes to reduce false positives.Classes with highly ranked detections are treated as high-confidence classes.
  • Motion-guided propagation: Motion-guided propagation uses optical flow or related motion information to propagate detections to adjacent frames, reducing false negatives.The operation addresses objects missed in individual frames but detected nearby in time.
  • Temporal tubelet re-scoring: Tubelet re-scoring tracks high-confidence detections into bounding-box sequences, classifies tubelets by score statistics, and widens positive-negative score margins.Positive scores are mapped higher and negative scores lower.
  • Model combination: Model combination min-max maps outputs from propagation and tubelet re-scoring to [0, 1], then merges proposal groups with NMS at IOU 0.5.The combined results form the final detections.

C. Still-image object detectors

The still-image detector stage uses DeepID-Net and CRAFT with multiple proposal, pretrained-model, finetuning, and score-averaging configurations. Their outputs provide the proposal groups processed by later T-CNN components.

  • Still-image object detectors: DeepID-Net and CRAFT use different region proposal methods, pretrained models, and training strategies as the two still-image detector families.Their outputs remain separate for subsequent T-CNN processing.
  • DeepID-Net: DeepID-Net proposals combine Selective Search and Edge Boxes, with cascaded selection and AlexNet-based scoring to remove easy negatives.Remaining proposal boxes receive scores for 200 ImageNet detection classes.
  • Pretraining: The provided-data track pretrains VGG and batch-normalized GoogLeNet on CLS data, while the external-data track uses ImageNet 3000-class data.Pretraining uses object-level annotations rather than image-level annotations.
  • Pipeline integration: T-CNN’s framework combines still-image detection with contextual suppression, motion propagation, tubelet re-scoring, and model combination.The full pipeline uses proposals from the still-image stage as inputs to temporal and contextual processing.
  • Finetuning: DET-trained networks and SVMs are finetuned or retrained with combined DET and VID data because their distributions and video statistics differ.VID contains a subset of DET classes, allowing class-index mapping before video-specific adaptation.
  • Score averaging: Multiple DeepID-Net CNN and SVM models are greedily selected and score-averaged to improve detection scores.Models are added when their averaged scores provide significant improvement.

2) CRAFT:

CRAFT extends Faster R-CNN with cascaded proposal and classification stages, then applies video-specific suppression and propagation to address false positives and false negatives. Motion-guided propagation shifts detections using optical flow, while multi-context suppression uses video-level class statistics.

  • CRAFT: CRAFT extends Faster R-CNN with cascaded RPN and Fast R-CNN stages for proposal generation, object/background scoring, and class-wise classification.The cascaded RPN achieves about 93% recall with approximately 100 proposals per image, while cascaded Fast R-CNN supports hard negative mining.
  • Multi-context suppression: Multi-context suppression ranks detections across each video, preserves high-confidence classes, and lowers low-confidence class scores to reduce false positives.The method exploits correlations among the few classes that typically co-exist in a video and identifies outlier detections across frames.
  • Motion-guided propagation: Motion-guided propagation augments detections by transferring boxes and scores to adjacent frames according to mean optical-flow vectors.It targets missed detections caused by missing proposals or low scores from bad poses and motion blur; duplicate boxes are removed with NMS.
  • Motion-guided propagation: MGP reduces false negatives within small temporal windows, but long-range propagation is limited by optical-flow accuracy.The cited analysis assumes perfect optical flow for its average mis-detection-rate reduction, while final NMS limits newly introduced false positives.

E. Tubelet re-scoring

Tubelet re-scoring combines long-term tracking, spatial max-pooling, and tubelet classification to enforce temporal consistency. A Bayesian classifier uses detection-score statistics to separate positive and negative tubelets before globally remapping their scores.

  • High-confidence tracking: Tracking generates long tubelets from high-confidence anchors, while stopping when tracking confidence falls below a threshold reduces false-positive tubelets.Tracks are generated bidirectionally, and new anchors are selected from remaining detections to form additional tubelets.
  • Tubelet classification and rescoring: Tubelet classification uses statistics such as mean, median, or top-k detection scores as inputs to a 1-D Bayesian classifier.The top-k feature performs best in the reported experiment.
  • Spatial max-pooling: Spatial max-pooling replaces tracked boxes with the highest-scoring overlapping detections from still-image detectors.Candidate detections must overlap each tubelet box beyond an IoU threshold, simulating conventional NMS while improving box selection.
  • Tubelet classification and rescoring: Positive tubelets are mapped to [0.5, 1] and negative tubelets to [0, 0.5], increasing score margins between the two groups.This global remapping is the final score-adjustment step after classification.

A. Dataset

The evaluation uses ImageNet VID for video object detection and YouTubeObjects for object localization, alongside experiments on training-data configurations and temporal processing. The reported qualitative results emphasize tight boxes, temporal consistency, and reduced false positives.

  • Dataset: ImageNet VID contains 3862 annotated training snippets, 555 validation snippets, and 937 test snippets with variable durations.Training and validation snippets are fully annotated, while test ground-truth annotations are not publicly available.
  • Dataset: YouTubeObjects evaluates localization across 10 object classes using 870 training clips and 334 test clips.Its annotations are sparse compared with the fully annotated ImageNet VID setting.
  • Qualitative results: Qualitative results show tight object boxes, consistency across adjacent frames, and substantially reduced false positives from the combined video-processing stages.The reported effects are attributed respectively to combined region proposals, motion-guided propagation and tracking, and multi-context suppression.
  • Parameter settings: The baseline DeepID-Net trained only on DET data has mean AP 49.8, while the enhanced configuration reaches 70.7 with the listed techniques.These values describe still-image detector configuration experiments rather than the full video framework result.
  • Parameter settings: A 2:1 DET-to-VID training-data ratio gives the best reported performance for DeepID-Net and CRAFT single models and is used for finetuning.Positive and negative samples from both datasets are also used to train DeepID-Net SVMs, which gives the best SVM performance.
  • Parameter settings: Frame sampling by a factor of 2 produces no significant validation-performance difference, while unsampled-frame detections are generated by interpolation and MGP.The MGP window is empirically set to 7 frames, with three frames propagated forward and backward.

A. Results on the ImageNet VID dataset

T-CNN improves video detection by combining still-image detections with temporal consistency and contextual information. It achieves strong qualitative behavior, component-level gains, and top rankings in ILSVRC2015 and ILSVRC2016 VID.

  • The detections remain consistent across adjacent frames without obvious false negatives, while bounding boxes stay tight and false positives are limited in complex scenes.These qualitative properties are attributed to high-quality proposals, motion-guided propagation and tracking, and multi-context information.
  • Adding temporal and contextual information through MCS, MGP, and tubelet re-scoring improves still-image detector results by up to 6.7 percentage points.MGP contributes 0.8–2.5%, MCS 2.2–3.8%, and tubelet re-scoring about 0.4–1.2% for different models.
  • The framework ranks 1st on the ILSVRC2015 provided-data track, winning 28 of 30 classes, and ranks 2nd on the additional-data track, winning 11 classes.
  • In ILSVRC2016 VID with provided training data, the framework ranks No. 2, while a four-model ensemble improves validation results by 12.5% and test results by 9.0% over ILSVRC2015.MCS+MGP still provides about 3.3% improvement with the stronger baseline.

B. Results analysis

The analysis examines MGP under camera motion and reports the added computational costs of temporal and contextual processing. MGP remains beneficial in moving-camera videos, while tracking is the main runtime bottleneck.

  • MGP improves results consistently on the moving-camera subset and performs better there than on the full validation set.The moving-camera subset contains 401 of 555 validation videos.
  • Optical-flow generation for MGP runs at about 10 fps, whereas the deep neural network tracker used for tubelet re-scoring runs at 0.5 fps.Tubelet tracking for the training and validation videos takes about 3 days, making tracking the main computational bottleneck.
  • The full processing schedule takes about 1 week for ImageNet pre-training, 2 days for DET fine-tuning, 2 days for VID fine-tuning, and 4 days for testing on an 8-GPU server.

C. Localication on YouTubeObjects (YTO) Dataset

The framework was evaluated for object localization on YouTubeObjects without fine-tuning and outperformed prior methods, while motion-guided propagation was sensitive to window size.

  • The system was evaluated on YouTubeObjects for object localization without fine-tuning because its classes are a subset of ImageNet VID.For each test frame, only the detection with the maximum score was retained, and performance was measured using CorLoc.
  • The proposed framework outperformed previous YouTubeObjects methods by a large margin, including about 15% over the state-of-the-art method.
  • A propagation window of 3 improved localization performance by about 0.5%, whereas increasing it to 5 caused a performance drop.
  • The framework achieved state-of-the-art performance on ImageNet's video object-detection task and won the provided-data VID challenge at ILSVRC2015.
  • The VID task remained new and under-explored, and incorporating the proposed components into end-to-end systems was identified as future work.
Loading 1604.02532v4…