Source-linked AI summary
Detect to Track and Track to Detect
Christoph Feichtenhofer, Axel Pinz, Andrew Zisserman
TL;DR
Video detection systems had become increasingly complex because they typically combined frame-level detection with separate tracking and post-processing. The paper introduces a unified ConvNet that jointly detects and tracks using shared features, correlation, and tracklet linking. On ImageNet VID, it achieves better single-model performance than the previous challenge winner and offers faster tracking with temporal stride.
Problem
Video object detection requires handling large-scale sequences with motion blur, lower quality, partial occlusion, and varied poses, while recent high-accuracy systems rely on complex multistage processing.
Method
The paper extends R-FCN with end-to-end joint detection and tracking, using shared convolutional features, cross-frame correlation, track regression, and tracklet linking.
Results
The method achieves better single-model performance than the previous ImageNet challenge winner on ImageNet VID while being conceptually simpler and much faster.
Takeaways & Limitations
Joint detection and tracking provides mutual benefits and supports an efficient unified framework for video object detection.
Takeaways & Limitations
Tube-based reweighting assumes that the detector fails on at most half of a tube's frames.
Abstract
from arXiv · showhide
Recent approaches for high accuracy detection and tracking of object categories in video consist of complex multistage solutions that become more cumbersome each year. In this paper we propose a ConvNet architecture that jointly performs detection and tracking, solving the task in a simple and effective way. Our contributions are threefold: (i) we set up a ConvNet architecture for simultaneous detection and tracking, using a multi-task objective for frame-based object detection and across-frame track regression; (ii) we introduce correlation features that represent object co-occurrences across time to aid the ConvNet during tracking; and (iii) we link the frame level detections based on our across-frame tracklets to produce high accuracy detections at the video level. Our ConvNet architecture for spatiotemporal object detection is evaluated on the large-scale ImageNet VID dataset where it achieves state-of-the-art results. Our approach provides better single model performance than the winning method of the last ImageNet challenge while being conceptually much simpler. Finally, we show that by increasing the temporal stride we can dramatically increase the tracker speed.
1. Introduction
Video object detection is difficult because videos introduce scale, motion, quality, occlusion, and pose challenges, while high-performing systems rely on cumbersome multistage post-processing. The paper proposes a unified ConvNet that jointly detects and tracks objects using shared features, correlation, and end-to-end training.
- Motivation: Recent video-detection systems typically detect objects frame by frame and then propagate scores or link detections using trackers and other post-processing.A leading challenge entry combined multiple detectors, context suppression, multiscale processing, tracking, optical flow, and ensembles.
- Motivation: ImageNet VID contains around 1.3M images and adds challenges from motion blur, lower quality, partial occlusion, and unconventional poses.VID is substantially larger than the DET and COCO image-detection datasets.
- Approach: D&T directly infers tracklets over multiple frames by simultaneously performing detection and tracking with a fully convolutional ConvNet.The architecture extends R-FCN and is trained end-to-end with joint detection and tracking losses.
- Approach: Shared convolutional features support both tasks, while cross-correlation between adjacent frames estimates local displacement at multiple feature scales.The network uses multiple input frames and can use a ResNet-101 trunk.
- Results: On ImageNet VID, D&T achieves better single-model performance than the previous challenge winner while remaining conceptually simpler and much faster.The authors also report a temporally strided version that increases tracker speed.
2. Related work
Prior work spans region-based and direct image detectors, deep-feature trackers, and video systems that link frame detections through tracking or score propagation. The paper positions D&T as a fully convolutional extension of R-FCN that integrates detection and tracking rather than relying on extensive separate post-processing.
- Object detection: Popular object detectors include region-proposal methods such as R-CNN, Faster R-CNN, and R-FCN, alongside one-step predictors such as YOLO and SSD.
- Object detection: R-FCN reduces region-classification cost by moving region-wise operations to the network's end through position-sensitive RoI pooling.Its fully convolutional design is described as simple, efficient, and competitive with Faster R-CNN.
- Tracking: Deep-feature trackers use correlation or regression, but many handle only a single target and may not model changes in object scale and aspect ratio.A Siamese regression tracker can model scale and aspect changes but requires target-specific processing and substantial augmentation.
- Video object detection: Video-detection methods generate tubelet proposals or propagate detector scores across frames using trackers, 1D CNNs, or optical flow.These approaches build video-level results from frame-based detections.
- Video object detection: Optical-flow propagation can be erroneous, and key frames may miss features present in current frames.These issues are identified as sources of error in prior video-detection approaches.
3. D&T Approach
D&T extends R-FCN into a multi-frame architecture that jointly detects objects and regresses their cross-frame tracks. It shares convolutional features, adds correlation features and RoI tracking, and trains these tasks end-to-end.
- D&T Approach: D&T generates tracklets from two or more frames while jointly performing object detection and tracking.The approach links frame-level detections through across-frame tracklets over a video's temporal extent.
- D&T Approach: The architecture extends R-FCN with shared convolutional features, region proposals, per-frame classification and regression, and an inter-frame RoI tracking regressor.The tracking regressor pools features from both frames at the same proposal region to predict box transformations between frames.
- Correlation features: Unlike single-target correlation or regression trackers, D&T computes correlation maps for all feature-map positions to track multiple objects simultaneously.RoI pooling operates on these maps, allowing the end-to-end network to produce detections and tracks for multiple objects.
- D&T Approach: A multi-task objective combines classification, bounding-box regression, and tracking losses, with tracking targets assigned only to ground-truth objects appearing in both frames.Classification applies to all RoIs, box regression to foreground RoIs, and tracking regression to RoIs with cross-frame correspondence.
- Correlation features: Correlation maps compare feature vectors across local spatial offsets, producing multi-scale features used by RoI tracking for cross-frame regression.The local correlation output has shape H_l × W_l × (2d+1) × (2d+1), and correlations are computed at conv3, conv4, and conv5.
4. Linking tracklets to object tubes
The method links frame-level detections with across-frame tracklets to form long-term, class-specific object tubes. Tube scores are then reweighted to improve video-level detection robustness, while the formulation imposes temporal and processing constraints.
- Tube construction: Across-frame tracks link detections over time to build long-term object tubes from redundant, smoothly changing video frames.The linking score combines class detections with tracks describing box transformations between frames.
- Tube construction: The class-wise linking score combines detection probabilities with track correspondences across frames.Pairwise terms evaluate whether candidate detections overlap sufficiently with tracked boxes.
- Tube construction: Optimal video paths are found by maximizing scores over the video duration and solving the resulting optimization with the Viterbi algorithm.After selecting a tube, its detections are removed and the procedure is repeated for remaining regions.
- Tube reweighting: Detection scores within each class-specific tube are reweighted using the mean of its highest-scoring 50% of frames.The authors report less than 0.5% mAP variation when varying the selected proportion from 10% to 100%.
- Limitations: The tube formulation assumes detector failures affect at most half of a tube's frames and forces every tube to span the whole video.The authors do not prune detections over time, leaving removal of later low-score detections for future work.
5. Experiments
Experiments evaluate D&T on ImageNet VID using standard validation protocols, compare detection and tracking variants across models, and examine online and temporally strided testing. The results show strong accuracy, runtime advantages, and limited gains from short-window multi-frame score augmentation.
- Evaluation setup: ImageNet VID validation contains 30 classes across 3862 training and 555 validation videos, with performance measured as mAP over the 30 classes.The test-set ground truth is unavailable, so evaluation follows established validation protocols.
- Training and testing: The training procedure combines ImageNet VID and DET data, subsampling VID to 10 frames per video and limiting DET sampling to at most 2k images per class.D&T is fine-tuned from R-FCN using pairs of adjacent VID frames, with DET iterations interleaved to reduce dataset bias.
- Online capabilities and runtime: 78.7% mAP is achieved with causal online rescoring versus 79.8% mAP for noncausal rescoring, while correlation and ROI tracking add 14 ms per 1000x600-pixel image.Unoptimized tube linking takes an average of 46 ms per frame on one CPU core.
- Temporally strided testing: A temporal stride of τ = 10 yields 78.6% mAP, only 1.2% below full-frame evaluation, while reducing video processing duration by roughly a factor of 10.Augmenting detections with tracked proposals at t+10 raises accuracy from 78.6 to 79.2% mAP, whereas short temporal windows provide little gain.
- Results: 82.0% mAP is obtained with an Inception-v4 D&T backbone, while ResNeXt-101 and ResNet-101 achieve 81.4% and 79.8% mAP, respectively.The table compares frame-level Detection, video-level Detection & Tracking, and Detection & Tracking with averaged adjacent-frame scores.
6. Conclusion
The paper presents a unified framework for simultaneous object detection and tracking in video, with a fully convolutional architecture trained end-to-end. Joint detection and tracking achieves competitive accuracy while remaining simple and efficient.
- D&T provides a unified framework for simultaneous object detection and tracking in video.
- The fully convolutional architecture supports end-to-end joint training for detection and tracking.
- Jointly performing detection and tracking yields mutual benefits for both tasks.
- The method achieves accuracy competitive with the winner of the last ImageNet challenge while remaining simple and efficient.