Source-linked AI summary
Object Detection in Video with Spatiotemporal Sampling Networks
Gedas Bertasius, Lorenzo Torresani, Jianbo Shi
TL;DR
Video object detection must handle frames degraded by blur, defocus, unusual poses, and occlusion while exploiting temporal information from nearby frames. STSN learns deformable spatiotemporal sampling locations directly for the detection task, achieving higher ImageNet VID accuracy than prior methods with a simpler, flow-free training design.
Problem
Video object detection is difficult because individual frames can contain blur, defocus, unusual poses, or occlusions, despite videos providing richer information across views and poses.
Method
STSN uses deformable convolutions to learn spatial feature samples from nearby supporting frames for detection in a reference frame, trained end-to-end from detection labels.
Results
STSN outperforms FGFA and D&T on ImageNet VID, including 78.9 versus 78.8 mAP against FGFA and 78.9 versus 75.8 without temporal post-processing against D&T.
Takeaways & Limitations
STSN provides higher video object detection accuracy with a simpler fully differentiable design that does not require optical-flow computation, supervision, or training data.
Abstract
from arXiv · showhide
We propose a Spatiotemporal Sampling Network (STSN) that uses deformable convolutions across time for object detection in videos. Our STSN performs object detection in a video frame by learning to spatially sample features from the adjacent frames. This naturally renders the approach robust to occlusion or motion blur in individual frames. Our framework does not require additional supervision, as it optimizes sampling locations directly with respect to object detection performance. Our STSN outperforms the state-of-the-art on the ImageNet VID dataset and compared to prior video object detection methods it uses a simpler design, and does not require optical flow data for training.
1 Introduction
Video object detection is harder than image detection because frames may contain blur, defocus, unusual poses, or occlusions, while temporal information offers additional views of the same object. STSN addresses this challenge by learning spatiotemporal feature samples end-to-end for detection.
- Video detection faces motion blur, defocus, unusual poses, and occlusions that make directly applying image-level detectors difficult.Video nevertheless contains richer information about object instances across poses and viewpoints.
- The central modeling challenge is exploiting temporal information effectively for video object detection.
- Prior methods often enforced temporal coherence through second-stage post-processing, preventing end-to-end training.
- STSN uses deformable convolutions across space and time to learn useful feature samples from nearby frames while optimizing detection accuracy.It is trained end-to-end using video frames labeled with bounding boxes.
- STSN achieves better accuracy than the state of the art on ImageNet VID without complex flow-network design or large amounts of flow data.
2 Related Work
Video object detection methods have evolved from temporal post-processing and flow-based pipelines toward end-to-end approaches that learn temporal feature relationships directly. STSN specifically samples features across frames without relying on optical-flow computation or supervision.
- STSN differs from standard deformable CNNs by learning to sample features temporally across video frames rather than only spatially.
- Earlier video detectors used optical flow, prediction propagation, tracking, or temporal graphs to make detections coherent across frames.
- FGFA jointly estimates optical flow and detects objects, then aligns and aggregates adjacent-frame features before detection.
- STSN is discriminatively trained from detection labels in a single end-to-end stage, avoiding separate optical-flow CNN training.The paper reports roughly eight days for FGFA’s two training stages versus four days for STSN.
3 Background: Deformable Convolution
Deformable convolution extends standard convolution by learning input-dependent offsets to alter its sampling grid. The sampled values are then combined with learned weights, with fractional offsets handled through bilinear interpolation.
- Standard 2D convolution samples a uniformly spaced grid R and computes a weighted sum of the sampled input values.For a 3 × 3 kernel with dilation 1, R contains the nine neighboring offsets.
- Deformable convolution augments the regular grid with data-conditioned offsets ∆p_n for each sampling location.The number of offsets equals the number of grid positions, N = |R|.
- The deformable operation samples x at offset-adjusted locations p_0 + p_n + ∆p_n and combines them using weights w(p_n).
- Because offsets are typically fractional, deformable convolution uses bilinear interpolation to obtain sampled feature values.Offsets are predicted by a separate convolutional layer and learned jointly with the convolution weights.
4 Spatiotemporal Sampling Network
STSN incorporates temporal information into video object detection by sampling useful features from nearby frames with deformable convolutions, then aggregating them for the reference frame. Its sampling is conditioned on the reference frame and trained end-to-end for detection.
- Temporal setup: STSN treats each video frame as a reference and uses nearby preceding and subsequent frames as supporting frames for detection.The system uses 2K supporting frames: K preceding and K subsequent frames.
- Feature sampling: The backbone computes object-level features for each frame, while spatiotemporal sampling blocks select relevant supporting-frame features conditioned on the reference frame.The reference and supporting feature tensors are concatenated before offsets are predicted for sampling.
- Feature aggregation: Sampled features from the selected frames are temporally aggregated with a per-pixel weighted summation, then passed to the detection network for bounding boxes and class probabilities.The aggregation weights indicate each supporting frame’s importance to the reference frame and are normalized with a softmax.
- Feature sampling: Deformable convolutional layers predict location offsets and use them to sample informative points from the supporting feature map.The sampling mechanism outputs a resampled feature tensor that can be used for object detection in the reference frame.
- Feature sampling: Four deformable convolutional layers iteratively refine feature maps and offsets before the final offsets sample points from the supporting feature map.The final sampled feature map is produced using the fourth offsets and the original supporting feature map.
- Design rationale: The learnable sampling module replaces optical-flow-based temporal correspondence with discriminative training from object-detection labels, without optical-flow training data.The architecture uses a deformable CNN based on ResNet-101 as its backbone and is trained end-to-end.
5 Experimental Results
Experiments on ImageNet VID evaluate STSN against static, flow-based, and temporally post-processed baselines, then analyze supporting-frame choices and qualitative sampling behavior. STSN achieves strong detection results without optical-flow supervision, while learned temporal sampling helps address blur and occlusion.
- Experimental Setup: ImageNet VID contains 3,862 training and 555 testing clips across 30 object categories, with bounding-box annotations.Frames are extracted at 25–30 fps.
- Quantitative Results: Without temporal post-processing, STSN outperforms D&T by mAP 78.9 versus 75.8.The authors interpret this as evidence that STSN produces strong spatiotemporal features directly.
- Quantitative Results: Adding Seq-NMS improves STSN and surpasses D&T+ with mAP 80.4 versus 79.8.D&T+ and STSN+ apply temporal post-processing to CNN outputs.
- Ablation Studies: Detection performance increases with more supporting frames before plateauing at T = 27.Supporting-frame analysis finds the largest weights near the reference frame, while distant frames such as k = −9 and 9 still contribute substantially.
- Qualitative Results: STSN learns object-aligned sampling and captures object motion without explicit optical-flow supervision, supporting detection under blur or occlusion.Qualitative examples show temporal sampling correcting misclassification and missed detections caused by occluded objects.
6 Conclusion
STSN is a simpler, fully differentiable video object-detection architecture that achieves higher accuracy than FGFA without optical flow computation or optical-flow training data.
- STSN provides higher video object-detection accuracy than the state-of-the-art FGFA method.
- Compared with FGFA, STSN uses a simpler design and does not require optical flow computation.
- STSN is fully differentiable and can be trained end-to-end without optical flow training data.
- Future work will investigate more complex designs for spatiotemporal sampling blocks.