Source-linked AI summary
Sequence Level Semantics Aggregation for Video Object Detection
Haiping Wu, Yuntao Chen, Naiyan Wang, Zhaoxiang Zhang
TL;DR
Fast motion causes appearance degradation that makes video object detection difficult for single-frame detectors, while existing aggregation methods emphasize nearby frames or rely on optical flow or recurrent networks. The paper introduces full-sequence Sequence Level Semantics Aggregation (SELSA), interprets it through spectral clustering, and reports state-of-the-art results on ImageNet VID and EPIC KITCHENS without sophisticated post-processing.
Problem
Fast motion causes motion blur, camera defocus, and pose variation, making single-frame detection difficult; existing methods mainly use temporally nearby information.
Method
SELSA treats video as unordered frames, clusters and enhances proposal-level semantic features across the full sequence, and relates the method to spectral clustering.
Results
The method achieves state-of-the-art results on ImageNet VID and EPIC KITCHENS, including 82.7 mAP with ResNet-101 and 84.3 mAP with ResNeXt-101.
Takeaways & Limitations
Full-sequence proposal-level aggregation provides a simple alternative to optical-flow or recurrent feature aggregation and does not require sophisticated post-processing.
Takeaways & Limitations
Optical-flow comparisons are bounded by settings where object location, appearance, and pose change dramatically, making flow estimation unreliable.
Abstract
from arXiv · showhide
Video objection detection (VID) has been a rising research direction in recent years. A central issue of VID is the appearance degradation of video frames caused by fast motion. This problem is essentially ill-posed for a single frame. Therefore, aggregating features from other frames becomes a natural choice. Existing methods rely heavily on optical flow or recurrent neural networks for feature aggregation. However, these methods emphasize more on the temporally nearby frames. In this work, we argue that aggregating features in the full-sequence level will lead to more discriminative and robust features for video object detection. To achieve this goal, we devise a novel Sequence Level Semantics Aggregation (SELSA) module. We further demonstrate the close relationship between the proposed method and the classic spectral clustering method, providing a novel view for understanding the VID problem. We test the proposed method on the ImageNet VID and the EPIC KITCHENS dataset and achieve new state-of-the-art results. Our method does not need complicated postprocessing methods such as Seq-NMS or Tubelet rescoring, which keeps the pipeline simple and clean.
1. Introduction
Video object detection is challenged by fast-motion appearance degradation, while existing temporal or post-processing approaches incompletely exploit video information. SELSA reframes VID at the full-sequence level and reports strong results on two datasets.
- Fast motion causes motion blur, camera defocus, and large pose variation, where still-image detectors often fail.
- Existing post-processing and feature-calibration methods use video information but may be sub-optimal or depend on unreliable motion estimation.Two-stage methods are not jointly optimized, while optical flow becomes unsatisfactory under fast motion.
- The paper treats video detection as a sequence-level multi-shot problem rather than only a sequence of temporally nearby frames.It models video as unordered frames and seeks an invariant representation for each class across the full sequence.
- SELSA introduces a simple module that clusters and enhances features across the entire sequence before detection.The module is presented as a way to fully utilize video information within the deep object-detection pipeline.
- 82.7 mAP with Faster-RCNN and ResNet-101, and 84.3 mAP with ResNeXt-101, improve state-of-the-art results on ImageNet VID.Additional EPIC KITCHENS experiments show generalization to more complex scenes.
2. Related Work
Related video-detection methods add temporal information through post-processing, optical-flow feature propagation, or recurrent memory. SELSA instead aggregates instance-level semantic features across a multi-shot sequence without temporal alignment or ordered memory passing.
- Region-relation methods model appearance and geometry relations among proposals within a single image for joint reasoning or duplicate removal.
- Two-stage video methods post-process still-image detections by linking boxes, propagating boxes, or rescoring tracked tubelets.
- Optical-flow methods propagate and align features across frames to improve accuracy or reduce redundant computation.DFF, FGFA, and MANet progressively incorporate flow-based propagation, alignment, and instance-level calibration.
- Optical-flow calibration can be error-prone when object location, appearance, and pose change dramatically, making flow estimation unreliable.SELSA avoids temporal feature alignment by aggregating at the proposal level.
- Recurrent methods pass information through videos in temporal order using RNN, association LSTM, or spatial-temporal memory modules.SELSA instead forms clusters and aggregates instance-level features in a multi-shot view.
3. Method
SELSA aggregates proposal features across a video's semantic neighborhood rather than relying on a short temporal window. Its proposal-similarity aggregation is connected to spectral clustering, framing feature refinement as intra-class variance reduction.
- Motivation: SELSA addresses the limited temporal coverage and redundancy of short-window aggregation by selecting features from a semantic neighborhood across the video.The method treats semantic neighbors as less susceptible to appearance degradation lasting over time.
- Sequence Level Semantics Aggregation: Semantic similarities link proposals across frames, and similarity-guided aggregation produces richer proposal features for detection.The architecture extracts proposals, computes cross-frame similarities, and aggregates features from other proposals.
- Sequence Level Semantics Aggregation: SELSA normalizes similarities with softmax across proposals before aggregating features from randomly selected frames, preserving feature magnitude.The enhanced proposal features are then passed to the detection head, and the module is optimized end-to-end.
- A Spectral Clustering Viewpoint: SELSA is closely related to spectral clustering: supervised learning optimizes the stochastic matrix while the desired proposal partition is given.This viewpoint connects the method to normalized minimum cut and interprets aggregation as reducing intra-class feature variance.
- A Spectral Clustering Viewpoint: The method constructs a proposal similarity graph in which proposals are nodes, similarities are edges, and row-normalized weights define random-walk transitions.The transition probability between subgraphs measures the risk of incorrectly aggregating features across classes.
4. Experiments on ImageNet VID
Experiments on ImageNet VID evaluate SELSA’s design choices, sampling strategies, post-processing dependence, and comparison with prior systems. The results show that sequence-level semantic aggregation benefits from diverse full-video features and achieves strong performance without requiring video-level post-processing.
- Experimental setup: The ImageNet VID evaluation reports mAP@IoU=0.5 and motion-specific mAP on the validation set.The model is trained with a mixture of ImageNet VID and DET data; ablation studies use ResNet-101, while final results also use ResNeXt-101.
- Effectiveness of SELSA: SELSA raises mAP from the 73.62 single-frame baseline to 80.25, a 6.63 mAP improvement.The method samples semantic neighbors from the full video sequence to enhance proposal features.
- Effectiveness of SELSA: Fast-motion performance reaches 61.38 mAP, improving by 9.95 mAP over the baseline.This motion-specific result is reported for the proposed SELSA method.
- Sampling strategies: Using 21 aggregation frames instead of 5 improves performance by 1.04 mAP, while increasing sampling stride from 1 to 10 improves it from 77.02 to 79.36 mAP.The larger stride has a greater general influence than the number of testing frames, consistent with the value of sample diversity for sequence-level aggregation.
- Sampling strategies: Shuffled testing with 21 frames reaches 80.25 mAP, and five shuffled frames match the performance of 21 strided frames.The shuffled setting samples semantic neighbors uniformly from the full sequence without relying on temporal information or cross-frame feature alignment.
- Comparison and post-processing: Without video-level post-processing, SELSA achieves 80.25 mAP, outperforming FGFA by 3.95 mAP and MANet by 2.15 mAP.With a stronger ResNeXt-101 backbone, SELSA reaches 83.11 mAP without post-processing; adding Seq-NMS gives only a minor impact in the reported analysis.
5. Additional Experiments on Epic Kitchen
SELSA is evaluated on EPIC KITCHENS, a challenging egocentric dataset with many object classes and both seen and unseen test kitchens. It improves over the Faster R-CNN baseline despite non-optimal training and hyperparameter choices.
- Dataset and Evaluation Setup: EPIC KITCHENS contains 454,255 object bounding boxes across 290 classes, with evaluation on seen and unseen kitchen splits.The dataset includes 272 training sequences and 160 evaluation sequences, with annotations at one-second intervals.
- Dataset and Evaluation Setup: SELSA samples frames within a ±10s window during both training and inference on EPIC KITCHENS.
- Results and Analysis: 1.4/2.94 mAP gains over Faster R-CNN are obtained on the Seen/Unseen splits, respectively.
- Results and Analysis: The EPIC KITCHENS results are promising even though the training scheme and hyperparameter selection are far from optimal.
6. Conclusion
The paper reframes video object detection around full-sequence feature aggregation and introduces SELSA as a simple semantic aggregation module. The method is reported to improve robustness and performance without sophisticated post-processing.
- SELSA aggregates semantic features across frames at the full-sequence level instead of relying on optical flow or recurrent neural networks.
- Proposal-level aggregation makes SELSA more robust to motion blur and large pose variation.
- SELSA achieves superior performance without sophisticated post-processing methods.