PaperlayerView exact revision on arXiv

Source-linked AI summary

TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformers

Qianyu Zhou, Xiangtai Li, Lu He, Yibo Yang, Guangliang Cheng, Yunhai Tong, Lizhuang Ma, Dacheng Tao

arXiv:2201.05047v4cs.CV

TL;DR

Video object detection has not been thoroughly explored with DETR-like end-to-end models, despite the complexity of existing temporal aggregation and post-processing pipelines. The paper proposes TransVOD and its improved variants, using spatial-temporal Transformers to aggregate object queries and frame memories. TransVOD++ reaches 90.0% mAP on ImageNet VID, while TransVOD Lite reaches 83.7% mAP at around 30 FPS.

  • Problem

    DETR-like detectors had not been well explored for video object detection, where existing systems often require hand-crafted temporal aggregation and complicated post-processing.

  • Method

    TransVOD is an end-to-end spatial-temporal Transformer framework that aggregates spatial object queries and frame memory encodings, with TransVOD++ and TransVOD Lite adding object-level fusion and sequence-to-sequence clip prediction.

  • Results

    TransVOD++ achieves 90.0% mAP on ImageNet VID, while TransVOD Lite achieves 83.7% mAP at around 30 FPS on a single V100 GPU.

  • Takeaways & Limitations

    The three TransVOD models provide end-to-end VOD systems that remove many hand-crafted components and complicated post-processing methods while covering accuracy and real-time settings.

  • Takeaways & Limitations

    TransVOD has heavy computation costs in TDTE, and its performance remains limited; TransVOD Lite also shows failures under large motion blur, background change, and occlusion.

Abstract

from arXiv · show

Detection Transformer (DETR) and Deformable DETR have been proposed to eliminate the need for many hand-designed components in object detection while demonstrating good performance as previous complex hand-crafted detectors. However, their performance on Video Object Detection (VOD) has not been well explored. In this paper, we present TransVOD, the first end-to-end video object detection system based on spatial-temporal Transformer architectures. The first goal of this paper is to streamline the pipeline of VOD, effectively removing the need for many hand-crafted components for feature aggregation, e.g., optical flow model, relation networks. Besides, benefited from the object query design in DETR, our method does not need complicated post-processing methods such as Seq-NMS. In particular, we present a temporal Transformer to aggregate both the spatial object queries and the feature memories of each frame. Our temporal transformer consists of two components: Temporal Query Encoder (TQE) to fuse object queries, and Temporal Deformable Transformer Decoder (TDTD) to obtain current frame detection results. These designs boost the strong baseline deformable DETR by a significant margin (3%-4% mAP) on the ImageNet VID dataset. Then, we present two improved versions of TransVOD including TransVOD++ and TransVOD Lite. The former fuses object-level information into object query via dynamic convolution while the latter models the entire video clips as the output to speed up the inference time. We give detailed analysis of all three models in the experiment part. In particular, our proposed TransVOD++ sets a new state-of-the-art record in terms of accuracy on ImageNet VID with 90.0% mAP. Our proposed TransVOD Lite also achieves the best speed and accuracy trade-off with 83.7% mAP while running at around 30 FPS on a single V100 GPU device.

1 INTRODUCTION

Video object detection needs temporal modeling for challenging frame changes, but existing approaches often rely on complex hand-crafted pipelines. TransVOD introduces an end-to-end spatial-temporal Transformer family that simplifies VOD while improving accuracy and speed.

  • Video object detection is difficult because motion blur, occlusion, camera refocusing, and rare poses change object appearance across frames.
  • Existing VOD systems commonly use hand-crafted feature aggregation and post-processing components, including optical flow, recurrent networks, relation networks, and tubelet linking.
  • TransVOD extends DETR with a spatial-temporal Transformer that links object queries and memory encodings across frames for end-to-end current-frame detection.Its temporal Transformer includes TDTE, TQE, and TDTD components.
  • TransVOD++ adds Hard Query Mining, Query and RoI Fusion, dynamic-convolution appearance injection, and a stronger backbone, reaching 90% mAP on ImageNet VID.The changes also replace spatial fusion with QRF and avoid TDTE through object-level RoI features.
  • TransVOD Lite treats VOD as sequence-to-sequence prediction, outputs multiple frame results simultaneously, and achieves 83.7% mAP at around 30 FPS.Using temporal windows allows the method to exploit GPU memory for faster inference.

2 RELATED WORK

Prior VOD research mainly improves accuracy through temporal feature fusion or targets real-time inference, but both directions commonly retain specialized architectural components and human-designed priors.

  • VOD methods must detect objects in each frame and link the same objects across frames, motivating accuracy and real-time research directions.
  • Accuracy-focused VOD: Accuracy-focused methods aggregate features from nearby or entire sequences using flow-based warping, attention, non-local operations, or deformable convolutions.
  • Real-time VOD: Real-time VOD methods seek to preserve or improve accuracy while increasing speed, using designs such as object trackers, patchwork cells, and Convolutional LSTMs.
  • TransVOD Lite: TransVOD Lite instead models the entire VOD pipeline as sequence-to-sequence prediction and reports higher performance than a strong image baseline at faster speed.

3 METHOD

TransVOD extends DETR-like detection to video by combining spatial object queries and feature memories across frames with temporal Transformer modules. TransVOD++ adds object-level fusion and hard query mining, while TransVOD Lite predicts multiple frames simultaneously to reduce inference cost.

  • TransVOD Framework: TransVOD uses Spatial Transformers to extract object queries and compact feature memories from each frame before temporal fusion.The spatial detector comprises a Spatial Transformer Encoder and Decoder based on a modified Deformable DETR without multi-scale features.
  • TransVOD Framework: TransVOD is trained end-to-end and uses temporal Transformers to learn contexts across different video frames.The framework replaces separately designed temporal aggregation components with learned spatial-temporal Transformer modules.
  • TransVOD Framework: TDTE fuses spatial-temporal feature memories, while TQE links object queries across frames and TDTD produces current-frame detection results.TQE progressively aggregates reference-frame queries to enhance the current-frame query, and TDTD combines temporal queries with temporal feature memories.
  • TransVOD++: TransVOD++ adds Query and RoI Fusion and Hard Query Mining to inject object-level appearance information and reduce redundant queries and targets.QRF uses RoI features with dynamic convolutions, while HQM selects hard queries during temporal Transformer training.
  • TransVOD Lite: TransVOD Lite models video detection as sequence-to-sequence prediction, taking multiple frames as input and outputting all frame results simultaneously.Its temporal window size Tw controls the input clip length; larger Tw increases inference speed while increasing memory usage.
  • Training: Hungarian matching assigns spatial and temporal predictions to ground-truth boxes through one-to-one label assignment during training.The temporal Transformer uses loss functions based on the box and class predictions produced by its feed-forward output layers.

4.1 Experimental Setup

Experiments use ImageNet VID as the main video object detection benchmark, with ResNet and Swin Transformer backbones and specified Transformer training settings.

  • Datasets: ImageNet VID provides 3,862 training videos and 555 validation videos with annotated bounding boxes across 30 classes.The official test annotations are unavailable, so the paper follows common VOD protocols for training and evaluation.
  • Backbones: The experiments evaluate ResNet and Swin Transformer backbones, with Swin used to obtain stronger performance.The backbones are pretrained on ImageNet, and the experiments do not use multi-scale FPN features directly.
  • Training details: Training follows Deformable DETR with AdamW, learning rates of 2 × 10^-4 for Transformers and 2 × 10^-5 for the backbone, and weight decay of 10^-4.Transformer weights are initialized with Xavier initialization.

4.2 Main Results

TransVOD and its variants outperform strong video object detection baselines on ImageNet VID, with TransVOD++ emphasizing accuracy and TransVOD Lite emphasizing real-time speed-accuracy trade-offs.

  • ResNet-50 results: 79.9% mAP with ResNet-50 gives TransVOD a 1.3%∼2.6% absolute improvement over the best competitor, MEGA.TransVOD++ further raises performance to 80.5% mAP on the ImageNet VID validation set.
  • Real-time results: The real-time comparison evaluates accuracy and speed together, with TransVOD Lite reported as outperforming most existing models while using fewer parameters.Table 2 reports comparisons on the ImageNet VID validation set.
  • Stronger backbone results: 82.0% mAP is achieved by TransVOD++ with ResNet-101, including gains of +5.6% mAP over FGFA and +3.8% mAP over MANet.FGFA and MANet aggregate features using optical flow estimation.
  • Stronger backbone results: 90.0% mAP is achieved by TransVOD++ with Swin Base, outperforming previous works by about 4%∼5% mAP.This result is reported as the paper’s strongest accuracy result on ImageNet VID.
  • Real-time results: 83.7% mAP at nearly 30 FPS is achieved by TransVOD Lite with Swin-Tiny, while its Swin-Base version reaches 90.1% mAP at around 15 FPS.The Swin-Base model uses 46.9M parameters, fewer than the approximately 100M parameters cited for another video object detector.

4.3 Ablation Study and Analysis

Ablations show that TransVOD’s temporal query components improve deformable DETR baselines, while TransVOD++ and TransVOD Lite extend the framework through query fusion and sequence-to-sequence prediction.

  • TransVOD: TransVOD improves Deformable DETR baselines from 76.0% to 79.9% mAP with ResNet-50 and from 88.3% to 89.6% with Swin-Base.The full model combines TDTE, TQE, and TDTD components.
  • TransVOD: +2.9% and +1.0% mAP are obtained by adding TQE alone with ResNet-50 and Swin-Base, respectively.Using TDTE and TDTD alone gives additional gains of +1.1% and +0.5% on the two backbones.
  • TransVOD: One TDTD decoder layer is sufficient under the basic ablation setting and is used by default.The basic setting uses four reference frames, one TQE encoder layer, and one TDTE encoder layer.
  • TransVOD: The coarse-to-fine TQE aggregation strategy outperforms both coarse-to-coarse and fine-to-fine query selection.With one TQE encoder layer and four reference frames, the best performance uses the top 100 spatial object queries per reference frame.
  • TransVOD: Increasing reference frames improves mAP until performance tends to stabilize at eight frames, while the reported TransVOD experiments use 14 reference frames.This ablation uses three TQE encoder layers, one TDTE encoder layer, and one TDTD decoder layer.

Effect of each component in TransVOD++ on strong

The experiments examine component contributions, backbone effects, pre-training, temporal window size, and query-number choices in TransVOD++ and TransVOD Lite. Results show accuracy-speed trade-offs depend on the backbone and temporal window configuration.

  • Effect of each component: 1.4% mAP improvement comes from adding RoI and Query Fusion, while Hard Query Mining adds 0.3% mAP and 1.6% mAP on small objects.These results are reported for TransVOD++ on a strong baseline.
  • Backbone effects: 2.3%∼4.2% mAP improvements for TransVOD++ exceed TransVOD Lite’s 0.8%∼1.6% mAP gains with ResNet50 and ResNet-101 backbones.With Swin Transformer backbones, TransVOD Lite achieves almost the same performance as TransVOD++ because the single-frame baselines are already strong.
  • Multi-level feature fusion: 2.3% mAP50:95 gain from multi-level feature fusion exceeds its 0.6% mAP50 gain, indicating stronger benefits for more accurate detection.The paper adopts simple multi-level feature fusion by default with Swin Transformer backbones.
  • COCO pre-training: Removing COCO pre-training causes a huge performance drop, so the TransVOD series is pre-trained on COCO.The passage attributes this to pre-training on large-scale datasets.
  • Temporal window size: Increasing temporal window size improves both accuracy and FPS for Swin Tiny and Swin Base in TransVOD Lite.The selected best window size is Tw = 15, and sequential inputs still achieve the best performance compared with methods in Table 2.

4.4 Visualization and Analysis

Visualization compares TransVOD with a still-image detector and analyzes TransVOD Lite’s failure cases. Temporal modeling improves confidence and reduces missed or false detections, but motion blur, background change, and occlusion remain challenging.

  • Visual detection results: TransVOD models long-range dependencies across video frames, increasing correct-prediction confidence and reducing missed or false detections relative to Deformable DETR.The visual comparison includes motion blur and part occlusion cases.
  • Visual sampling locations: TransVOD’s temporal object queries sample more suitable context than spatial queries for detecting objects in video.The sampling-location visualization is presented as evidence for temporal query fusion.
  • Failure case analysis: TransVOD Lite misses detections under larger motion blur or background changes and falsely detects a car as a bus under large occlusion.The analysis identifies more stable temporal modeling and improved occlusion handling as future-work needs.

5 CONCLUSION

The paper concludes that TransVOD uses spatial-temporal Transformers to simplify video object detection while improving performance over deformable DETR. TransVOD++ and TransVOD Lite extend the framework toward higher accuracy and real-time detection.

  • Conclusion: TransVOD aggregates spatial object queries and frame memory encodings with temporal Transformers, removing many hand-crafted components and complicated post-processing.The framework is presented as a new perspective on feature aggregation for video object detection.
  • Conclusion: 3%-4% mAP improvement over strong deformable DETR baselines is reported on ImageNet VID across various baselines.The paper describes TransVOD as the first application of Transformers to video object detection.
  • Conclusion: TransVOD++ improves object-query use through Query and RoI Fusion and Hard Query Mining, while TransVOD Lite models video detection as sequence-to-sequence prediction.The two versions target improved performance and real-time video object detection, respectively.
Loading 2201.05047v4…