PaperlayerView exact revision on arXiv

Source-linked AI summary

SeqFormer: Sequential Transformer for Video Instance Segmentation

Junfeng Wu, Yi Jiang, Song Bai, Wenqing Zhang, Xiang Bai

arXiv:2112.08275v2cs.CV

TL;DR

Video instance segmentation must detect, classify, segment, and track objects across videos, where changing locations and appearances complicate consistent instance modeling. SeqFormer decomposes a shared instance query into frame-level box queries, independently attends to frames, and aggregates temporal information for dynamic mask prediction. It reaches 47.4 AP with ResNet-50 and 49.0 AP with ResNet-101 on YouTube-VIS, while achieving 59.3 AP with Swin.

  • Problem

    Video instance segmentation requires simultaneous detection, classification, segmentation, and tracking, including accurate tracking across an entire video.

  • Method

    SeqFormer independently attends to each frame using decomposed box queries, aggregates aligned features into a video-level instance representation, and dynamically predicts mask sequences.

  • Results

    47.4 AP with ResNet-50 and 49.0 AP with ResNet-101 surpass previous state-of-the-art results by 4.6 and 4.4, while Swin reaches 59.3 AP on YouTube-VIS 2019.

  • Takeaways & Limitations

    SeqFormer naturally handles instance tracking without additional tracking branches or post-processing and is presented as a strong baseline for video instance segmentation.

Abstract

from arXiv · show

In this work, we present SeqFormer for video instance segmentation. SeqFormer follows the principle of vision transformer that models instance relationships among video frames. Nevertheless, we observe that a stand-alone instance query suffices for capturing a time sequence of instances in a video, but attention mechanisms shall be done with each frame independently. To achieve this, SeqFormer locates an instance in each frame and aggregates temporal information to learn a powerful representation of a video-level instance, which is used to predict the mask sequences on each frame dynamically. Instance tracking is achieved naturally without tracking branches or post-processing. On YouTube-VIS, SeqFormer achieves 47.4 AP with a ResNet-50 backbone and 49.0 AP with a ResNet-101 backbone without bells and whistles. Such achievement significantly exceeds the previous state-of-the-art performance by 4.6 and 4.4, respectively. In addition, integrated with the recently-proposed Swin transformer, SeqFormer achieves a much higher AP of 59.3. We hope SeqFormer could be a strong baseline that fosters future research in video instance segmentation, and in the meantime, advances this field with a more robust, accurate, neat model. The code is available at https://github.com/wjf5203/SeqFormer.

1 Introduction

Video instance segmentation combines detection, classification, segmentation, and tracking, making consistent object tracking across videos challenging. SeqFormer addresses this by independently attending to each frame while maintaining a shared instance representation for sequence prediction.

  • 1 Introduction: Video instance segmentation requires simultaneous detection, classification, segmentation, and tracking, with tracking across an entire video creating additional difficulty.Earlier tracking-by-detection methods can be sensitive to occlusions and motion blur.
  • 1 Introduction: SeqFormer models each video-level instance with a stand-alone query while performing attention independently on each frame.This lets attention follow an instance across changing positions, sizes, shapes, and appearances.
  • 1 Introduction: Frame-level box queries anchor the same instance independently on each frame, are refined through decoder layers, and align instance features in a coarse-to-fine manner.The resulting box sequences support instance association across frames.
  • 1 Introduction: The aggregated video-level representation predicts classification and dynamic mask-head parameters, allowing SeqFormer to produce mask sequences without separate tracking branches or hand-crafted post-processing.The model predicts classification, box sequences, and mask sequences in one step.
  • 1 Introduction: 47.4 AP with ResNet-50 and 49.0 AP with ResNet-101 exceed previous state-of-the-art results by 4.6 and 4.4, respectively, on YouTube-VIS 2019.With a Swin transformer backbone, SeqFormer reaches 59.3 AP.

2 Related Work

Prior video instance segmentation methods extend image segmentation with tracking branches or clip-level processing, while transformer approaches model video instances with queries. SeqFormer’s architecture uses framewise box-query decomposition and temporal feature aggregation to generate instance sequences.

  • 2 Related Work: Video instance segmentation extends image instance segmentation by segmenting and tracking all object instances across a video sequence.Image instance segmentation focuses on detecting instances and segmenting their pixels in static images.
  • 2 Related Work: Tracking-by-detection methods add tracking or re-identification mechanisms, whereas clip-based methods generate mask sequences and match instances across overlapping frames.These approaches represent the main earlier VIS paradigms described by the paper.
  • 2 Related Work: Transformer-based VIS methods introduced instance queries, but VisTR uses a fixed number of queries tied to video length and the maximum number of instances.SeqFormer instead shares one query across frames for instances with the same identity.
  • 2 Related Work: SeqFormer decomposes an initial instance query into frame-level box queries, aggregates their features into an instance query, and dynamically generates mask sequences.The architecture keeps box queries as frame anchors without interactions between them.

3 Method

SeqFormer uses frame-independent attention with video-level instance queries to locate and associate instances across frames, then generates class, box, and mask sequences.

  • Query Decompose Transformer Decoder: SeqFormer learns a robust video-level instance representation by querying each frame independently while maintaining one shared instance query.This design follows an instance across changing positions and appearances without treating frame instances as unrelated.
  • Query Decompose Transformer Decoder: Frame-specific box queries are refined through deformable attention to predict bounding boxes and preserve each instance's position information.The decoder repeatedly updates box queries using frame features, producing a box sequence for each instance.
  • Output Heads: Learned weighted aggregation of box queries produces the video-level representation used for classification and dynamic mask-head parameter generation.The aggregation reduces redundant information from frames without the instance and provides richer object cues for mask prediction.
  • Output Heads: Dynamic convolution uses the instance representation to generate mask sequences with frame-relative coordinate cues and shared parameters across frames.The mask head applies three 1 × 1 convolutions to frame-specific feature maps, enabling segmentation of the same identity throughout the video.
  • Instance Sequences Matching and Loss: Training predicts a fixed set of N video-level instances and uses Hungarian matching based on classification and box costs before applying auxiliary losses.The matching cost omits mask similarity because mask-level comparison is computationally expensive.

4 Experiment

Experiments evaluate SeqFormer on YouTube-VIS benchmarks and ablate its query decomposition, spatio-temporal representation, temporal aggregation, and frame usage. SeqFormer achieves strong benchmark performance, while the ablations identify frame-wise spatial-temporal modeling and weighted aggregation as important design choices.

  • Main Results: 59.3 AP is achieved on YouTube-VIS 2019 with a Swin transformer backbone, while ResNet-50 reaches 47.4 AP and YouTube-VIS 2021 reaches 40.5 AP.On YouTube-VIS 2019, the ResNet-50 result improves from 45.1 to 47.4 AP with COCO pseudo-video training; on YouTube-VIS 2021, the ResNet-50 result surpasses previous methods by 3.9 AP.
  • Ablation Study: Decomposing the shared instance query into frame-level box queries is identified as critical for SeqFormer’s instance query design.The box queries are maintained through decoder layers as frame anchors for the same instance and support independent refinement across frames.
  • Ablation Study: 7.4 AP is gained by retaining spatial-temporal feature dimensions and capturing instance features independently across frames, while multi-scale features add 2.6 AP.The comparison uses flattened versus retained spatial-temporal dimensions and evaluates single-scale versus multi-scale features.
  • Ablation Study: Weighted-sum aggregation improves AP by 1.9 over alternative temporal information aggregation settings.The aggregation experiment combines aligned frame features into the video-level instance representation.
  • Ablation Study: A mask head generated from only one input frame still produces 38.1 AP, indicating robust instance representation under reduced frame input.The generated mask head is applied to each frame to obtain mask sequences.

5 Conclusion

SeqFormer is presented as an efficient transformer architecture for video instance segmentation that independently attends to each frame and shares an instance query across a video-level instance. The authors position it as a strong baseline because it handles tracking without extra tracking branches or post-processing.

  • SeqFormer performs attention independently on each frame while learning a shared instance query for each video-level instance.
  • Instance-query decomposition aligns instance features and enables natural tracking without additional tracking branches or post-processing.
  • The authors report that SeqFormer surpasses existing state-of-the-art video instance segmentation methods by a large margin.

A Appendix

The appendix visualizes SeqFormer on four challenging YouTube-VIS 2019 validation cases, using consistent colors to represent each instance’s mask sequence.

  • The visualization covers varied poses, overlapping nearby instances, reappearance after occlusion, and severe occlusion by another instance.
  • Matching colors denote the mask sequence of the same instance across frames.

A.1 Visualization

The visualization shows how SeqFormer’s decoder attention evolves from shared initial sampling to independently maintained frame-level box queries. This produces coarse-to-fine spatial attention that follows instance motion.

  • The first decoder layer uses the same initial instance query to predict identical sampling-point distributions across frames.
  • From the second decoder layer, independently maintained frame-level box queries predict current-frame sampling points and are refined by sampled features.
  • SeqFormer therefore attends to different spatial locations following instance motion in a coarse-to-fine manner.

A.2 Aggregation of Temporal Information

SeqFormer aggregates aligned features into a video-level instance representation while weighting temporal features to reduce the influence of frames where the instance is absent.

  • Aligned features are aggregated into an instance query used to generate a video-level instance representation.
  • The model learns temporal aggregation weights from box queries because frames without an instance can provide useless or harmful features.
  • Frames without the instance receive lower aggregation weights.

A.3 Qualitative Comparisons

Qualitative comparisons indicate that SeqFormer produces more temporally stable mask predictions than other methods, with additional results available in the supplementary material.

  • SeqFormer produces more stable mask predictions over time than other methods in qualitative comparisons.
  • Additional video results and comparisons are provided in the supplementary material.
  • Figure 7 visualizes normalized softmax weights alongside their corresponding video frames.

A.4 Clip Matching

SeqFormer can process long videos through overlapping clips and clip-level instance-mask matching, with slightly worse but still competitive performance than full-video evaluation.

  • Overlapping video clips are matched using space-time soft IoU on their overlapping frames to associate clip-level instance masks.This clip-matching procedure follows IFC.
  • SeqFormer remains competitive when evaluated clip-wise, although performance is slightly worse than in the full-video setting.The comparison is reported in Table 7.
Loading 2112.08275v2…