Source-linked AI summary
End-to-End Video Instance Segmentation with Transformers
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, Huaxia Xia
TL;DR
Video instance segmentation must classify, segment, and track instances across frames, but existing approaches often rely on sophisticated pipelines. VisTR uses a Transformer for direct end-to-end parallel sequence prediction with sequence-level matching and segmentation, achieving the best single-model result and highest speed on YouTube-VIS.
Problem
Video instance segmentation requires simultaneous classification, segmentation, and tracking of object instances across video frames, while existing methods typically use sophisticated pipelines.
Method
VisTR uses a Transformer to decode ordered instance-mask sequences directly and supervises them with sequence-level matching and segmentation.
Results
VisTR achieves 40.1% mask mAP at 57.7 FPS, reported as the best and fastest result among single-model methods on YouTube-VIS.
Takeaways & Limitations
VisTR provides a simpler end-to-end framework that integrates instance segmentation and tracking through a shared similarity-learning perspective.
Abstract
from arXiv · showhide
Video instance segmentation (VIS) is the task that requires simultaneously classifying, segmenting and tracking object instances of interest in video. Recent methods typically develop sophisticated pipelines to tackle this task. Here, we propose a new video instance segmentation framework built upon Transformers, termed VisTR, which views the VIS task as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, effective instance sequence matching and segmentation strategy, which supervises and segments instances at the sequence level as a whole. VisTR frames the instance segmentation and tracking in the same perspective of similarity learning, thus considerably simplifying the overall pipeline and is significantly different from existing approaches. Without bells and whistles, VisTR achieves the highest speed among all existing VIS models, and achieves the best result among methods using single model on the YouTube-VIS dataset. For the first time, we demonstrate a much simpler and faster video instance segmentation framework built upon Transformers, achieving competitive accuracy. We hope that VisTR can motivate future research for more video understanding tasks.
1 Meituan 2 The University of Adelaide, Australia
VisTR is presented as a fast video instance segmentation model, with implementation code made publicly available.
- VisTR achieves the highest reported speed among existing video instance segmentation models.
- Implementation code is available at the provided VisTR repository.
1. Introduction
VisTR addresses video instance segmentation by directly predicting ordered instance-mask sequences with a Transformer, combining segmentation and tracking in a simpler end-to-end framework. The paper reports strong single-model results on YouTube-VIS.
- Video instance segmentation requires classifying, segmenting, and tracking object instances across video frames.
- Existing VIS methods use sophisticated pipelines, multiple steps, or human-designed association rules, motivating a simple end-to-end framework.
- VisTR treats instance segmentation and tracking as related similarity-learning problems within one framework.
- VisTR views VIS as a parallel sequence decoding problem and directly outputs the mask sequence for each video instance.
- Instance sequence matching performs bipartite matching between predicted and ground-truth instance sequences and supervises each sequence as a whole.
- VisTR achieves 40.1% mask mAP at 57.7 FPS, reported as the best and fastest result among single-model methods.
2. Related work
Related work distinguishes VIS from video object segmentation and reviews pipeline-based VIS methods alongside Transformer approaches for computer vision.
- Video object segmentation: Video object segmentation segments class-agnostic foreground objects, whereas VIS segments and tracks all instances from fixed object categories.
- Video instance segmentation: VIS requires classifying and segmenting instances in each frame while linking the same instance across frames.
- Video instance segmentation: MaskTrack R-CNN extends Mask R-CNN with a tracking branch and external memory for instance features across frames.
- Transformers: Transformers use self-attention to model long-range dependencies and have begun showing promise in computer vision.
3. Our Method: VisTR
VisTR treats video instance segmentation as direct sequence prediction, using a CNN, Transformer, sequence matching, and sequence segmentation modules. Its matching supervises instance sequences jointly, while segmentation aggregates cross-frame features to produce mask sequences.
- Overall architecture: VisTR models video instance segmentation as direct sequence prediction, outputting each instance’s mask sequence in input-frame order.The framework uses a fixed-size sequence of predictions decoded in one pass through the Transformer decoder.
- Overall architecture: The architecture combines a CNN backbone, encoder-decoder Transformer, instance sequence matching module, and instance sequence segmentation module.The CNN extracts multi-frame features; the Transformer models pixel- and instance-level feature relations before matching and segmentation produce supervised mask sequences.
- Transformer architecture: The Transformer concatenates frame features into a clip-level sequence and decodes N = n · T instance features whose ordering links predictions across frames.Predictions retain input-frame order, with the same instance-query positions corresponding across images.
- Instance sequence matching: Instance sequence matching finds a one-to-one assignment between predicted and ground-truth sequences using a pairwise cost and the Hungarian algorithm.Ground-truth sequences are padded with ∅ when fewer than n instances occur, and the resulting assignment supervises each sequence as a whole.
- Instance sequence matching: The matching loss combines class negative log-likelihood, box loss, and mask loss, with box costs using sequence-level L1 and generalized IOU terms.The Hungarian loss trains the whole framework after the optimal sequence assignment is obtained.
- Instance sequence segmentation: The segmentation module computes frame-specific similarity features, propagates same-instance features across frames with 3D convolutions, and directly outputs the mask sequence.It concatenates T frame features and uses three 3D convolutional layers with Group Normalization and ReLU; mask supervision combines Dice and Focal loss.
4. Experiments
Experiments on YouTube-VIS evaluate VisTR’s temporal design, component choices, comparisons with existing methods, and qualitative tracking performance. Ablations show benefits from longer and ordered temporal inputs, positional encodings, Transformer features, and instance sequence segmentation, while main results emphasize accuracy and speed.
- Experimental Setup: YouTube-VIS contains 2,238 training, 302 validation, and 343 test clips with per-pixel masks, category labels, and instance labels across 40 categories.Evaluation uses AP and AR with video mask-sequence IoU thresholds; the closed test set leads the authors to evaluate on validation data.
- Ablation Study: AP increases monotonically from 29.7% to 33.3% as input sequence length grows from 18 to 36 frames.The authors interpret this as evidence that more temporal information helps VisTR learn better.
- Ablation Study: Time-ordered input achieves 1 point higher performance than randomly ordered input, confirming the importance of temporal order.The study motivates this comparison by the continuity of object movement in real scenes.
- Ablation Study: Transformer-encoded features achieve 1.3 points higher AP than CNN-encoded features, reaching beyond the CNN baseline of 32.0% AP.The authors attribute this to self-attention updating features through pairwise similarities and jointly modeling spatial and temporal features.
- Ablation Study: The ablations identify temporal information, positional encodings, instance queries, global encoder self-attention, and instance sequence segmentation as important to final performance.The instance sequence segmentation module alone improves the result by 1.1 points.
- Main Results: VisTR achieves the best single-model AP without bells and whistles and reaches 27.7 FPS with ResNet-101, or 57.7 FPS excluding data loading.The reported speed advantage is attributed to parallel decoding and the absence of post-processing; qualitative results show tracking and segmentation in overlaps, position changes, nearby same-category instances, and varied poses.
5. Conclusion
VisTR presents video instance segmentation as a direct, end-to-end parallel sequence decoding problem built on Transformers. It simplifies the task by integrating tracking and segmentation, while achieving the best result and highest speed among single-model methods on YouTube-VIS.
- VisTR frames video instance segmentation as a direct end-to-end parallel sequence decoding and prediction problem built upon Transformers.
- The framework integrates instance tracking and segmentation within the same framework, considerably simplifying the overall pipeline.
- VisTR achieves the best result and highest speed among methods using a single model on the YouTube-VIS dataset.