Source-linked AI summary
Mask2Former for Video Instance Segmentation
Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, Alexander G. Schwing
TL;DR
Video instance segmentation requires jointly segmenting and tracking objects, while prior approaches are often specialized for video. This paper adapts universal image segmentation by having Mask2Former attend to 3D spatio-temporal features and directly predict 3D instance volumes. It achieves state-of-the-art performance, reaching 60.4 AP on YouTubeVIS-2019 and 52.6 AP on YouTubeVIS-2021.
Problem
Prior video instance segmentation methods are specifically designed for video, leaving image and video segmentation research disconnected.
Method
Mask2Former adapts to videos by applying masked attention to spatio-temporal volumes, adding temporal positional encoding, and directly predicting 3D instance volumes.
Results
60.4 AP on YouTubeVIS-2019 and 52.6 AP on YouTubeVIS-2021 establish state-of-the-art video instance segmentation performance.
Takeaways & Limitations
Universal image segmentation architectures can generalize to video instance segmentation without modifying the architecture, loss, or training pipeline.
Abstract
from arXiv · showhide
We find Mask2Former also achieves state-of-the-art performance on video instance segmentation without modifying the architecture, the loss or even the training pipeline. In this report, we show universal image segmentation architectures trivially generalize to video segmentation by directly predicting 3D segmentation volumes. Specifically, Mask2Former sets a new state-of-the-art of 60.4 AP on YouTubeVIS-2019 and 52.6 AP on YouTubeVIS-2021. We believe Mask2Former is also capable of handling video semantic and panoptic segmentation, given its versatility in image segmentation. We hope this will make state-of-the-art video segmentation research more accessible and bring more attention to designing universal image and video segmentation architectures.
1. Introduction
Video instance segmentation must segment and track objects across time, but existing approaches are specialized for video. Mask2Former shows that a universal image segmentation architecture can generalize directly to videos by sharing queries across frames and predicting 3D instance volumes.
- Video instance segmentation jointly segments and tracks objects, unlike standard image segmentation.
- Existing video methods use track embeddings or object queries, but remain specifically designed for video data.
- Mask2Former achieves state-of-the-art video instance segmentation without modifying its architecture, loss, or training pipeline.
- For more than one frame, shared queries let Mask2Former segment and track object instances across frames.
- 60.4 AP on YouTubeVIS-2019 and 52.6 AP on YouTubeVIS-2021 result from directly predicting 3D volumes that track instances across time.
2. Background
Video instance segmentation extensions typically process frames independently with post-processing or require video-specific architectural changes. Mask2Former instead processes the 3D video volume while retaining the image model's architecture, loss, and training pipeline.
- Per-frame methods independently segment each frame and associate predicted masks across frames during post-processing.
- Per-instance track embeddings and instance center offsets are examples of added mechanisms for extending image models to video.
- Video-specific transformer models process 3D volumes through cross-attention but disconnect image and video segmentation research.
- Mask2Former achieves state-of-the-art video instance segmentation without modifying the architecture, loss, or training pipeline.
3. Mask2Former for Videos
Mask2Former adapts to video segmentation by processing videos as 3D spatio-temporal volumes, applying masked attention and temporal positional encoding, and directly predicting 3D instance masks.
- 3. Mask2Former for Videos: Videos are represented as T × H × W spatio-temporal volumes, enabling Mask2Former to process temporal and spatial features jointly.T denotes frames, while H and W denote spatial dimensions.
- 3.1. Joint spatio-temporal masked attention: Masked attention is applied to spatio-temporal features using a binary mask from the previous decoder layer, with disallowed locations assigned −∞.The mask is obtained by resizing and thresholding the previous 3D mask prediction at 0.5.
- 3.1. Joint spatio-temporal masked attention: The attention computation uses query features with transformed spatio-temporal keys and values, while classification remains unchanged.Queries are generated from the previous decoder-layer features, and keys and values cover T H_l W_l locations.
- 3.2. Temporal positional encoding: Temporal positional encoding is decoupled from spatial encoding by adding non-parametric sinusoidal temporal and spatial encodings with broadcasting.Both encodings can handle arbitrary sequence lengths.
- 3. Mask2Former for Videos: Instance masks are predicted as 3D volumes through a dot product between query mask embeddings and spatio-temporal pixel embeddings.A sigmoid converts the dot-product scores into mask predictions over time and space.
4. Experiments
The experiments evaluate Mask2Former on YouTubeVIS-2019 and YouTubeVIS-2021 using unchanged architecture, loss, and training pipeline, with whole-video inference producing 3D masks without post-processing.
- 4. Experiments: Mask2Former is evaluated on the YouTubeVIS-2019 and YouTubeVIS-2021 video instance segmentation benchmarks without modifying the architecture, loss, or training pipeline.The implementation follows IFC settings and uses AdamW with a step learning-rate schedule.
- 4. Experiments: Whole-video inference produces 3D mask predictions without post-processing, supports variable-length sequences, and retains the top 10 predictions per video.Inference resolution depends on the backbone: 360 pixels for ResNet and 480 pixels for Swin Transformer.
- 4.3. Results: More than 6 AP separates Mask2Former from IFC under the exact same training parameters.Mask2Former also outperforms SeqFormer without using extra COCO images for data augmentation.