Source-linked AI summary
VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
Narges Norouzi, Idil Esen Zulfikar, Niccolò Cavagnero, Tommie Kerssies, Bastian Leibe, Gijs Dubbelman, Daan de Geus
TL;DR
Existing video segmentation systems rely on complex specialized components for segmentation and tracking, raising the question of whether a simpler model can retain their accuracy. VidEoMT moves both functions into a ViT encoder through query propagation and fusion, achieving comparable accuracy with 5×–10× speedups and up to 160 FPS. The results suggest that a sufficiently large, well-pretrained ViT can replace much of the functionality previously handled by downstream components.
Problem
Existing video segmentation models combine segmenters with complex specialized tracking components, creating architectural complexity and computational overhead.
Method
VidEoMT unifies segmentation and temporal association in an encoder-only ViT using query propagation and query fusion.
Results
VidEoMT achieves comparable accuracy with 5×–10× speedups across multiple benchmarks, reaching up to 160 FPS with a ViT-Large backbone.
Takeaways & Limitations
A sufficiently large and well-pretrained ViT can take over much of the functionality previously handled by complex downstream video-segmentation components.
Takeaways & Limitations
Online predictions may depend only on the current and earlier frames, and query propagation can reduce recognition of newly appearing objects as learned-query influence diminishes.
Abstract
from arXiv · showhide
Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/
1. Introduction
VidEoMT asks whether video segmentation can match complex models with a simpler encoder-only design. It uses query propagation and fusion within a ViT, achieving comparable accuracy with substantially higher speed.
- Video segmentation requires per-frame object localization and classification plus temporal matching across frames.
- Existing methods combine specialized components for segmentation and tracking, motivating a simpler alternative.
- VidEoMT replaces complex specialized components with a single ViT-style model whose large-scale pre-training supplies much of their functionality.
- VidEoMT unifies segmentation and temporal association within one ViT encoder using query propagation and query fusion.Propagation carries object queries across frames, while fusion combines them with learned queries to identify newly appearing objects.
- Up to 10× faster, VidEoMT achieves accuracies comparable to the state of the art across video segmentation benchmarks.With ViT-Large, it reaches up to 160 FPS on YouTube-VIS and reports 5×–10× speedups on VIPSeg and VSPW with negligible accuracy impact.
2. Related Work
Video segmentation extends image segmentation by requiring objects to be segmented, classified, and tracked over time. Existing methods typically separate segmentation from tracking and add specialized components that improve accuracy but reduce efficiency.
- Video segmentation includes video instance, panoptic, and semantic segmentation, with the shared objective of segmenting, classifying, and tracking objects.
- Mask Transformer architectures from image segmentation are extended to video by adding specialized tracking components or enhancing temporal representations.
- Universal video segmentation models typically use a decoupled pipeline in which a segmenter generates frame-level predictions and a tracker associates them over time.
- Specialized components generally increase accuracy but reduce efficiency, motivating analysis of whether these models can be simplified.
3. Method
VidEoMT reformulates online video segmentation as an encoder-only ViT pipeline that integrates segmentation and temporal association without a dedicated tracker. It propagates object queries across frames and fuses them with learned queries to preserve temporal continuity while detecting newly appearing objects.
- Task Definition: Online video segmentation must produce masks and class labels per frame while associating the same objects across time.
- Preliminaries: Existing models typically separate frame-level segmentation from temporal association through a segmenter and a tracker.
- VidEoMT: Inspired by EoMT, the method injects learned queries into the final ViT layers to produce segmentation predictions without specialized decoder components.EoMT showed that encoder-only image segmentation can remain competitive while improving efficiency; VidEoMT extends this simplification to video.
- VidEoMT: VidEoMT replaces this decoupled pipeline with a single ViT-style encoder that performs segmentation and temporal modeling jointly.
- VidEoMT: Query propagation carries object-level queries across frames, enabling temporal continuity within the encoder.
- VidEoMT: Query fusion combines transformed previous-frame queries with learned queries, balancing temporal context with adaptability to newly appearing objects.The learned and propagated queries are combined by element-wise addition, supported by supervision that preserves query order across frames.
4. Experiments
VidEoMT is evaluated across six video-segmentation benchmarks spanning instance, panoptic, and semantic segmentation. The experiments report task-specific accuracy metrics alongside FPS and FLOPs for computational-efficiency assessment.
- Datasets and Evaluation Metrics: Six benchmarks cover VIS, VPS, and VSS: OVIS and three YouTube-VIS versions, VIPSeg, and VSPW.
- Datasets and Evaluation Metrics: VIS uses AP and AR, VPS uses VPQ and STQ, and VSS uses mIoU and mVC.
- Implementation Details: Training uses a DINOv2-pretrained ViT backbone, five-frame temporal windows, mixed precision, AdamW, and layer-wise learning-rate decay.The default batch size is 8, with learning rate 10^-4 and layer-wise decay factor 0.6.
- Computational Efficiency: Efficiency is measured with FPS and GFLOPs, using batch size 1 on an NVIDIA H100 with FlashAttention v2 and torch.compile enabled.
5. Results
VidEoMT progressively removes specialized tracking components and restores temporal modeling with lightweight query mechanisms. Across video segmentation benchmarks, it delivers comparable or better accuracy with substantially higher speed.
- Stepwise Reduction: Removing context-aware features and re-identification layers increases speed by 1.8× to 74 FPS with almost no accuracy impact.
- Stepwise Reduction: Eliminating the tracker raises speed from CAVIS’s 15 FPS to 162 FPS but causes a 7.6 AP drop.
- Stepwise Reduction: Query propagation improves AP by +2.6 without increasing computational cost, while query fusion recovers nearly all original accuracy.
- Main Results: On VIPSeg, VidEoMT sacrifices 2.2 VPQ relative to DVIS-DAQ while delivering nearly 19× higher speed.
- Main Results: VidEoMT outperforms DVIS++ on VSPW by +2.1 mIoU and +0.8 mVC16 while running more than 5× faster.
- Further Comparisons: Compared with EoMT plus CAVIS, VidEoMT achieves slightly better AP while being nearly 4× faster.
6. Conclusion
VidEoMT unifies segmentation and temporal association within one ViT encoder, replacing heavy specialized modules with query propagation and fusion. It preserves or improves accuracy while achieving order-of-magnitude speedups across video segmentation benchmarks.
- VidEoMT unifies segmentation and temporal association within a single ViT encoder.
- Heavy specialized modules are replaced with lightweight query propagation and an efficient query fusion mechanism.
- VidEoMT achieves an order-of-magnitude speedup while preserving or improving accuracy across multiple video segmentation benchmarks.
Supplementary Material
The supplementary material contains implementation details and additional experiments.
- The supplementary material includes implementation details, additional experiments, and qualitative results.
- Implementation Details: VidEoMT uses a DINOv2-pretrained ViT backbone and is trained first for image segmentation, then fine-tuned with temporal modeling for video segmentation.
A.2. Evaluation
The evaluation uses frame-by-frame online processing and progressively removes specialized components from CAVIS to assess encoder-only designs and training choices.
- Evaluation setup: Evaluation processes videos frame by frame for online segmentation and measures efficiency using FPS and GFLOPs.Metrics are measured on a single NVIDIA H100 GPU with batch size one, after warm-up iterations.
- Component ablation: The ablation replaces CAVIS’s segmenter with EoMT before removing context-aware features, re-identification layers, and tracker blocks.These steps isolate the contribution of specialized components in the original architecture.
- Component ablation: The final ablation propagates segmentation queries from frame t −1 into frame t, then fuses propagated queries with learnable queries.Query fusion is the resulting architecture’s mechanism for combining temporal information with learned queries.
- Training: Training uses the same settings as CAVIS for subsequent ablations, including sampled frames, batch sizes, GPUs, and task-specific iteration schedules.The supplied passage specifies eight GPUs, batch size eight, and five sampled frames, while the iteration schedule varies by dataset.
- Training: Optimization uses AdamW with a 10^-4 learning rate, layer-wise decay, polynomial decay, and two-stage linear warm-up.The loss combines cross-entropy for classification with binary cross-entropy and Dice losses for segmentation.
A.5. Architectures of Alternative Approaches
The alternative architectures model temporal information in a decoder, including a TrackFormer-style strategy that propagates selected track queries alongside learned queries.
- Alternative architectures: The comparison uses a DINOv2 plus ViT-Adapter encoder and a nine-layer Mask2Former-style Transformer decoder.Temporal modeling is tested with query fusion and a TrackFormer-based design.
- TrackFormer-based design: TrackFormer-style modeling keeps first-frame queries whose classification score exceeds 0.8, converts them into track queries, and concatenates them with 400 learned queries in later frames.The decoder updates propagated track queries and adds newly detected queries above the same threshold.
- TrackFormer-based design: Track queries are removed when their score remains below 0.8 for five consecutive frames, indicating that the tracked object has disappeared.This provides an explicit persistence rule for propagated objects.
B.1. Query Propagation Methods
Query propagation experiments compare no propagation, direct propagation, reset rules, TrackFormer-style propagation, and VidEoMT’s fusion strategy across accuracy and efficiency.
- Propagation strategies: VidEoMT fuses learnable queries with propagated track queries and compares this design with alternative temporal propagation methods.The comparison is presented as an evaluation of query propagation strategies.
- Propagation strategies: The no-propagation variant performs worst because it lacks explicit temporal modeling.It receives only learnable queries, similarly to EoMT, while being fine-tuned for video segmentation.
- Propagation strategies: Direct propagation introduces temporal modeling but struggles to detect new objects as the influence of learnable queries diminishes over time.This exposes the adaptability trade-off in carrying only previous-frame output queries forward.
- Propagation strategies: Non-object reset improves over direct propagation but still underperforms the default fusion approach.The reset replaces a propagated query with a learnable query when it predicted no object in the previous frame.
- Propagation strategies: TrackFormer-style propagation is slightly less accurate and considerably slower than fusion because it filters duplicate detections before propagation.The supplied results identify fusion as the most accurate and efficient strategy.
- Model comparison: VidEoMT consistently outperforms extending EoMT with a CAVIS tracker in both efficiency and accuracy across backbones.This comparison highlights the encoder-only design against a tracker-augmented alternative.
B.3. Impact of Pre-training
The section attributes part of the speed difference among pretrained backbones to rotary positional embeddings and tests this attribution by disabling RoPE.
- Pre-training backbones: DINOv3 and EVA-02 are slower than DINOv2 despite similar GFLOPs, and the paper attributes significant slowdown to rotary positional embeddings.RoPE introduces additional element-wise operations in attention layers.
- Pre-training backbones: Disabling RoPE makes the affected models faster, supporting the claim that RoPE is a main source of slowdown.The passage presents this as confirmation from the backbone comparison.
C. Qualitative Results
The paper presents qualitative comparisons of CAVIS and VidEoMT for video instance and panoptic segmentation across three datasets. These visualizations cover YouTube-VIS 2019, OVIS, and VIPSeg.
- Figures B to D visualize predictions from both methods for VIS and VPS across YouTube-VIS 2019, OVIS, and VIPSeg.
- Qualitative video instance segmentation results compare CAVIS with VidEoMT on YouTube-VIS 2019.
- Qualitative video instance segmentation results compare CAVIS with VidEoMT on OVIS.
- Qualitative video panoptic segmentation results compare CAVIS with VidEoMT on VIPSeg.