PaperlayerView exact revision on arXiv

Source-linked AI summary

LVMT: Video Mask Transformer for Long-term Video Segmentation

Narges Norouzi, Niccolò Cavagnero, Idil Esen Zulfikar, Bastian Leibe, Gijs Dubbelman, Daan de Geus

arXiv:2609.34895v2cs.CV

TL;DR

Efficient online video segmentation models struggle to track objects through long occlusions, and training on long videos is constrained by memory and gradient issues. LVMT combines adaptive GRU memory with chunk-based Truncated Query Propagation. It outperforms state-of-the-art methods across video segmentation tasks, including on OVIS, while retaining efficiency.

  • Problem

    Efficient video segmentation methods struggle with long-term tracking because query propagation cannot adaptively retain relevant object information and long-video training is difficult.

    Source: §1 · pp. 1–2

  • Method

    LVMT combines lightweight GRU memory that adaptively propagates object information with Truncated Query Propagation, which carries queries across video chunks while truncating backpropagation within each chunk.

    Source: §1 · p. 2

  • Results

    LVMT outperforms state-of-the-art methods across video segmentation tasks; on OVIS, it beats DVIS-DAQ by +2.4 AP and is over 10× faster.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The fixed-size GRU state avoids growing memory use and computation, while TQP adds no inference-time overhead or architecture changes.

    Source: §1 · p. 2

    TQP bounds peak training memory without changing inference cost, but sequential chunk processing increases training time and computation with the number of chunks.

    Source: D. Limitation · p. 15

Abstract

from arXiv · show

Existing online video segmentation methods struggle to track objects in long, complex videos with long-term occlusions. We hypothesize that this limitation is caused by (i) the inability of their temporal propagation mechanism to adaptively select the object information that is propagated across time, and (ii) their inability to be trained on long videos due to memory requirements and vanishing gradients. To address the first limitation, we propose to use a lightweight GRU-based temporal propagation module that can learn to select which information it keeps in memory and propagates across time. Second, to allow training on long videos, we introduce Truncated Query Propagation (TQP), a training strategy in which the model processes a video in chunks of frames, where information about tracked objects is propagated between chunks but backpropagation is only conducted in individual chunks, enabling longer temporal supervision without out-of-memory issues, inference overhead, or vanishing gradients. The resulting model is called the Long-term Video Mask Transformer (LVMT). Extensive experiments on six benchmarks show that LVMT sets a new state of the art across a range of video segmentation tasks, while retaining the speed of the highly efficient model it is based on, making it 10X faster than the prior state of the art. Code: https://www.tue-mps.org/lvmt

1. Introduction

LVMT targets long-term tracking failures in efficient video segmentation by adding adaptive GRU memory and chunk-based training. It improves accuracy while preserving efficiency, including on benchmarks involving long-term occlusion.

1. Introduction

  • On OVIS, LVMT exceeds PMT by at least +4.6 AP across ViT-L/B/S at similar prediction speeds, and beats DVIS-DAQ by +2.4 AP while being over 10× faster.The paper reports that the gains are further validated on video instance, panoptic, and semantic segmentation benchmarks.Source: Figure 1 · p. 1, §1 · p. 2
  • Existing efficient methods struggle with identity switches in long, complex videos, especially when objects are occluded for long periods.Their query propagation cannot adaptively select object information, and repeatedly adding generic queries can dilute information about occluded objects.Source: §1 · pp. 1–2
  • A lightweight GRU memory adaptively selects information from the previous time step to retain and propagate, potentially preserving occluded-object information for re-identification.Unlike a growing query history, the GRU-based memory is intended to avoid increasing memory use and computation with video length.Source: §1 · p. 2
  • Short-clip training leaves some long disappearance spans beyond the training horizon, while longer recurrent training can cause out-of-memory errors and vanishing gradients.The authors identify these as practical barriers to improving identity recovery through longer video training.Source: §1 · p. 2
  • Truncated Query Propagation processes videos in chunks, carries queries across chunk boundaries, and truncates backpropagation within each chunk.The authors state that this enables longer-term training without memory issues or vanishing gradients; as a training-time strategy, TQP adds no inference overhead or architecture changes.Source: §1 · p. 2

2. Related Work

Prior video segmentation work seeks simpler and faster architectures, but long-term tracking remains difficult for efficient query-propagation models. LVMT combines lightweight GRU-based propagation with chunk-based training to target long-horizon consistency.

2. Related Work

  • Universal video segmentation models address video instance, panoptic, and semantic segmentation, often using separate per-frame segmenters and temporal trackers that increase complexity and reduce efficiency.Recent encoder-based approaches replace many of these specialized components with large, extensively pretrained ViTs.Source: §2 · p. 2
  • VidEoMT uses a large pretrained ViT and simple temporal query propagation, while PMT adds a lightweight decoder so the encoder can remain frozen and be reused across tasks.Both retain efficient designs but struggle with long-term tracking because query propagation is non-adaptive and long-video training is difficult.Source: §2 · p. 2
  • Explicit query memories can retain object queries indefinitely, but their memory and computation grow with video length; VOS memory-bank methods also scale poorly with video length and object count.TrackFormer’s short-term object history has also been described as inefficient for video segmentation because duplicate-query removal requires mask non-maximum suppression.Source: §2 · pp. 2–3
  • SAM3 uses vision and text encoders plus task-specific modules, and its speed decreases as the number of tracked entities increases.LVMT instead aims for efficient long-term modeling with lightweight GRU-based propagation and truncated chunk-based training.Source: §2 · p. 3

3. Preliminaries

Online video segmentation predicts masks and labels over time while maintaining object correspondences using only current and previously observed frames. The preliminaries describe encoder-based segmentation and query propagation methods that inform LVMT.

3. Preliminaries

  • Online video segmentation predicts a mask and category label for each object at each time step while maintaining stable correspondences across frames.Predictions at time t may depend only on the current frame and previously observed frames; “object” can mean instances in VIS, classes in VSS, or both in VPS.Source: §3 · p. 3
  • EoMT uses learnable object queries within a ViT encoder and a lightweight head to produce a mask and class label for each query.Its streamlined design removes complex decoder components described as largely redundant at increased model and pretraining scale.Source: §3 · p. 3
  • VidEoMT propagates previous-frame output queries by adding their linear projections to learnable queries before processing the current frame.The propagated queries carry temporal information, while learnable queries support detection of newly appearing objects.Source: §3 · p. 3
  • PMT moves query processing into a separate lightweight decoder, allowing its ViT encoder to remain frozen and support multi-task deployment.EoMT and VidEoMT instead require full-encoder fine-tuning because pretrained attention layers must adapt to injected queries.Source: §3 · p. 3

4. Long-term Video Mask Transformer

LVMT extends PMT with GRU-based query propagation and Truncated Query Propagation (TQP) to model long-term information while training on longer videos. The section describes how the GRU updates query memory and how TQP carries temporal state across chunks while truncating gradients.

4.1. GRU-based Query Propagation

  • The GRU replaces fixed query fusion with an adaptive hidden state updated from previous-frame output queries, allowing selective retention of object information across time.Learnable queries initialize the hidden state; the updated state is fed to the current frame’s decoder and can retain information about absent objects for re-identification.Source: §4.1 · pp. 3–4

4.2. Truncated Query Propagation (TQP)

  • Long disappearance spans exceed the model’s short training horizon, while naively extending clips can worsen accuracy through vanishing gradients and increase memory use.The analyzed videos’ average disappearance span is about 2.6× the training clip length.Source: §4.2 · p. 5
  • TQP applies truncated backpropagation to GRU query propagation so long-horizon supervision does not require backpropagating through the full video sequence.The method is intended to keep training memory-efficient and improve optimization stability.Source: §4.2 · p. 5
  • TQP divides videos into ordered chunks, carries the hidden state forward after detaching it, and accumulates chunk gradients for one optimizer update after the video.This preserves temporal information across chunks while bounding the computational graph and avoiding full-sequence backpropagation.Source: §4.2 · p. 5

5. Experiments

The experiments evaluate LVMT across video instance, panoptic, and semantic segmentation benchmarks, alongside tracking and computational-efficiency measures. The section also specifies the default training configuration and comparison protocol.

Datasets

  • The evaluation covers six standard benchmarks spanning video instance, panoptic, and semantic segmentation.VIS includes OVIS, YouTube-VIS 2022, 2019, and 2021; VPS uses VIPSeg, and VSS uses VSPW.Source: §5 · p. 5

Implementation Details

  • By default, models use a frozen ViT-L encoder and train on temporal chunks of 5 frames within a 15-frame window.The setup corresponds to 3 chunks; comparisons use PMT’s batch size, learning rate, and scheduler, and prior-work resolutions and iteration counts.Source: §5 · p. 6

Performance Metrics

  • Performance measures include task-specific segmentation metrics, identity-preservation and tracking metrics, and inference-speed and compute measures.VIS uses AP and AR; VPS uses VPQ and STQ; VSS uses mIoU and mVC; tracking includes IDF1, AssA, MT, ML, and IDS; efficiency reports FPS and GFLOPs.Source: §5 · p. 6

6. Results

The results evaluate LVMT across video segmentation benchmarks and ablate its memory and training design. They report accuracy gains over prior methods and show how GRU propagation and TQP contribute to long-clip performance.

6.1. Main Results

  • On OVIS, LVMT outperforms PMT by +4.6 AP while preserving simplicity and efficiency.TQP adds +3.8 AP over naive longer-clip training while maintaining inference efficiency.Source: §6.1 · p. 6, Table 1 · p. 6

6.2. Comparison with State-of-the-Art Models

  • On VIS, LVMT exceeds frozen-encoder CAVIS by +3.3 AP on OVIS and +9.3 AP on YouTube-VIS 2022, and is over 10× faster than fine-tuned DVIS-DAQ and LOMM.LVMT also beats DVIS-DAQ by +2.4 AP on OVIS and LOMM by +3.3 AP on YouTube-VIS 2022.Source: §6.2 · p. 7, Table 2 · p. 7
  • On VIPSeg VPS, LVMT improves over fine-tuned DVIS-DAQ by +2.9 VPQ with DINOv3 and is over 10× faster than DVIS-DAQ with DINOv2.With DINOv2, the models differ by 0.8 VPQ; LVMT is faster and allows encoder reuse.Source: §6.2 · p. 7, Table 4 · p. 7
  • On VSPW VSS, LVMT exceeds VidEoMT by +0.9 mVC and +2.4 mIoU with DINOv3, and PMT by +0.4 mVC and +0.7 mIoU.The comparisons use frozen encoders and report only slightly lower FPS than VidEoMT and virtually identical FPS to PMT.Source: §6.2 · pp. 7–8, Table 5 · p. 7

6.3. Ablations

  • The GRU offers the best overall trade-off among tested memory mechanisms, with the highest AP while remaining among the fastest variants.It improves over LSTM by +0.7 AP and the memory-bank baseline by +2.2 AP.Source: §6.3 · p. 8, Table 6 · p. 8
  • On 10-frame training, GRU and TQP together reach 56.5 AP versus 54.5 AP without GRU; without TQP, clips of 15 frames or longer cause out-of-memory errors.Naive 10-frame training without GRU also yields 51.1 vs. 52.6 AP, suggesting the gradient issue also affects PMT’s default query fusion; increasing training from 15 to 20 frames adds only +0.1 AP.Source: §6.3 · p. 8, Table 7 · p. 8

7. Conclusion

LVMT combines adaptive GRU memory with TQP to address long-term video tracking. It outperforms state-of-the-art methods across video segmentation tasks while retaining high efficiency.

7. Conclusion

  • LVMT combines lightweight GRU memory, which adaptively propagates relevant information, with TQP’s chunk-based training for stable, memory-efficient optimization on long sequences.TQP truncates backpropagation within video chunks while propagating queries across chunk boundaries.Source: §7 · p. 8
  • LVMT outperforms state-of-the-art methods across all video segmentation tasks while being over 10× faster.Source: §7 · p. 8
  • LVMT is consistently more accurate than the efficient PMT baseline at comparable speed.Source: §7 · p. 8

Supplementary Material

The supplementary material includes implementation details, additional experiments, ablations, and qualitative results. Its training notes specify the backbone, decoder initialization, temporal chunking, optimization settings, and supervision choices.

Appendix

  • The appendix contents list implementation details, additional experiments and ablations, and qualitative results.Source: Appendix · p. 11

A.1. Training

  • Training uses frozen pretrained DINOv3 encoders by default, a decoder initialized from PMT checkpoints, and a GRU query-propagation module trained from scratch on each target video dataset.The decoder is pretrained for image-level COCO instance segmentation without temporal supervision.Source: §A.1 · p. 11

A.2. Hyperparameters

  • Default training uses 5-frame chunks within a 15-frame window, with 200 learnable queries and shared-parameter GRU cells; supervision combines classification cross-entropy, binary cross-entropy, and Dice losses.The appendix also specifies optimizer and schedule settings, fixed ground-truth-to-query assignments across frames, and deep supervision across decoder layers.Source: §A.2 · p. 11

A.3. Evaluation

LVMT is evaluated on YouTube-VIS benchmarks and through analyses of tracking consistency, model size, and encoder fine-tuning. Across these comparisons, it improves accuracy while retaining efficiency, with performance gains depending on benchmark difficulty and backbone setting.

A.3. Evaluation

  • Evaluation processes videos online, one frame at a time, and measures computational cost in GFLOPs and runtime speed in FPS.FPS and GFLOPs are averaged over validation frames, with measurements conducted on a single NVIDIA H100 GPU.Source: §A.3 · p. 11

A.4. Tracking metrics

  • Tracking evaluation reports IDF1, AssA, track completeness through MT and ML, and total identity switches, with fewer switches indicating more stable identity preservation.MT counts objects tracked for at least 80% of their lifespan; ML counts objects tracked for less than 20%.Source: §A.4 · p. 11

B.1. Comparison with State-of-the-Art Models

  • On YouTube-VIS 2019 and 2021, DINOv2-pretrained LVMT beats LOMM by +1.0 AP and +1.4 AP, respectively, while being more than 10× faster.LVMT also improves over VidEoMT and PMT while maintaining comparable efficiency; gains are smaller on the less challenging, more saturated YouTube-VIS 2019 and 2021 datasets.Source: §B.1 · p. 12

B.2. Identity Consistency Analysis

  • High-IDS OVIS videos contain approximately 3× more objects, nearly 5× higher disappearance rates, and more than twice the mean disappearance length of low-IDS videos.Mean disappearance length is 13 frames, 2.6× the model's five-frame training horizon; the analysis motivates longer-horizon training with TQP.Source: §B.2 · p. 12

B.3. Effect of GRU and TQP on VidEoMT

  • On VidEoMT, GRU propagation adds +1.7 AP, TQP without GRU adds +2.0 AP, and combining both improves over baseline by +2.3 AP.The gains are smaller than for LVMT, and the authors hypothesize encoder fine-tuning on limited video data may increase overfitting and limit long-horizon supervision.Source: §B.3 · pp. 12–13, Table C · p. 13

B.4. Effect of Model Size

  • Across ViT-S, ViT-B, and ViT-L, LVMT achieves higher AP than VidEoMT and PMT at similar FPS to PMT.Its advantage over PMT grows as the backbone becomes smaller.Source: §B.4 · p. 13, Table D · p. 13

B.5. Effect of Encoder Fine-Tuning

  • With DINOv3, encoder fine-tuning yields negligible or negative AP gains, while frozen-encoder LVMT remains strongest overall at essentially the same computational cost.With DINOv2, fine-tuning improves AP for both PMT and LVMT.Source: §B.5 · p. 13

B.6. Analysis of Temporal Gradient Flow

The analysis finds that TQP preserves stronger gradients within chunks while carrying query values across chunk boundaries. It also examines GRU memory retention during occlusion and the effects of chunk size and hidden-state initialization.

B.6. Analysis of Temporal Gradient Flow

  • With TQP, within-chunk gradient retention rises from approximately 5% to 37% as the backward path is shortened.Query values still cross chunk boundaries, but gradients do not propagate into preceding chunks.Source: §B.6 · p. 14

B.7. GRU Memory Retention Under Occlusion

  • During occlusion, the update-gate value for occluded-object queries rises from approximately 0.66 to 0.75, indicating about 10% more retention of information from visible frames.The update-gate values for unmatched queries and queries of visible objects remain virtually unchanged; the gate value falls toward its pre-occlusion level as objects reappear.Source: §B.7 · pp. 14–15

C.1. Effect of Chunk Size on TQP

  • TQP performs best with a five-frame chunk; one-frame chunks provide too little temporal context, while ten-frame chunks reduce AP to 54.3.The authors use F = 5 frames in the main experiments.Source: §C.1 · p. 15

C.2. Effect of Hidden-State Initialization.

  • Initializing the GRU hidden state with shared learnable object queries performs best among the tested strategies.The authors attribute this to the queries' task-aligned optimization for object classification and mask prediction; zero, random, and dedicated learnable initializations perform worse.Source: §C.2 · p. 15

D. Limitation

TQP bounds peak training memory without changing inference cost, but increases training time through sequential chunk processing.

D. Limitation

  • TQP increases wall-clock training time and computation because each iteration requires a forward and backward pass for every video chunk.The number of chunk passes is M = ⌈Ttrain/F⌉; retaining TQP's benefits without added training time is identified as future work.Source: D. Limitation · p. 15

E. Qualitative Results

In the selected heavily occluded OVIS sequence, LVMT preserves object identities where PMT switches identities among visually similar motorcycles and riders.

E. Qualitative Results

  • On the illustrated OVIS sequence, PMT makes identity switches at frames 4 and 8 under heavy occlusion, while LVMT preserves more consistent identities.The frames compare PMT, LVMT, and ground-truth annotations for visually similar motorcycles and riders.Source: E. Qualitative Results · p. 15, Figure C · p. 16

Related papers

Loading 2609.34895v2…