SAM-MT: Real-Time Interactive Multi-Target Video Segmentation

Ruiqi Shen; Chang Liu; Henghui Ding

Source-linked AI summary

SAM-MT: Real-Time Interactive Multi-Target Video Segmentation

Ruiqi Shen, Chang Liu, Henghui Ding

arXiv:2607.08688v1Jul 9, 2026cs.CV

TL;DR

Multi-target video segmentation often repeats single-target processing, causing frame rates to fall as targets increase. SAM-MT uses target queries and shared global context to decouple latency from target count, achieving 36+ FPS for 10 targets while maintaining SAM2-level segmentation performance.

Problem
Existing video segmentation methods repeat object-wise processing for each target, limiting real-time multi-target segmentation as target count increases.
Method
SAM-MT represents targets with explicit queries alongside shared global context, using decoupled masked attention and sparse memory for identity-preserving propagation.
Results
36+ FPS for 10 targets, with performance on par with SAM2.1-B+ across six VOS benchmarks.
Takeaways & Limitations
SAM-MT provides near-single-object efficiency as target number increases while maintaining robust video segmentation performance.
Takeaways & Limitations
Because SAM-MT inherits SAM’s vision-only architecture, it lacks reasoning capability for complex high-level tasks.

Abstract

from arXiv · show

Modern Video Object Segmentation (VOS) involves tracking and segmenting user-specified targets. While recent approaches have achieved remarkable performance in single-target scenarios, extending them to multi-target settings typically involves replicating the single-target processing for each individual object, resulting in reduced frame rates (FPS) with unbounded latency as target count increases. Built upon Segment Anything 2 (SAM2), we propose SAM-MT, which addresses this by transforming the model into an interactive framework for real-time Multi-Target video segmentation. SAM-MT uses explicit queries to represent different individual targets, in parallel with a shared representation for global context. It employs decoupled masked attention to keep individual identities distinct from cross-target interference, and sparse memory for stable temporal evolution, along with specialized strategies for occlusion handling and overlap prevention. SAM-MT successfully decouples latency from the number of targets, achieving real-time speed on par with single-target baselines (>36 FPS for 10 targets) while maintaining SAM2's robust video segmentation performance.

1 Introduction

SAM-MT transforms SAM2 into a real-time interactive framework for multi-target video segmentation by combining shared global context with target-specific queries and decoupled temporal processing. It preserves target identities and SAM2-level segmentation performance while decoupling computational cost from target count.

Motivation: Existing VOS methods rely on object-wise memory and propagation, so multi-target processing repeats computations and computation grows with target count.This limits their suitability for real-time multi-target tracking and segmentation, particularly in crowded scenes requiring identity preservation.
Framework: SAM-MT combines a shared representation for global context with parallel target queries representing individual targets.This hybrid design extends the SAM2 architecture while avoiding redundant object-wise processing.
Identity Preservation: Decoupled masked attention prevents cross-target interference while queries share global context, preserving individual target identities throughout sequences.Attention between queries of different targets is restricted, while shared access to global context remains available.
Temporal Robustness: A lightweight query-based sparse memory captures individual targets’ temporal evolution for per-target re-identification across frames.Strided frame sampling improves occlusion handling under GPU constraints, and overlap-prevention supervision enhances instance-level separation.
Results: 36+ FPS is achieved for 10 targets, a 6× speedup over SAM2.1-B+, while SAM-MT delivers competitive performance across six challenging VOS benchmarks.The framework maintains near-single-object efficiency as target density increases and performs on par with SAM2.1-B+ across six VOS benchmarks.

2 Related Work

Prior VOS systems are primarily single-target or object-wise, so extending them to multiple targets requires repeated per-object processing and makes streaming costly. Query-based transformers instead use learnable queries that cross-attend to pixel-level features for target representations and mask prediction.

The single-target bottleneck of VOS: STM-style VOS is inherently single-target, matching dense pixel-level memory for a specific object against query frames.Multi-target extension typically replicates the single-object pipeline for each target.
The single-target bottleneck of VOS: STCN, XMem, and Cutie share frame-level image features but retain independent mask decoding and memory encoding for each object.Thus, feature sharing does not remove object-wise processing.
The single-target bottleneck of VOS: SAM-family video segmentation methods still replicate the single-object pipeline for each target, making multi-target streaming costly.This preserves the object-wise processing paradigm despite their success in video segmentation.
Query-based Segmentation: Query-based transformers use learnable queries that cross-attend to pixel-level features to produce target representations, followed by task-specific mask-prediction heads.Established query-based approaches have surpassed traditional CNN-based segmentation baselines.

3 Method

SAM-MT converts video segmentation into a scalable interactive multi-target framework by assigning explicit target queries alongside shared global context. Decoupled attention, query-based sparse memory, and identity-aware temporal updates preserve target separation and identity consistency without replicating dense per-object processing.

Framework overview: SAM-MT uses point-based interactions with target IDs and explicit target queries, enabling arbitrary numbers of user-specified targets alongside shared global queries.Input points are encoded as queries and combined with SAM2’s global queries in the decoder.
Decoupled masked attention: Decoupled masked attention blocks cross-target interference while allowing each target query to attend to shared global context and image features.The masking rule isolates queries belonging to different targets, while global queries remain accessible to all targets.
Target-query consolidation: Each target’s refined point queries are consolidated into one target query for mask prediction and propagation, while subsequent frames directly update the propagated single query.Consolidation occurs only in the initial frame; later frames use one propagated target query per target.
Query-based sparse memory: SAM-MT uses dense memory only for the combined target mask and stores individual target queries in a shared FIFO sparse memory over a temporal window.Sparse memory reduces per-target storage from HW dense tokens to one target-query token, enabling longer temporal windows.
Identity transformer: An identity transformer retrieves each target’s historical queries using an identity-aware mask, preserving identity consistency and mitigating identity drift across frames.Each target query attends only to its own history in the sparse memory.

4 Experimental Setup

SAM-MT is trained in two stages, progressing from static click-based multi-target image segmentation to video training for temporal identity consistency. Experiments evaluate the method across six challenging VOS benchmarks under an interactive initialization protocol.

Training Data: The model is trained on SA-V sequences containing at least three concurrent targets, representing approximately 35% of the original training set.This filtering encourages robust multi-target handling.
Training Scheme: Two-stage training first targets one-shot multi-target image segmentation from clicks, then promotes temporal identity consistency across video frames.The second stage extends training from images to sequences.
Strided Sampling: Eight frames are sampled per sequence using consecutive and 4-frame-strided sampling across a 32-frame window to capture short- and long-term dynamics.This sampling targets motion, occlusion, and reappearance while respecting GPU memory constraints.
Point Sampling: Training simulates user interaction by sampling 1–5 positive and 0-2 negative points per target in the initial frame.Grid-based and random sampling improve spatial coverage.
Evaluation Protocol: Evaluation covers six challenging VOS benchmarks, while SAM-MT and SAM2 receive two positive clicks per target and other baselines receive ground-truth masks.The benchmarks include MOSEv2, MOSEv1, LVOSv2, LVOSv1, SA-V validation, and SA-V test splits.

5 Experiments

SAM-MT delivers competitive zero-shot video segmentation while preserving near-single-object efficiency as target counts increase. Experiments show that its decoupled attention, identity-aware memory, and sparse-memory design improve scalability, identity consistency, and accuracy–speed trade-offs.

Segmentation Performance: 43.0 J & ˙F on MOSEv2 demonstrates strong zero-shot segmentation, with gains of 2.0 and 2.3 J &F points over SAM2.1-B+ on LVOSv2 and LVOSv1.SAM-MT also achieves competitive J &F on both SA-V splits.
Scalability: SAM-MT maintains near-single-object efficiency as target density scales, achieving the highest FPS across target counts despite processing 1024p inputs.The synthetic benchmark contains 20 sequences with target counts from 1 to 20, each padded or truncated to 100 frames.
Scalability: 3094 MB→3785 MB across 1→20 targets shows nearly stable SAM-MT memory use, while SAM2.1-B+ increases from 3043 MB to 8585 MB.The comparison evaluates VRAM usage under different target counts on the synthetic benchmark.
Scalability: 36.9→35.8 FPS on MOSEv2 for ≥5 targets shows minor degradation, while SAM2.1-B+ falls from 32.1 to 11.5 FPS under the same setting.Cutie shows a similar degradation trend, whereas DeAOT remains relatively stable but has lower overall FPS.
Qualitative Robustness: SAM-MT preserves precise, consistent segmentation and distinct identities in dense scenes, including circus performances, duck flocks, and crowded fitness groups.In identity-ambiguous scenes, decoupled masked attention helps resolve visually similar targets where SAM2.1-B+ later experiences tracking loss or identity drift.
Ablations: Unrestricted query interaction drops J & ˙F by 5.5 points from 43.0 to 37.5, while isolating global context causes a 3.7-point decrease, validating decoupled masked attention.Removing the identity-aware mask causes a 3.9-point drop in J & ˙F from cross-target memory pollution; three identity-transformer blocks balance speed and accuracy.

6 Conclusion

SAM-MT tackles the SAM family’s multi-target bottleneck for real-time interactive video segmentation. It uses lightweight target queries alongside shared global context, maintaining SAM2’s strong segmentation performance while achieving near-single-object efficiency and robust real-time performance.

6 Conclusion: SAM-MT tackles the SAM family’s multi-target bottleneck for real-time interactive video segmentation.It is presented as the first framework addressing this bottleneck.
6 Conclusion: Lightweight queries represent individual targets in parallel with shared global context, enabling near-single-object efficiency as target count increases.This design supports efficiency that remains close to single-object processing as the number of targets grows.
6 Conclusion: SAM-MT maintains SAM2’s strong segmentation performance across VOS benchmarks while achieving real-time speed and robust performance in dense in-the-wild scenarios.The conclusion attributes these results to extensive experiments.