Source-linked AI summary
Report of the 8th LSVOS Challenge: Complex and Multimodal Video Object Segmentation
Chang Liu, Henghui Ding, Lingyi Hong, Ning Xu, Linjie Yang, Yuchen Fan, Canyang Wu, Jinrong Zhang, Xusheng He, Ce Bian, Xianjing Han, Jianlong Wu, Mingqi Gao, Sijie Li, Jungong Han, JeongRae Kim, Chaehyun Kim, Changwon Lim, Jungyoon Lee, Gyuil Lim, Doeon Kim, Seong-heum Kim, Pranjal Aggarwal, Sean Welleck, Yiwen Ren, Jianing Liu, Yingxin Wang, Kexin Zhang, Licheng Jiao, Lingling Li, Xu Liu, Jinxing Zhou, Suiyi Zhao, Yanghao Zhou, Ruohao Guo, Liangtao Shi, Jinxia Xie, Xiantao Hu, Ting Liu
TL;DR
Video segmentation remains difficult when targets disappear, reappear, or are confused with similar objects, and referring queries may lack a valid target. This report compares three challenge tracks and the methods of their leading teams. Across tracks, leading solutions combine foundation mask generation and propagation with task-specific reasoning and target-existence verification.
Problem
Video segmentation remains challenging under occlusion, disappearance, small targets, and visual similarity, while referring tasks also require temporal grounding and decisions about target validity.
Source: §1 · pp. 1–2Method
The report describes three challenge tracks and their protocols, reviews the top three teams in each, and compares recurring design components across their methods.
Source: §1 · p. 2Results
Across the tracks, leading systems use foundation models for mask generation and propagation alongside task-specific routing, memory, multimodal reasoning, and explicit target-existence decisions.
Source: §6 · p. 14Takeaways & Limitations
The report identifies unifying semantic reasoning, memory, and propagation while reducing multi-model cost as a future direction.
Source: §6 · p. 14For mmm, when no valid competitor exists, the method reduces to the original target-only readout.
Source: §3.2 · p. 6
Abstract
from arXiv · showhide
This report summarizes the 8th Large-scale Video Object Segmentation (LSVOS) Challenge, held in conjunction with ECCV 2026. The challenge evaluates video segmentation in three complementary settings: complex semi-supervised video object segmentation on MOSEv2, text-guided referring video object segmentation on MeViSv2-Text, and audio-guided referring video object segmentation on MeViSv2-Audio. We describe the tasks and evaluation protocols and review the methods of the top three teams in each track. Across the nine leading solutions, foundation segmentation models are combined with target-aware memory, multimodal reasoning, explicit target-existence verification, agentic interaction, and corrective tracking. These systems illustrate a broader transition from single-model mask propagation toward modular pipelines that reason about object identity, query validity, and temporal reliability.
1 Introduction
The report frames video segmentation as a problem of maintaining target identity through difficult visual evidence and of grounding motion descriptions while deciding whether a valid target exists. It introduces three complementary challenge tracks and reviews their protocols, leading solutions, and recurring design components.
1 Introduction
- Complex scenes challenge long-term target propagation when objects are small, occluded, visually similar to distractors, or absent before reappearing [11].MOSE and MOSEv2 benchmark these failure modes.Source: §1 · p. 1
- Referring video segmentation requires grounding motion, temporal order, and interactions, including deciding when a query has no valid target.MeViSv2 supports both text and spoken motion descriptions.Source: §1 · p. 2
- MOSEv2 tests identity preservation under ambiguous evidence, MeViSv2-Text tests motion and temporal grounding, and MeViSv2-Audio adds spoken-input uncertainty.Across tracks, the tasks involve target-existence decisions, mask initialization, and long-term propagation.Source: §1 · p. 2
- The report documents tracks, protocols, official results, and top-three methods, then compares recurring components including foundation models, memory, verification, and corrective re-initialization.It aims to describe which designs remain reliable across the tracks’ uncertainties.Source: §1 · p. 2
2 The 8th LSVOS Challenge
The challenge covers mask-initialized segmentation, text-guided motion-expression segmentation, and audio-guided segmentation, with evaluation protocols tailored to their tasks. The reported leaders top their respective tracks, while FudanSeg is presented as an auxiliary MOSEv2 reference outside the official ranking.
2.1 Challenge Tracks
- MOSEv2 segments first-frame target instances through disappearance, reappearance, occlusion, and same-category distractors; MeViSv2-Text and Audio ground motion expressions in video.The audio track uses spoken expressions, adding transcription uncertainty while retaining target-existence and mask-prediction requirements.Source: §2.1 · p. 2
2.2 Evaluation Protocol
- MOSEv2 uses J & ˙F, while the MeViSv2 tracks use J & F and Final = (J &F + N-acc. + T-acc.)/3 to combine mask quality with target-existence accuracy.MeViSv2 scoring penalizes hallucinating a target for an invalid query or suppressing a valid one.Source: §2.2 · p. 3
2.3 Challenge Results
- FudanSeg reaches 64.71 J & ˙F, exceeding the SAM 3 baseline by 1.99 points, but is excluded from the official ranking.Its disappearance score rises from 78.85 to 84.75, while reappearance falls from 25.32 to 23.65.Source: §2.3 · p. 3
- HITsz-Dragon leads MOSEv2 at 69.82 J & ˙F; SSUPER leads the MeViSv2 text track and AEXBY the audio track, with Final scores of 90.81 and 76.96, respectively.Table 1 identifies MOSEv2 as final test-set results, while Table 2 identifies MeViSv2 as final test-set leaderboard results.Source: §2.3 · p. 3, Table 1 · p. 3, Table 2 · p. 3
3 Top Solutions in the MOSEv2 Track
The three leading MOSEv2 methods adapt or augment SAM 3 to preserve target identity over long videos. They combine selective corrective agents, competitive memory retrieval, or training-free dual-memory fusion.
3.1 1st Place: HITsz-Dragon
- HITsz-Dragon routes tiny targets to corrective tracking and semantically distinctive targets to identity verification, while regular targets use SAM 3 propagation.The routing agent tests normalized target area and semantic distinctiveness; specialized agents intervene selectively.Source: §3.1 · p. 4
3.2 2nd Place: mmm
- mmm adds same-class non-target competitors to SAM 3 memory readout and applies deterministic restoration to recover weak target evidence after disappearance or long occlusion.When no valid competitor exists, readout reduces to the original target-only approach; restoration ranges from 1.0 to 1.5.Source: §3.2 · pp. 5–6
3.3 3rd Place: AISTAT
- AISTAT keeps pretrained SAM 3 frozen and uses short- and long-term memory branches, combining their responses with a sequence-relative schedule.Both banks retain the first-frame reference; short-term memory stores recent observations, while long-term memory samples history at fixed intervals.Source: §3.3 · pp. 6–7
4 Top Solutions in the MeViSv2-Text Track
The leading MeViSv2-Text systems separate semantic target decisions from mask generation and propagation. They use staged verification, agent-operated segmentation software, or event decomposition followed by key-frame localization.
4.1 1st Place: SSUPER
- SSUPER uses four semantic stages to ground the expression, generate and select full-video masklets, and audit target existence before StyleRefiner changes mask geometry.Its existence audit distinguishes temporary invisibility from true absence, while the geometry refiner preserves the presence decision.Source: §4.1 · pp. 7–8
4.2 2nd Place: CUA Generalists
- CUA Generalists lets a multimodal agent select, inspect, and revise object identities through typed software actions, while the software handles masks, tracking, state, and output serialization.The agent can preview propagated masks, correct identities, and submit an empty sequence when no object matches.Source: §4.2 · pp. 9–10
4.3 3rd Place: HITsz-Dragon
- HITsz-Dragon decomposes each motion event into target instances, selects a clear key frame and discriminative description, then localizes and propagates each instance independently.SAM 3 tracking proceeds in both temporal directions, and plural-instance masks are merged frame by frame.Source: §4.3 · pp. 10–11
5 Top Solutions in the MeViSv2-Audio Track
The three leading MeViSv2-Audio entries combine candidate-track generation with query-aware selection, correction, or verification. Their pipelines differ in how they assess target presence and handle uncertain or inconsistent masks.
5.1 1st Place: AEXBY
- AEXBY transcribes speech, builds complementary candidate tracks, selects by label-free temporal agreement, applies explicit direction, count, and plural corrections, then gates masks with a target-presence classifier.Its candidate routes include point-prompted SAM tracks, direct text-conditioned masks, and structured tracks; agreement selects the candidate with the greatest mean overlap against the others, without ground-truth labels or cross-model confidence comparisons.Source: §5.1 · pp. 11–12
5.2 2nd Place: StopTheRoll
- StopTheRoll ranks full-video SAM 3.1 trajectories against structured motion queries, gates target presence, replaces control-positive tracks, and sends only remaining empty outputs to GPT-assisted recovery.Recovery may normalize a query when a chronological storyboard supports repair; GPT checks mask-overlay storyboards but never produces mask pixels, and recovery cannot overwrite a non-empty track.Source: §5.2 · pp. 12–13
5.3 3rd Place: Agent-VOS
- Agent-VOS combines audio transcription and joint video-text analysis to create per-instance descriptions and key frames, initialize bidirectional mask tracking, then verify overlays and empty inconsistent tracks.MLLMs handle speech and video-language understanding while SAM-based models perform dense segmentation and tracking.Source: §5.3 · p. 14
6 Conclusion and Discussion
Across the tracks, foundation models provide mask generation and propagation, while task-specific modules address prompting and reliability. Text- and audio-based systems explicitly reason about query validity and target existence; the report calls for unifying these components while reducing multi-model cost.
6 Conclusion and Discussion
- Across tracks, foundation models such as SAM 3 generate and propagate masks while task-specific modules choose prompts; MOSEv2 methods use routing, memory, and corrective re-initialization to preserve identity.These mechanisms address occlusion, reappearance, and same-category interference.Source: §6 · p. 14
- Text- and audio-based systems explicitly reason about query understanding and target existence, verify tracks, distinguish absence from invisibility, and arbitrate masks.Agentic loops inspect overlays and revise uncertain predictions; future work should unify semantic reasoning, memory, and propagation while reducing multi-model cost.Source: §6 · p. 14