Source-linked AI summary
Motion-Aware Reasoning from Speech to Mask Tracks: Runner-up Solution for the MeViS-Audio Track of the 8th LSVOS Challenge 2026
Jinxing Zhou, Suiyi Zhao, Yanghao Zhou, Ruohao Guo
TL;DR
Speech-guided referring video object segmentation must recover mask tracks from spoken motion descriptions while distinguishing linguistic instructions from sound emitted by objects and handling no-target queries. Speech2MaskTrack compiles speech into structured constraints, ranks SAM3.1 trajectories with motion and relation evidence, and applies gated SaSaSa2VA replacement with empty-only GPT-assisted recovery. It achieved an official final score of 0.705662 and second place in the challenge ranking.
Problem
Speech-guided referring video object segmentation must connect speech recognition, motion-centric temporal grounding, mask tracking, and explicit no-target handling under semantic and visual uncertainty.
Method
Speech2MaskTrack compiles ASR transcripts into structured motion constraints, ranks SAM3.1 candidate tracks with complete-trajectory evidence, and uses gated SaSaSa2VA replacement plus empty-only verified recovery.
Results
0.705662 official final score; Speech2MaskTrack achieved second place in the 8th LSVOS Challenge's MeViS-Audio track.
Takeaways & Limitations
The system supports delayed commitment, explicit abstention for likely absent targets, and recovery that cannot overwrite nonempty mask tracks.
Takeaways & Limitations
The method remains vulnerable to ASR or query-parsing errors, missing candidates, brittle presence gating, brief events, incorrect nonempty tracks, and severe early occlusion.
Abstract
from arXiv · showhide
Speech-guided referring video object segmentation aims to recover the mask tracks of objects specified by a spoken motion description. Here, speech carries a linguistic instruction rather than acoustic evidence from a sounding object, so a solution must connect speech recognition, motion-centric temporal grounding, mask tracking, and explicit no-target handling. We introduce Speech2MaskTrack, our approach for the MeViS-Audio track of the 8th LSVOS Challenge. Speech2MaskTrack transcribes the spoken query and compiles it into structured constraints over category, count, direction, interaction role, and temporal phase. SAM3.1 enumerates multiple instance tracks, which TRACE ranks using complete-trajectory motion and relation evidence. A frozen lexical presence gate may suppress the ranked SAM3.1 base prediction. When the gate predicts that a target is present, an available full-expression-conditioned SaSaSa2VA track replaces the SAM3.1 mask. Only outputs that remain empty enter GPT-assisted recovery, which invokes SaSaSa2VA again under query- and mask-level verification. Speech2MaskTrack achieved second place in the official challenge ranking.
1 Introduction
MeViSv2-Audio identifies video objects from spoken motion descriptions rather than target-emitted sound, requiring speech interpretation, temporal motion grounding, and explicit handling of absent targets. Speech2MaskTrack addresses these uncertainties with structured query compilation, trajectory ranking, asymmetric mask selection, and empty-only recovery.
- Task context: MeViSv2-Audio uses spoken motion expressions to identify targets, unlike MOSEv2's first-frame masks and MeViSv2-Text's textual expressions.Its audio is a spoken query, not sound emitted by the target.
- Problem: Speech recognition can corrupt high-value semantic slots, including category, count, direction, negation, temporal order, and interaction role.These errors can propagate into prompt generation and alter which object or trajectory is considered valid.
- Approach: Speech2MaskTrack compiles ASR output into explicit motion constraints while retaining raw transcription evidence and uncertain hypotheses.The representation covers target identity, count, direction, temporal order, and interaction roles.
- Approach: TRACE ranks complete SAM3.1 trajectories using learned compatibility together with camera-compensated, phase-aware, and target–reference evidence.The ranker retains fallback hypotheses rather than committing immediately to one candidate.
- Output policy: A frozen lexical presence gate couples control-positive SaSaSa2VA replacement with GPT-verified recovery restricted to outputs that remain empty.Recovery cannot overwrite a nonempty prediction, preserving an asymmetric abstention policy.
2 Task Formulation
The task predicts a binary mask sequence for each frame from a video and spoken motion query, allowing one or multiple referred objects. It also requires all-zero output for invalid referents and evaluates both target-present segmentation and no-target behavior.
- Prediction: Given video frames V and a spoken motion query A, the system predicts a binary mask m_t for every frame.The mask sequence is M = {m_t} from t = 1 to T, with each mask defined over the frame's spatial dimensions.
- Prediction: The output may contain one or multiple referred objects.
- No-target handling: If the spoken query has no valid referent, the correct prediction is an all-zero mask sequence.This explicitly represents target absence rather than forcing a segmentation.
- Evaluation: For target-present expressions, evaluation reports region similarity J, boundary accuracy F, and their mean J&F.No-target accuracy measures empty predictions for absent queries, while target accuracy measures nonempty predictions for present queries.
- Evaluation: The final challenge metric discourages both forced grounding of absent queries and overly aggressive suppression of valid targets.It combines segmentation quality with no-target and target accuracy.
3 Method
Speech2MaskTrack converts spoken motion queries into structured constraints, ranks complete SAM3.1 trajectories with motion-aware evidence, and applies gated replacement and empty-only recovery.
- Speech Transcription and Structured Query: ASR transcripts are compiled into motion programs containing target and reference entities, spatial constraints, and action-direction phases with interaction roles.Candidate ASR repairs and unresolved ambiguities are retained rather than treated as verified facts.
- Candidate Generation and Propagation: SAM3.1 generates identity-consistent, complete mask-track candidates independently, retaining per-frame masks, geometry, visibility, confidence, and prompt-role metadata.Tracks form the candidate pool for the ranked SAM3.1 base prediction.
- Motion-Aware Candidate Ranking and Base Prediction: TRACE combines learned compatibility with expert motion, direction, temporal-phase, relation, role, visibility, and segmentation-confidence evidence to rank candidate tracks.The fixed coefficient λ balances the learned and expert scoring branches, and count-aware selection can retain multiple tracks for plural queries.
- Presence Gating: A frozen lexical presence gate converts low-score presence cases into provisional empty outputs while retaining ranked tracks for control-positive cases.Control-absent outputs bypass the main SaSa replacement but may later enter recovery.
- Control-Positive Replacement: For control-positive outputs, an available full-expression-conditioned SaSaSa2VA track replaces the SAM3.1 mask; otherwise, the ranked SAM3.1 prediction is retained.The replacement branch runs after SAM3.1 ranking and presence gating, and the two masks are not averaged.
- Empty-Prediction Recovery: Only outputs remaining empty enter GPT-assisted recovery, where query adjudication, SaSaSa2VA regeneration, and mask arbitration may fill the result without overwriting nonempty predictions.Successful GPT arbitration is required for accepting a recovery candidate.
4 Experiments
Speech2MaskTrack was evaluated on the MeViS-Audio challenge ranking and qualitative validation cases spanning motion, role, temporal-order, occlusion, and attribute-based grounding. It ranked second officially, with strong mask and no-target performance relative to the third-ranked entry but lower target-present accuracy.
- Official leaderboard: The StopTheRoll entry, implemented with Speech2MaskTrack, ranked second in the official final ranking.
- Official leaderboard: Relative to the third-ranked entry, J&F improved by 0.0787 and N-acc. by 0.0689, while T-acc. was 0.0964 lower.
- Qualitative results: The rabbit case maintains identity across a brief late leap, with JF = 0.9561.
- Qualitative results: Figure 3 evaluates temporally sparse action, passive-role disambiguation, temporal-order grounding under occlusion, and relative-attribute tracking through interaction.
- Qualitative results: The horse case follows the passive recipient of a tail strike, while the cow and darker-dog cases preserve identity under occlusion and close interaction.
5 Limitations
Speech2MaskTrack remains vulnerable to error propagation, incomplete candidate coverage, brittle presence gating, early occlusion, costly inference, and unquantified component contributions.
- Incorrect ASR or structured-query parsing can misdirect downstream modules, while TRACE cannot recover objects absent from the SAM3.1 candidate pool.
- The lexical presence gate may be brittle to unusual paraphrases or ASR corruption, and storyboard-based recovery may miss brief discriminative events.
- The non-destructive policy cannot repair an incorrect but nonempty track, while severe early occlusion can reduce region overlap despite accurate later tracking.
- Several large models and video passes increase inference latency and implementation complexity, while final component gains remain unquantified without ablations.
6 Conclusion
The paper presents Speech2MaskTrack as a candidate-first framework for speech-guided referring video object segmentation and reports runner-up performance in the MeViS-Audio challenge. It combines structured motion constraints, trajectory ranking, mask replacement, and conservative recovery while exposing sensitivity to early occlusion.
- Speech2MaskTrack converts speech into structured semantic and temporal constraints for referring video object segmentation.
- The framework ranks SAM3.1 trajectories with camera-compensated motion and relation evidence, then replaces control-positive masks with full-expression-conditioned SaSaSa2VA tracks.
- GPT-verified SaSaSa2VA regeneration is invoked only for outputs that remain empty.
- The entry achieved an official final score of 0.705662 and ranked runner-up in the MeViS-Audio track.
- Qualitative cases illustrate late-event grounding, passive-role disambiguation, temporal-order reasoning, and attribute-aware identity preservation, while exposing sensitivity to early occlusion.