Source-linked AI summary
MLLM-Assisted Audio VOS: A 3rd Place Report for the MeViS-Audio Track, 8th LSVOS Challenge
Liangtao Shi, Jinxia Xie, Xiantao Hu, Ting Liu
TL;DR
Audio-guided video object segmentation requires aligning acoustic cues with visual targets throughout a video. The paper proposes a training-free pipeline combining MLLMs for audio-text and video-text reasoning with SAM-based models for mask generation and verification, achieving competitive MeViS-Audio performance and ranking 3rd in the 8th LSVOS Challenge.
Problem
The MeViS-Audio Track requires models to associate an audio clip with its visual target and maintain that correspondence throughout the video.
Source: §1 · p. 2Method
A four-stage training-free pipeline converts audio to text, jointly analyzes video and text, performs text-based segmentation and mask tracking, and verifies mask-text consistency.
Source: §2 · p. 2, §2.1 · p. 3, §2.4 · p. 4Results
The framework achieves competitive performance and ranks 3rd in the MeViS-Audio Track of the 8th LSVOS Challenge.
Source: §4 · p. 5Takeaways & Limitations
The results demonstrate the effectiveness of combining MLLM reasoning with SAM-based segmentation for audio-guided video object segmentation.
Source: §4 · p. 5, §3.2 · p. 5
Abstract
from arXiv · showhide
In this technical report, we present a training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language Models (MLLMs) with SAM-based segmentation models. We decompose the task into several stages and identify suitable foundation models for each stage. Without introducing additional model training or task-specific fine-tuning, our approach leverages the strong multimodal reasoning capabilities of MLLMs to model text-visual correspondence and employs SAM-based models for accurate object mask generation. The proposed framework demonstrates the effectiveness of leveraging foundation models for audio-guided video segmentation and achieves competitive performance in the MeViS-Audio Track of the 8th LSVOS Challenge.
1 Introduction
The 8th LSVOS Challenge evaluates video object segmentation across complex visual, language-based, and newly introduced audio-guided settings. The MeViS-Audio Track requires models to associate acoustic cues with visual objects and maintain that correspondence over time, motivating a training-free MLLM-assisted framework that places third.
- The 8th LSVOS Challenge covers complex, text-based referring, and audio-guided video object segmentation tracks.Source: §1 · p. 1
- The MeViS-Audio Track requires localizing and segmenting an object throughout a video from an associated audio clip.Source: §1 · p. 2
- Audio-guided segmentation adds audio-visual alignment to the challenge of maintaining object correspondence over time.Source: §1 · p. 2
- The framework converts audio to text, uses an MLLM for visual-textual reasoning and keyframe selection, applies RVOS/VOS models, and verifies mask-text consistency.Source: §1 · p. 2
- The proposed framework achieves 3rd place in the MeViS-Audio Track of the 8th LSVOS Challenge.Source: §1 · p. 2
2 Method
The method is a four-stage, training-free pipeline that converts audio into text, refines target descriptions with joint video-text reasoning, generates and tracks masks, and verifies semantic consistency.
- The training-free pipeline comprises audio-to-text conversion, video-text joint analysis, text-based segmentation with mask tracking, and mask-text verification.Source: §2 · p. 2
Stage 1: Audio-to-Text Conversion
- ASR converts the input audio A into a textual query q, enabling existing text-conditioned models without task-specific training.Source: §2.1 · p. 3
- The MLLM refines the coarse transcription into fine-grained referring prompts when targets, attributes, or spatial-temporal cues are complex.Source: §2.1 · p. 3
Stage 2: Video-Text Joint Analysis
- For each target, joint video-text analysis produces a referring prompt and representative keyframe for initializing subsequent tracking.Source: §2.2 · p. 3
Stage 3: Text-based Video Segmentation and Mask-based Object Tracking
- MomentSeg predicts coarse masks over the video, while DAM4SAM bidirectionally propagates and refines the keyframe mask.Source: §2.3 · pp. 3–4
Stage 4: Mask-Text Consistency Verification
- Gemini verifies semantic consistency between merged predicted masks and transcribed text, discarding inconsistent masks.Source: §2.4 · p. 4
3 Experiments
Experiments evaluate the pipeline on the MeViS-Audio test set using region, boundary, accuracy, and final-score metrics. Ablations show that temporal refinement improves mask scores, while semantic verification substantially improves no-target recognition and the final score.
- The evaluation uses region similarity J, boundary accuracy F, N-acc., T-acc., and the final score.Source: §3.1 · p. 4
Ablation Study
- 53.09% J &F is obtained by MomentSeg, while adding DAM4SAM raises J to 53.81%, F to 59.63%, and J &F to 56.72%.Source: §3.2 · pp. 4–5
- 66.80% final score is achieved after consistency verification, up from 63.54%.Source: §3.2 · p. 5
- 96.55% N-acc. is achieved after verification, increasing from 41.38% despite decreased pixel-level segmentation metrics.Source: §3.2 · p. 5
- The ablation attributes mask-quality and temporal-consistency improvements to DAM4SAM and robustness improvements to consistency verification.Source: §3.2 · p. 5
4 Conclusion
The report presents a training-free framework combining MLLMs with SAM-based models for audio-guided video object segmentation. Its four-stage design demonstrates competitive performance on the MeViS-Audio challenge and ranks third.
- The framework integrates MLLMs with SAM-based segmentation models without task-specific training.Source: §4 · p. 5
- The approach combines audio understanding, multimodal reasoning, mask propagation, and semantic verification for temporally consistent segmentation.Source: §4 · p. 5
- The system achieves competitive performance and ranks 3rd in the MeViS-Audio Track of the 8th LSVOS Challenge.Source: §4 · p. 5