Source-linked AI summary
MLLM-Assisted Audio VOS: A 3rd Place Report for the MeViS-Audio Track, 8th LSVOS Challenge
Liangtao Shi, Jinxia Xie, Xiantao Hu, Ting Liu
TL;DR
Audio-guided video object segmentation requires aligning acoustic cues with visual targets throughout a video. The paper proposes a training-free pipeline combining MLLMs for audio-text and video-text reasoning with SAM-based models for mask generation and verification, achieving competitive MeViS-Audio performance and ranking 3rd in the 8th LSVOS Challenge.
Problem
The MeViS-Audio Track requires models to associate an audio clip with its visual target and maintain that correspondence throughout the video.
Method
A four-stage training-free pipeline converts audio to text, jointly analyzes video and text, performs text-based segmentation and mask tracking, and verifies mask-text consistency.
Results
The framework achieves competitive performance and ranks 3rd in the MeViS-Audio Track of the 8th LSVOS Challenge.
Takeaways & Limitations
The results demonstrate the effectiveness of combining MLLM reasoning with SAM-based segmentation for audio-guided video object segmentation.
Abstract
from arXiv · showhide
In this technical report, we present a training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language Models (MLLMs) with SAM-based segmentation models. We decompose the task into several stages and identify suitable foundation models for each stage. Without introducing additional model training or task-specific fine-tuning, our approach leverages the strong multimodal reasoning capabilities of MLLMs to model text-visual correspondence and employs SAM-based models for accurate object mask generation. The proposed framework demonstrates the effectiveness of leveraging foundation models for audio-guided video segmentation and achieves competitive performance in the MeViS-Audio Track of the 8th LSVOS Challenge.
1 Introduction
The 8th LSVOS Challenge evaluates video object segmentation across complex visual, language-based, and newly introduced audio-guided settings. The MeViS-Audio Track requires models to associate acoustic cues with visual objects and maintain that correspondence over time, motivating a training-free MLLM-assisted framework that places third.
- The 8th LSVOS Challenge covers complex, text-based referring, and audio-guided video object segmentation tracks.
- The MeViS-Audio Track requires localizing and segmenting an object throughout a video from an associated audio clip.
- Audio-guided segmentation adds audio-visual alignment to the challenge of maintaining object correspondence over time.
- The framework converts audio to text, uses an MLLM for visual-textual reasoning and keyframe selection, applies RVOS/VOS models, and verifies mask-text consistency.
- The proposed framework achieves 3rd place in the MeViS-Audio Track of the 8th LSVOS Challenge.
2 Method
The method is a four-stage, training-free pipeline that converts audio into text, refines target descriptions with joint video-text reasoning, generates and tracks masks, and verifies semantic consistency.
- The training-free pipeline comprises audio-to-text conversion, video-text joint analysis, text-based segmentation with mask tracking, and mask-text verification.
- Stage 1: Audio-to-Text Conversion: ASR converts the input audio A into a textual query q, enabling existing text-conditioned models without task-specific training.
- Stage 1: Audio-to-Text Conversion: The MLLM refines the coarse transcription into fine-grained referring prompts when targets, attributes, or spatial-temporal cues are complex.
- Stage 2: Video-Text Joint Analysis: For each target, joint video-text analysis produces a referring prompt and representative keyframe for initializing subsequent tracking.
- Stage 3: Text-based Video Segmentation and Mask-based Object Tracking: MomentSeg predicts coarse masks over the video, while DAM4SAM bidirectionally propagates and refines the keyframe mask.
- Stage 4: Mask-Text Consistency Verification: Gemini verifies semantic consistency between merged predicted masks and transcribed text, discarding inconsistent masks.
3 Experiments
Experiments evaluate the pipeline on the MeViS-Audio test set using region, boundary, accuracy, and final-score metrics. Ablations show that temporal refinement improves mask scores, while semantic verification substantially improves no-target recognition and the final score.
- The evaluation uses region similarity J, boundary accuracy F, N-acc., T-acc., and the final score.
- Ablation Study: 53.09% J &F is obtained by MomentSeg, while adding DAM4SAM raises J to 53.81%, F to 59.63%, and J &F to 56.72%.
- Ablation Study: 66.80% final score is achieved after consistency verification, up from 63.54%.
- Ablation Study: 96.55% N-acc. is achieved after verification, increasing from 41.38% despite decreased pixel-level segmentation metrics.
- Ablation Study: The ablation attributes mask-quality and temporal-consistency improvements to DAM4SAM and robustness improvements to consistency verification.
4 Conclusion
The report presents a training-free framework combining MLLMs with SAM-based models for audio-guided video object segmentation. Its four-stage design demonstrates competitive performance on the MeViS-Audio challenge and ranks third.
- The framework integrates MLLMs with SAM-based segmentation models without task-specific training.
- The approach combines audio understanding, multimodal reasoning, mask propagation, and semantic verification for temporally consistent segmentation.
- The system achieves competitive performance and ranks 3rd in the MeViS-Audio Track of the 8th LSVOS Challenge.