Source-linked AI summary
AuralSAM2: Enabling SAM2 Hear Through Pyramid Audio-Visual Feature Prompting
Yuyuan Liu, Yuanhong Chen, Chong Wang, Junlin Han, Junde Wu, Can Peng, Jingkun Chen, Yu Tian, Gustavo Carneiro
TL;DR
Audio guidance is underexplored in SAM2, while existing integrations can compromise promptability and efficiency. AuralSAM2 uses pyramid-based sparse and dense audio-visual prompts plus audio-guided contrastive learning, improving benchmark accuracy with minimal impact on interactive efficiency.
Problem
SAM2 lacks integrated audio guidance, leaving human-in-the-loop users to search manually for sounding targets despite audio’s potential to guide segmentation.
Method
AuralSAM2 attaches AuralFuser to frozen SAM2 to generate sparse and dense feature-level prompts across a multi-scale pyramid, with AudioCon aligning modalities.
Results
8.25% J&F improvement on AVSBench (V1m), with further gains from the feature pyramid and AudioCon and minimal impact on interactive efficiency.
Takeaways & Limitations
The framework integrates audio into SAM2 while preserving its prompt-driven interface and efficiency for promptable segmentation.
Takeaways & Limitations
Audio remains structurally mismatched with SAM prompting because it lacks spatial anchoring and unfolds on a different temporal scale.
Abstract
from arXiv · showhide
Segment Anything Model 2 (SAM2) exhibits strong generalisation for promptable segmentation in video clips; however, its integration with the audio modality remains underexplored. Existing approaches either convert audio into visual prompts (e.g., boxes) via foundation models, or inject adapters into the image encoder for audio-visual fusion. Yet both directions fall short in human-in-the-loop scenarios due to limited prompt accuracy and increased inference overhead. In particular, these adapter-based methods often suffer from audio prompt dilution, where the signal gradually weakens as it propagates through the network. In this work, we propose AuralSAM2, which integrates audio into SAM2 while largely preserving its promptable segmentation capability. Its core module, AuralFuser, fuses audio and visual features to generate sparse and dense prompts. Guided by audio and built upon SAM2's feature pyramid, these prompts propagate auditory cues across visual layers, reinforcing cross-modal influence. To further align modalities, we introduce an audio-guided contrastive loss that emphasises auditory relevance in dominant visual features. Our method achieves notable accuracy gains on public benchmarks with only minimal impact on the interactive efficiency of promptable segmentation. Our code is available at https://github.com/yyliu01/AuralSAM2.
1. Introduction
SAM2 enables interactive video segmentation through visual prompts, but audio integration remains limited by inaccurate generated prompts, added inference overhead, degraded promptability, and audio prompt dilution. AuralSAM2 addresses these challenges by attaching an AuralFuser to frozen SAM2 to provide audio guidance without modifying image features.
- Background: SAM2 propagates human-provided visual prompts across video frames to segment targets throughout a clip.Its promptable segmentation paradigm supports points and boxes in a human-in-the-loop setting.
- Motivation: Audio cues can guide users toward sounding targets when visual search is slow, error-prone, or hindered by small or ambiguous objects.Audio also helps stabilise tracking under occlusion or among look-alike objects.
- Limitations of Existing Approaches: Foundation-model approaches often generate inaccurate visual prompts and add latency and API-query costs.A box prompt may capture internal patterns instead of the target object.
- Limitations of Existing Approaches: Adapter-based audio-visual fusion alters intermediate visual features, degrades SAM2’s promptable segmentation, and requires repeated SAM2 inferences for prompt engineering.Their ensemble results are nearly 6.5 FPS slower than their AVS results, affecting real-time feedback.
- Audio Prompt Dilution: Audio prompt dilution occurs when audio guidance progressively fades through the model, unlike box prompts that maintain strong cross-attention with pixel features.The phenomenon reflects a structural mismatch between SAM’s prompt design and audio’s available spatial anchoring.
- AuralSAM2: AuralSAM2 attaches AuralFuser externally to frozen SAM2, enabling audio perception without modifying image features or requiring repeated inferences.The module is designed to mitigate audio prompt dilution by enhancing audio-conditioned attention and generating complementary prompts.
2. Related Work
Related work spans vision foundation models and the SAM series, which supports human-in-the-loop segmentation through visual prompts. Audio-visual methods use external models to generate prompts but can suffer limited accuracy and slow inference, motivating AuralFuser’s direct feature-level fusion.
- Vision foundation models use millions of images and self-supervised learning to enhance feature representation.
- The SAM series introduces human-in-the-loop training with self-generated or human-refined visual prompts for static images and video clips.
- AL-Ref and SAM4AVS use large language or vision-language models to extract audio semantics and generate visual prompts zero-shot, but often face limited accuracy and slow inference.
- AuralFuser integrates audio as an external module without altering image-encoder features, avoiding repetitive inference and directly generating two sets of feature-level prompts through cross-modal fusion.
3. Method
AuralSAM2 integrates audio, language, and SAM2’s visual feature pyramid through AuralFuser to produce hierarchical sparse and dense prompts. It further uses audio-anchored contrastive learning and trains only selected parameters while preserving SAM2 components.
- AuralFuser: AuralFuser encodes audio with VGGish, text with RoBERTa, and visual features after Q-pooling to construct a three-level feature pyramid.Audio and text features are concatenated, while SAM2 visual features use pyramid resolutions indexed by k ∈ {1, 2, 3}.
- Training Objective: Training updates selected parameters while keeping the text encoder and SAM2 parameters fixed, and follows SAM2’s loss with supervision on the mask having the lowest segmentation loss.The optimized parameters include θ, such as the VGGish parameters θvgg; SAM2 parameters ϕ remain fixed.
- AuralFuser: At each pyramid level, self-attention processes audio-text and visual modalities separately before cross-modal fusion aligns their representations.Patch embedding projects features to a common resolution, and the fusion block combines early and late cross-modal information.
- Hierarchical Prompting: The fused hierarchy yields sparse prompts encoding global audio-language-informed context and dense prompts identifying potential sounding objects pixel by pixel.Sparse prompts select audio features from the combined representation, while dense prompts retain audio-language-enriched visual features.
- Audio-Guided Contrastive Learning: AudioCon uses audio as the anchor, pulling visual embeddings toward relevant audio and repelling visual negatives to counter visual dominance and modality imbalance.Unlike symmetric contrastive objectives, it avoids emphasizing attraction among pixel-level visual embeddings and organizes visual features around auditory cues.
4. Experiment
Experiments on Ref-AVS and AVSBench show that AuralSAM2 improves segmentation through audio-visual fusion, feature-pyramid prompting, and AudioCon while preserving interactive efficiency. Ablations confirm the importance of sparse and dense prompts and demonstrate stronger audio-visual alignment than standard contrastive learning.
- Ablation Studies: Adding audio and language improves J &F by 8.25% on AVSBench V1m and 9.54% on Ref-AVS Seen, followed by feature-pyramid gains of 3.55% and 2.57%.AudioCon adds a further 1.25% and 0.84% on the respective datasets.
- Prompt Ablation: Removing sparse prompts reduces J &F by 8.06%, while removing dense prompts decreases it by 11.61%, showing that both prompt types are essential.The ablation uses the AVSBench V1m subset with the Hiera large backbone.
- Ablation Studies on CL: AudioCon achieves an additional 0.77 J &F improvement over SupCon on AVS, indicating stronger audio-visual alignment.The comparison evaluates contrastive-learning variants on AVSBench V1m.
- Qualitative Results: Qualitative Ref-AVS results show AuralSAM2 correctly segments the flute when prior methods misidentify the piano or fail to capture the thick flute accurately.The reported example uses the expression “the object making a sound by being played by the woman.”
5. Conclusion
AuralSAM2 enables SAM2 to process audio without adapters or external foundation models. Its AuralFuser integrates multimodal features to generate feature-level prompts that guide decoding while preserving SAM2’s efficiency and generalizability in promptable segmentation.
- AuralSAM2 enables SAM2 to process audio without adapters or external foundation models.
- AuralFuser integrates multimodal features and directly generates sparse and dense feature-level prompts.
- The generated prompts guide the decoder without modifying image features, preserving SAM2’s efficiency and generalizability in promptable segmentation.