Source-linked AI summary
Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation
Tianrui Hui, Shaofei Huang, Qisong Han, Yaxiong Wang, Lechao Cheng, Zhedong Zheng, Zhun Zhong, Richang Hong, Meng Wang
TL;DR
OV-AVSS seeks pixel-level segmentation of sound-emitting objects across seen and unseen categories, but class-agnostic foregrounds create heterogeneous positives and unstable sounding patterns. AGCL reformulates the task around category-specific, audio-grounded cost representations using audio injection and distractor mining. On AVSBench-OV, it substantially outperforms prior methods, especially for unseen categories, with the full framework reaching 43.51% harmonic mIoU in the reported ablation.
Problem
OV-AVSS must segment and recognize sound-emitting objects from both seen and unseen categories, while the prior class-agnostic foreground definition groups semantically diverse objects into a heterogeneous positive set.
Method
AGCL reformulates foreground learning as category-specific and grounds visual-text cost representations with audio through AMCG, AGTA, and SDM.
Results
43.51% harmonic mIoU is achieved by the full AGCL framework in the reported distractor-mining ablation, while experiments show large gains over prior methods, especially on unseen categories.
Takeaways & Limitations
Audio-grounded cost learning supports soundingness discovery within categories and distractor discrimination across categories while preserving open-vocabulary recognition.
Abstract
from arXiv · showhide
Open-Vocabulary Audio-Visual Semantic Segmentation (OV-AVSS) aims to perform pixel-level segmentation of sound-emitting objects from an open set of categories. The previous method relies on a class-agnostic foreground definition, which groups semantically diverse objects into a heterogeneous positive set, causing the model to learn unstable sounding patterns and produce unreliable proposals. To address this, we reformulate the objective to be category-specific and propose a novel Acoustically Grounded Cost Learning (AGCL) framework to transform the static, audio-agnostic visual-text priors into dynamic, audio-grounded cost representations. For intra-category soundingness discovery, we devise Audio-Modulated Cost Generation (AMCG) and Audio-Guided Temporal Aggregation (AGTA) modules to enable both frame-level sounding region highlighting and video-level temporal refinement with a low-intrusive audio injection mechanism. For inter-category distractor discrimination, we introduce a Synergistic Distractor Mining (SDM) strategy, which selectively penalizes acoustically and semantically confusing negative categories to learn more discriminative decision boundaries. Extensive experiments on the AVSBench-OV dataset demonstrate that our method significantly outperforms previous state-of-the-art approaches, particularly on unseen categories. Code is available at https://github.com/spyflying/AGCL.
1 Introduction
OV-AVSS must segment and recognize sound-emitting objects across seen and unseen categories. AGCL addresses the instability of class-agnostic foreground learning by using category-specific, audio-grounded cost representations.
- OV-AVSS requires pixel-level segmentation and semantic recognition of sound-emitting objects across seen and unseen categories.
- The previous class-agnostic foreground definition groups semantically diverse sounding objects into a heterogeneous positive set with poor inner consistency.The resulting objective encourages unstable soundingness patterns and unreliable proposals.
- AGCL reformulates foreground learning as category-specific and transforms static, audio-agnostic visual-text priors into dynamic, audio-grounded cost representations.
- Intra-category soundingness discovery: AMCG injects audio into frame-wise visual features before visual-text correlation, highlighting sounding regions while preserving the original feature distributions.
- Intra-category soundingness discovery: AGTA aggregates category-wise cost embeddings using audio-derived temporal similarity to improve video-level temporal consistency of soundingness discovery.Together, AMCG and AGTA provide frame-level highlighting and video-level temporal refinement.
- Inter-category distractor discrimination: SDM selectively penalizes acoustically and semantically confusing distractor categories to sharpen inter-category decision boundaries.
2 Related Work
Related work spans audio-visual segmentation, open-vocabulary semantic segmentation, and cost-volume learning. AGCL extends visual-text cost learning by grounding its representations with audio for OV-AVSS.
- Audio-Visual Segmentation: Audio-visual segmentation evolved from binary sounding-object foreground-background segmentation to semantic segmentation that also identifies object categories.
- Audio-Visual Segmentation: OV-AVSS extends closed-set audio-visual semantic segmentation to open-vocabulary recognition of sound-emitting objects.
- Open-Vocabulary Semantic Segmentation: Open-vocabulary semantic segmentation methods generally use either two-stage proposal-classification pipelines or end-to-end pipelines.Two-stage methods typically extract object proposals and classify them with CLIP.
- Cost Learning: AGCL transforms static, audio-agnostic visual-text priors from plain cost learning into dynamic, audio-grounded cost representations.
- Cost Learning: CAT-Seg constructs and aggregates multi-modal cost volumes between CLIP image and text features for pixel-level vision-language alignment.Cost volumes represent dense matching costs that can be aggregated into fine-grained correspondences.
3 Acoustically Grounded Cost Learning
AGCL converts static visual-text cost priors into audio-grounded representations for category-specific sounding-object segmentation. AMCG and AGTA address intra-category soundingness, while SDM improves discrimination against acoustic and semantic distractors.
- Audio-Modulated Cost Generation: AMCG softly injects audio cues into selected CLIP vision-encoder blocks before visual-text correlation to highlight sounding regions per category.The modulation parameters are initialized to zero, allowing audio injection to take effect gradually while preserving the original visual features initially.
- Multimodal Feature Extraction: AGCL uses CLIP for visual and visual-perspective text features and CLAP for audio and acoustic-perspective category text features.Audio features are temporally aligned to the video, while category text uses separate visual and acoustic templates.
- Audio-Modulated Cost Generation: Audio-aware visual features are compared with CLIP text features to produce category-wise cost maps, which are converted into cost embeddings for subsequent refinement.The resulting cost embeddings retain temporal, spatial, category, and embedding dimensions.
- Multimodal-Guided Cost Refinement: AGTA aggregates category-wise cost embeddings across frames using audio-derived cross-frame attention, improving temporal consistency while reducing overfitting to seen acoustic patterns.Spatial aggregation separately refines object boundaries and suppresses background noise, while cross-category aggregation models category relationships using text guidance.
- Synergistic Distractor Mining: SDM selects acoustic distractors by audio-category similarity and semantic distractors by ground-truth-category similarity, then uses them to sharpen decision boundaries.Acoustic distractors are selected from the highest similarities in the audio-category matrix, while semantic distractors are selected from category similarities based on concatenated visual and acoustic text features.
- Loss Functions: The alignment loss uses an identity-matrix target to bring same-category multimodal features together and separate different-category features.The loss is implemented as a BCE loss between the similarity matrix and its identity target.
4 Experiment
Experiments on AVSBench-OV show that AGCL improves open-vocabulary segmentation across seen and unseen categories, with component studies supporting its audio injection, temporal guidance, and distractor mining designs. Qualitative and out-of-distribution evaluations further indicate stronger dynamic sounding-object localization and generalization.
- Experimental Setup: AVSBench-OV contains 70 categories, with 40 seen and 30 unseen categories, and uses mIoU for evaluation.The dataset has 5,184 training, 1,240 validation, and 1,490 testing videos; unseen-category videos are excluded from training.
- Comparison with State-of-the-art Methods: Using CLIP-ViT-L/14, AGCL reaches 45.59% unseen mIoU and 61.47% seen mIoU.The stronger CLIP backbone provides an additional performance gain for AGCL.
- Ablation Studies: Combining AMCG and AGTA yields an overall gain of nearly +4.5% mIoU over the baseline.AMCG improves sound-aware cost maps, while AGTA further enhances temporal coherence and sounding-region propagation across frames.
- Ablation Studies: Jointly mining acoustically and semantically confusing distractors produces the best harmonic mIoU of 43.51%.Mining either distractor type alone improves unseen-category mIoU but reduces seen-category performance; joint mining improves both.
- Ablation Studies: AMCG outperforms self-attention and cross-attention audio-injection designs on unseen categories, with a +2.64% gain versus +0.96% and +0.21%.Its soft feature rescaling highlights sounding regions while maintaining the original CLIP feature distribution.
- Ablation Studies: Audio-guided temporal aggregation performs best, because audio-based cross-frame similarity indicates sounding continuity for more precise temporal refinement.No-guidance and visual-guided aggregation mainly improve seen categories, while gains on unseen categories remain marginal.
- Further Discussion: In out-of-distribution evaluation on VPO, AGCL surpasses OV-AVSS across both MS and MSMI subsets and achieves the second-best overall performance.AGCL is trained only on AVSBench-OV and even exceeds models trained on VPO under an in-distribution setting.
5 Conclusion
AGCL reformulates OV-AVSS around category-specific audio-visual patterns, combining low-intrusive audio injection with distractor mining. Experiments show substantial improvements over previous methods.
- AGCL transforms static, audio-agnostic visual-text priors into dynamic, audio-grounded cost representations.
- AMCG and AGTA support intra-category soundingness discovery through frame-level highlighting and video-level temporal refinement.
- SDM mines acoustically and semantically confusing negative categories for inter-category discrimination.
- AGCL substantially improves performance over previous methods on the AVSBench-OV dataset.