Source-linked AI summary
Audio-Visual Segmentation
Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, Yiran Zhong
TL;DR
The paper addresses pixel-level identification of objects producing sound in video frames, extending prior audio-visual understanding beyond rough localization. It constructs AVSBench with single- and multi-source settings and proposes a temporal pixel-wise audio-visual interaction method with a regularization loss. Pretraining on the Single-source subset improves multi-source performance, while the framework can segment unseen categories with a possible performance drop.
Problem
AVS asks whether each image pixel corresponds to the given audio, addressing the lack of pixel-level audio-visual segmentation annotations and prior patch-level or coarser supervision.
Method
The paper builds AVSBench and an encoder-decoder AVS baseline with TPAVI for temporal pixel-wise audio-visual interaction and a regularization loss for audio-visual correlation.
Results
Pretraining on the Single-source subset improves PVT-v2 AVS with TPAVI from MJ 0.540 to 0.573, a 3.3% gain, on the reported multi-source setting.
Takeaways & Limitations
The experiments support using audio signals to guide pixel-wise visual segmentation and connect sounds with object appearance.
Takeaways & Limitations
The framework can segment unseen object categories, though performance may drop for unseen objects; Single-source training annotations also assume one consistent sounding object over time.
Abstract
from arXiv · showhide
We propose to explore a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark (AVSBench), providing pixel-wise annotations for the sounding objects in audible videos. Two settings are studied with this benchmark: 1) semi-supervised audio-visual segmentation with a single sound source and 2) fully-supervised audio-visual segmentation with multiple sound sources. To deal with the AVS problem, we propose a novel method that uses a temporal pixel-wise audio-visual interaction module to inject audio semantics as guidance for the visual segmentation process. We also design a regularization loss to encourage the audio-visual mapping during training. Quantitative and qualitative experiments on the AVSBench compare our approach to several existing methods from related tasks, demonstrating that the proposed method is promising for building a bridge between the audio and pixel-wise visual semantics. Code is available at https://github.com/OpenNLPLab/AVSBench.
1 Introduction
The paper introduces audio-visual segmentation (AVS), which predicts pixel-level masks of sounding objects rather than patch-level locations. It also presents AVSBench, two supervision settings, and a baseline using audio-visual interaction and regularization.
- Unlike AVC, AVEL, AVVP, and SSL, AVS targets pixel-wise object shapes rather than correspondence, event labels, or rough patch-level regions.
- AVS densely predicts whether each pixel corresponds to the given audio, producing masks for sounding objects.
- AVSBench provides pixel-level ground-truth labels and supports semi-supervised single-source and fully-supervised multiple-source segmentation settings.
- The proposed baseline uses a temporal pixel-wise audio-visual interaction module to guide visual segmentation with audio semantics.
- A regularization loss exploits audio-visual signal correlation to further enhance segmentation performance.
- The paper reports experiments comparing the proposed approach with related-task methods in both AVS settings.
2 Related Work
Prior audio-visual datasets and tasks provide event-level, frame-level, or patch-level supervision, but not pixel-level masks for sounding objects. AVSBench is introduced to address this gap with annotations tailored to audio-visual segmentation.
- Sound source localization predicts rough sounding regions using heat maps or class activation maps, without representing the actual object shape.
- The Single-source subset statistics illustrate category coverage, including 311 helicopter video samples.
- Existing AVE and LLP datasets provide frame-level category or event-boundary annotations rather than pixel-level segmentation masks.
- Flickr-SoundNet and VGG-SS provide bounding-box supervision for sound source localization, but boxes can include nonsounding regions and irregular object shapes.
3 The AVSBench
AVSBench is introduced as a pixel-level benchmark with single-source and multi-source subsets, supporting semi-supervised S4 and fully supervised MS3 settings. Its annotations identify sounding objects at sampled video frames, while the task predicts category-agnostic masks from audio and visual cues.
- Dataset construction: AVSBench provides pixel-level annotations for sounding objects in single-source and multi-source videos.The dataset is divided according to the number of sounding objects.
- Annotation: Each 5-second video is divided into five 1-second clips, with a binary mask for sounding pixels at each sampled frame.The last frame of each clip is annotated, and silent or nonsounding objects are excluded.
- Annotation: Single-source training uses one annotated frame per video, whereas multi-source training uses annotations for all five clips because sound sources may change over time.Validation and test splits contain annotations for all clips.
- Two Benchmark Settings: S4 is semi-supervised single-source segmentation, while MS3 is fully supervised multiple-source segmentation.S4 trains with only the first sampled frame labeled; MS3 trains with all five sampled frames labeled.
- Two Benchmark Settings: Both settings predict category-agnostic masks for sounding objects using audio to indicate targets and visual information to refine their shapes.The output is a binary H × W mask for each video clip.
4 A Baseline
The baseline uses a hierarchical encoder-decoder to combine audio and visual features for sounding-object segmentation. ASPP processes visual features across receptive fields, while TPAVI performs temporal pixel-wise audio-visual interaction before decoding the output mask.
- Encoder: The baseline extracts audio and hierarchical visual features independently, then fuses them in an encoder-decoder segmentation pipeline.Audio features come from VGGish, while visual features come from convolutional or transformer backbones.
- Cross-Modal Fusion: ASPP applies parallel filters with different rates to help recognize visual objects across different receptive fields.The processed visual features are passed to TPAVI at each encoder stage.
- Cross-Modal Fusion: TPAVI maps audio semantics onto visual pixels by transforming, duplicating, and comparing the whole-video audio feature with each visual feature map.The interaction uses dot-product similarity and updates each stage's visual features.
- Decoder: The decoder progressively enlarges fused features through four stages and produces the sounding-object mask M.The architecture follows a hierarchical encoder-decoder design.
- Training Objective: The audio-visual regularization loss aligns pooled masked visual features with audio features, but it is disabled for semi-supervised S4 because it does not help.For S4, the balance weight is set to λ = 0.
5 Experimental Results
Experiments on AVSBench compare the AVS framework with methods from sound source localization, video object segmentation, and salient object detection. Results show that pixel-wise audio guidance improves segmentation, especially when sound sources vary or multiple sources are present.
- Comparison with methods from related tasks: The AVS framework is evaluated against state-of-the-art methods from SSL, VOS, and SOD on the S4 and MS3 settings.The experiments use ResNet-50 and PVT-v2 backbones, with ImageNet-pretrained weights.
- Comparison with methods from related tasks: The AVS baseline consistently outperforms VOS and SOD methods across the semi-supervised S4 and fully-supervised MS3 settings.LGVT slightly exceeds the ResNet50 AVS baseline on S4 (MJ: 0.749 vs. 0.728), but AVS exceeds LGVT on MS3 (MJ: 0.479 vs. 0.407).
- Qualitative comparison: AVS produces accurate sounding-object masks and outlines, whereas SSL methods generate blurry or rough localization maps without precise pixel-level shapes.Qualitative examples include overlocalization by LVS and failure to locate the piano by MSSL.
- Qualitative comparison: Audio guidance helps AVS follow changing sounding objects that visual-only VOS and SOD methods miss or mask incorrectly.In examples where a scene changes from a sounding baby to a baby and dog, AVS follows the audio signal and segments the relevant shapes.
- Analysis of the core components: TPAVI improves segmentation over direct audio-visual feature addition by enabling temporal, pixel-wise interactions between sounds and visual pixels.With TPAVI, MJ rises from 0.728 to 0.705? No: the reported comparisons are 0.728 vs. 0.705 with ResNet50 on S4 and 0.531 vs. 0.516 with PVT-v2 on MS3.
- Analysis of the core components: Both LAVM variants improve performance, while multi-stage TPAVI fusion raises MS3 MJ from 0.490 to 0.540.LAVM-AV improves MJ by around 1% and MF by about 2%; applying TPAVI at all four stages yields a 5% MS3 gain.
6 Conclusion
The paper introduces AVS, AVSBench, and two source-specific supervision settings, then proposes a baseline method linking audio with pixel-wise visual semantics.
- AVS generates pixel-level binary masks for sounding objects in audible videos.
- AVSBench is the first pixel-level audio-visual segmentation benchmark with labels for sounding objects.
- The benchmark supports semi-supervised single-source S4 and fully-supervised multi-source MS3 settings.
- The proposed baseline uses temporal pixel-wise audio-visual interactions and a regularization loss to learn audio-visual correlations.
- The authors suggest AVS can support multimodal semantic segmentation that assigns labels to audio and visual segments.