Audio-Visual Segmentation

• Unlike AVC, AVEL, AVVP, and SSL, AVS targets pixel-wise object shapes rather than correspondence, event labels, or rough patch-level regions.Source: §1 · p. 2
• AVS densely predicts whether each pixel corresponds to the given audio, producing masks for sounding objects.Source: §1 · p. 2
• AVSBench provides pixel-level ground-truth labels and supports semi-supervised single-source and fully-supervised multiple-source segmentation settings.Source: §1 · p. 2
• The proposed baseline uses a temporal pixel-wise audio-visual interaction module to guide visual segmentation with audio semantics.Source: §1 · p. 3
• A regularization loss exploits audio-visual signal correlation to further enhance segmentation performance.Source: §1 · p. 3
• The paper reports experiments comparing the proposed approach with related-task methods in both AVS settings.Source: §1 · p. 3

• Sound source localization predicts rough sounding regions using heat maps or class activation maps, without representing the actual object shape.Source: §2 · p. 3
• The Single-source subset statistics illustrate category coverage, including 311 helicopter video samples.Source: Figure 2 · p. 4
• Existing AVE and LLP datasets provide frame-level category or event-boundary annotations rather than pixel-level segmentation masks.Source: §2 · p. 4
• Flickr-SoundNet and VGG-SS provide bounding-box supervision for sound source localization, but boxes can include nonsounding regions and irregular object shapes.Source: §2 · p. 4

Dataset construction

• AVSBench provides pixel-level annotations for sounding objects in single-source and multi-source videos.The dataset is divided according to the number of sounding objects.Source: §3.1 · pp. 4–5

Annotation

• Each 5-second video is divided into five 1-second clips, with a binary mask for sounding pixels at each sampled frame.The last frame of each clip is annotated, and silent or nonsounding objects are excluded.Source: §3.2 · p. 6
• Single-source training uses one annotated frame per video, whereas multi-source training uses annotations for all five clips because sound sources may change over time.Validation and test splits contain annotations for all clips.Source: §3.2 · pp. 6–7, Figure 3 · p. 6

Two Benchmark Settings

• S4 is semi-supervised single-source segmentation, while MS3 is fully supervised multiple-source segmentation.S4 trains with only the first sampled frame labeled; MS3 trains with all five sampled frames labeled.Source: §3.3 · p. 7
• Both settings predict category-agnostic masks for sounding objects using audio to indicate targets and visual information to refine their shapes.The output is a binary H × W mask for each video clip.Source: §3.3 · pp. 7–8

Encoder

• The baseline extracts audio and hierarchical visual features independently, then fuses them in an encoder-decoder segmentation pipeline.Audio features come from VGGish, while visual features come from convolutional or transformer backbones.Source: §4 · p. 8

Cross-Modal Fusion

• ASPP applies parallel filters with different rates to help recognize visual objects across different receptive fields.The processed visual features are passed to TPAVI at each encoder stage.Source: §4 · p. 8
• TPAVI maps audio semantics onto visual pixels by transforming, duplicating, and comparing the whole-video audio feature with each visual feature map.The interaction uses dot-product similarity and updates each stage's visual features.Source: Figure 5 · p. 8, §4 · p. 9

Decoder

• The decoder progressively enlarges fused features through four stages and produces the sounding-object mask M.The architecture follows a hierarchical encoder-decoder design.Source: §4 · p. 9

Training Objective

• The audio-visual regularization loss aligns pooled masked visual features with audio features, but it is disabled for semi-supervised S4 because it does not help.For S4, the balance weight is set to λ = 0.Source: §4 · p. 9

Comparison with methods from related tasks

• The AVS framework is evaluated against state-of-the-art methods from SSL, VOS, and SOD on the S4 and MS3 settings.The experiments use ResNet-50 and PVT-v2 backbones, with ImageNet-pretrained weights.Source: §5.2 · p. 10, §5.1 · p. 10
• The AVS baseline consistently outperforms VOS and SOD methods across the semi-supervised S4 and fully-supervised MS3 settings.LGVT slightly exceeds the ResNet50 AVS baseline on S4 (MJ: 0.749 vs. 0.728), but AVS exceeds LGVT on MS3 (MJ: 0.479 vs. 0.407).Source: §5.2 · p. 11

Qualitative comparison

• AVS produces accurate sounding-object masks and outlines, whereas SSL methods generate blurry or rough localization maps without precise pixel-level shapes.Qualitative examples include overlocalization by LVS and failure to locate the piano by MSSL.Source: Figure 6 · p. 11, §5.2 · p. 12
• Audio guidance helps AVS follow changing sounding objects that visual-only VOS and SOD methods miss or mask incorrectly.In examples where a scene changes from a sounding baby to a baby and dog, AVS follows the audio signal and segments the relevant shapes.Source: §5.2 · pp. 12–13, Figure 7 · p. 12

Analysis of the core components

• TPAVI improves segmentation over direct audio-visual feature addition by enabling temporal, pixel-wise interactions between sounds and visual pixels.With TPAVI, MJ rises from 0.728 to 0.705? No: the reported comparisons are 0.728 vs. 0.705 with ResNet50 on S4 and 0.531 vs. 0.516 with PVT-v2 on MS3.Source: §5.3 · p. 14
• Both LAVM variants improve performance, while multi-stage TPAVI fusion raises MS3 MJ from 0.490 to 0.540.LAVM-AV improves MJ by around 1% and MF by about 2%; applying TPAVI at all four stages yields a 5% MS3 gain.Source: §5.3 · pp. 15–16

• AVS generates pixel-level binary masks for sounding objects in audible videos.Source: §6 · p. 19
• AVSBench is the first pixel-level audio-visual segmentation benchmark with labels for sounding objects.Source: §6 · p. 19
• The benchmark supports semi-supervised single-source S4 and fully-supervised multi-source MS3 settings.Source: §6 · pp. 19–20
• The proposed baseline uses temporal pixel-wise audio-visual interactions and a regularization loss to learn audio-visual correlations.Source: §6 · p. 20
• The authors suggest AVS can support multimodal semantic segmentation that assigns labels to audio and visual segments.Source: §6 · p. 20