PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio-Visual Segmentation via Depth-Guided Collaborative Modeling

Zhaojin Fu, Yuyang Hong, Qi Yang, Zili Wang, Kun Ding, Shiming Xiang, Bin Fan

arXiv:2608.16285v1cs.CVcs.AI

TL;DR

Existing audio-visual segmentation methods often lack explicit spatial-structure modeling for robust localization. DGCM-AVS jointly models audio, visual, and depth cues, achieving state-of-the-art performance on AVSBench-Object and AVSBench-Semantic.

  • Problem

    Existing audio-visual segmentation methods lack explicit spatial-structure constraints, limiting robust cross-modal correspondence when objects resemble one another or are occluded.

  • Method

    DGCM-AVS jointly models audio, visual, and depth, using depth for object discrimination and as a bridge between audio and visual features.

  • Results

    DGCM-AVS achieves state-of-the-art performance on both the AVSBench-Object and AVSBench-Semantic benchmarks.

  • Takeaways & Limitations

    The results support depth as a promising source of complementary structural information and an effective bridge for cross-modal alignment in AVS.

Abstract

from arXiv · show

Audio-Visual Segmentation (AVS) is a fundamental task in multimodal perception that performs pixel-level segmentation of sounding objects in videos by leveraging both visual and audio cues. It has broad applications in video understanding, human-computer interaction, and autonomous driving. However, most existing AVS methods do not explicitly model geometric cues such as relative distance and occlusion, thereby limiting the robustness of cross-modal alignment. In human perception, spatial structure is naturally integrated with audio-visual evidence to accurately localize sounding objects. Motivated by this, we incorporate estimated depth as a spatial structural cue for AVS and propose DGCM-AVS, a tri-modal framework that jointly models audio, visual, and depth information. Specifically, we design a Depth-Aware Dynamic Modulator to improve the separation of adjacent objects while preserving intra-object feature consistency. Furthermore, we propose Depth-Guided Progressive Fusion, which uses depth as an intermediate bridge to progressively align audio cues with visual features. Compared to state-of-the-art methods, DGCM-AVS achieves relative improvements of 10.2 percent in M_J and 8.7 percent in M_F on the AVSS dataset. We believe our study highlights depth as a promising yet underexplored modality for AVS and may encourage further research in this direction.

I. INTRODUCTION

DGCM-AVS introduces depth as a spatial cue for audio-visual segmentation, addressing ambiguous localization and occlusion through collaborative audio, visual, and depth modeling. Its two main modules use depth to improve object discrimination, structural consistency, and audio-visual alignment.

  • Limitations of Existing AVS: Existing AVS methods emphasize cross-modal semantic alignment or mask-boundary refinement, but semantic cues alone can fail with similar-looking or partially occluded objects.Visual prompting can sharpen boundaries and improve region separation, yet the supplied passage indicates that spatial awareness remains insufficient for accurate localization.
  • Motivation: Explicit scene spatial structure can constrain cross-modal correspondence and improve segmentation stability when candidate objects resemble one another or are partially occluded.Depth ordering and occlusion cues are proposed as spatial information for more robust cross-modal alignment.
  • DGCM-AVS: DGCM-AVS collaboratively models audio, visual, and depth information to distinguish objects at different distances and positions while preserving structural integrity.The framework incorporates depth because it provides a direct cue to scene spatial structure.
  • Depth-Aware Dynamic Modulator: The Depth-Aware Dynamic Modulator combines region-level low-frequency visual representations with depth-residual high-frequency cues to improve inter-object discrimination and intra-object consistency.Neighborhood aggregation supplies stable regional features, while depth residuals emphasize boundary-sensitive structural cues.
  • Depth-Guided Progressive Fusion: Depth-Guided Progressive Fusion uses depth as a bridge for gradual interaction across audio, visual, and depth cues, improving audio-visual feature alignment.The module is designed for progressive cross-modal interaction rather than direct audio-visual alignment alone.

II. RELATED WORK … A. Overall Architecture

The paper reviews audio-visual segmentation and depth estimation before presenting DGCM-AVS, which integrates video, depth, and audio through dynamic modulation, progressive fusion, multi-scale interaction, and transformer decoding. Its architecture uses depth to improve object separation, detail modeling, and audio-visual alignment.

  • A. Audio-Visual Segmentation: AVS segments sounding objects pixel-wise by jointly leveraging audio and visual cues in complex scenes.AVSBench introduced Object and Semantic benchmark datasets and a baseline that embeds audio features into visual representations.
  • A. Audio-Visual Segmentation: Semantic-alignment methods use audio as queries to retrieve visual targets and enhance cross-modal representations.ECMVAE treats AVS as conditional multimodal variational autoencoding, separating shared and modality-specific latent codes.
  • B. Depth estimation: Depth estimation provides pixel-wise scene geometry, with recent methods emphasizing explicit geometric constraints or large-scale pretraining for transferable representations.These approaches provide cues about relative layout and depth gradients.
  • B. Depth estimation: Depth maps serve as geometric priors that remain useful under low illumination, blur, and occlusion, where RGB features are more affected by appearance changes.Prior work also uses depth-aware fusion, multi-stage injection, decoding, and cross-attention to improve structure and overlapped-scene boundaries.
  • A. Overall Architecture: The architecture combines multi-scale deformable attention, two-stage depth-guided progressive fusion, and a Transformer decoder to model details, align modalities, and reconstruct masks.The complete pipeline runs from multimodal feature extraction to semantic segmentation.
  • B. Depth estimation: Depth offers stable layout, scale, boundary, and occlusion cues that can constrain cross-modal representation learning and improve alignment.These cues are described as more robust than RGB texture and illumination.
  • A. Overall Architecture: DGCM-AVS encodes video, depth, and audio, then injects depth-derived structure into visual features through low-/high-frequency dynamic fusion.The Depth-Aware Dynamic Modulator enhances adjacent-object discrimination and inter-object separability.

B. Perceptual Element Encoding

DGCM-AVS encodes visual, estimated depth, and audio inputs as multi-scale or frame-level perceptual features. Visual and depth streams use selectable backbones, while audio is converted to Mel-spectrograms and embedded with pretrained VGGish.

  • Visual Encoding: The method extracts visual features from the input video with a visual encoder.The video input is denoted xv ∈ R^T ×3×H×W.
  • Depth Encoding: Estimated depth from Depth Anything V2 is converted to RGB format and processed by a separate depth encoder.This accommodates benchmarks that lack depth information and matches the visual input format.
  • Audio Encoding: The audio waveform is resampled to 16 kHz mono, converted into a 96×64 Mel-spectrogram, and encoded frame-wise by pretrained VGGish.The resulting audio features have dimension D = 128 and align with the T video frames.

C. Depth-Aware Dynamic Modulator

The Depth-Aware Dynamic Modulator separates visual and depth features into low- and high-frequency paths, then fuses them to combine stable region semantics with boundary-sensitive geometric cues. Its fused representation preserves sounding-object regions, clarifies contours, and suppresses background interference.

  • Modulator design: DADM uses scale-based spatial aggregation to separate low- and high-frequency components and explicitly model depth-guided local structure.Neighborhood aggregation acts as a low-pass operator, while high-frequency residuals capture rapid spatial variations.
  • Modulator design: Channel attention weights visual and depth features before dynamic perception kernels process their high- and low-frequency paths.The kernels are constructed from input features, enabling input-dependent feature perception.
  • Frequency branches: The low-frequency branch provides smooth, semantically consistent region responses, whereas the high-frequency depth residual highlights object contours and local structural changes.Low-frequency features can blur boundaries or over-expand regions, while depth residuals provide geometric cues for boundary refinement.
  • Fused representation: The fused feature preserves sounding-object integrity, enhances contour clarity, and suppresses background interference, producing predictions that better match ground truth along boundaries.The improvement is visualized on the MS3 and AVSS benchmarks.

D. Depth-Guided Progressive Fusion

Depth-Guided Progressive Fusion uses depth as a bridge to progressively align audio with sounding objects. Its Audio Depth Symbiotic and Target Search stages enhance candidate regions and refine multimodal correspondence for localization and segmentation.

  • Overview: DGPF uses depth as a bridge to progressively establish semantic correspondence between audio and sounding objects.The framework comprises Audio Depth Symbiotic and Target Search stages.
  • Audio Depth Symbiotic: Audio Depth Symbiotic projects audio features into the spatial domain to highlight candidate sounding regions.Depth features provide the spatial reference for this projection.
  • Audio Depth Symbiotic: The Feature Calibrator combines multi-scale boundary textures with globally pooled spatial layouts to produce more discriminative target-related queries.Query-guided weighting suppresses irrelevant noise while preserving stable structure for alignment.
  • Target Search: Target Search uses depth as a bridge to jointly leverage audio, visual, and depth semantics for precise localization and segmentation.It refines correspondence after the first stage enhances candidate regions.

IV. EXPERIMENTS · A. Dataset

The experiments use AVSBench-Object and its semantic extension, AVSBench-Semantic (AVSS), covering single-source, multi-source, and semantically annotated video segmentation settings. These datasets combine semi-supervised and fully annotated subsets with standardized splits and expanded semantic labels.

  • A. Dataset: AVSBench-Object comprises Single Sound Source Segmentation (S4) and Multiple Sound Source Segmentation (MS3) subsets.S4 contains single sounding objects, while MS3 contains multiple simultaneous sound sources.
  • A. Dataset: 4,932 five-second video clips make up S4, with each clip containing a single sounding object.Only the first frame of each five-frame sequence is annotated.
  • A. Dataset: S4 uses a 70/15/15 train/val/test split and provides annotations only for the first frame of each sequence.This setup yields a semi-supervised segmentation setting.
  • A. Dataset: MS3 contains 424 fully annotated videos with multiple simultaneous sound sources, representing a more complex auditory setting.Unlike S4, the MS3 videos are fully annotated.
  • A. Dataset: AVSBench-Semantic (AVSS) extends AVSBench-Object to audio-visual semantic segmentation while retaining the same train/val/test split as S4.It includes all AVSBench-Object samples and adds semantic labels.
  • A. Dataset: 11,356 videos comprise AVSS after adding new 10-second videos with frame-level semantic masks.The extension increases task complexity and makes semantic modeling and generalization more challenging.

B. Implementation Details

Experiments use fixed hardware, resolution, optimizer settings, and iteration budgets across S4, AVSS, and MS3. Evaluation follows standard AVS protocols with Jaccard and F-score metrics for region overlap and segmentation quality.

  • Training settings: Experiments run on an NVIDIA RTX 4090 at 224 × 224 resolution using Adam, 10−4 learning rate, 0.05 weight decay, and batch size 6.Training lasts 90K iterations on S4 and AVSS, and 20K on MS3.
  • Training settings: 90K iterations are used for S4 and AVSS, while MS3 uses 20K iterations.
  • Metrics: Evaluation uses the Jaccard index (MJ) and F-score (MF), with MJ measuring region-level overlap and MF combining Precision and Recall.MJ reflects foreground shape and coverage, while MF emphasizes segmentation-boundary quality and sensitivity to false positives and false negatives.

C. Main Results

DGCM-AVS consistently outperforms prior methods across S4, MS3, and AVSS, with especially strong gains in complex scenes involving multiple sources, occlusion, and clutter. Qualitative results show improved structural integrity and boundary accuracy, while failure cases reveal limited robustness to similar sources and rapid temporal changes.

  • Quantitative Analysis: DGCM-AVS consistently surpasses prior methods across S4, MS3, and AVSS, with particularly notable improvements on the more complex MS3 and AVSS datasets.The gains on S4 are relatively small because its scenes are simpler and contain less interference.
  • Quantitative Analysis: Depth cues are especially beneficial in challenging scenarios involving multiple sound sources, occlusion, or cluttered backgrounds.The results support depth as an effective bridge for cross-modal alignment and as complementary structural information beyond audio-visual cues alone.
  • Qualitative Analysis: DGCM-AVS preserves object integrity and sharpens boundaries under color similarity, texture confusion, partial occlusion, and high-frequency visual patterns.On MS3, it identifies all sounding objects despite semantically similar distractors, while competing methods include silent regions or miss targets.
  • Efficiency and Performance Comparison: 83.3 MJ and 91.3 MF: DGCM-AVS achieves the best S4 performance under a fair comparison using VGGish and ResNet-50 backbones.Its depth-guided collaborative modeling stabilizes cross-modal correspondence under candidate confusion, occlusions, and cluttered backgrounds.
  • Failure Case Analysis: DGCM-AVS can wrongly localize similar sound sources and remains insufficiently robust when sources move out of view or sounding states change rapidly.The paper attributes these failures partly to rare complex patterns in current benchmarks and strong temporal dependence in continuous aligned audio-video inputs.

D. Ablation Study

The ablation study validates DGCM-AVS’s depth-aware modulation and progressive fusion designs, showing that local-window choices, depth bridging, and depth quality affect boundary clarity, focus stability, and performance. It also finds that stronger depth estimates require the proposed framework to exploit their spatial cues effectively.

  • DADM and Backbone Weights: DGCM-AVS uses independent encoders for visual and depth modalities, while removing depth makes both DADM branches rely solely on visual features.The ablation definitions distinguish independent modality encoders from the w/o depth variant.
  • DADM Window Size: The adopted DADM windows (kl = 5, kh = 3) achieve the best performance while preserving stable region responses and clear boundaries.A smaller kh = 3 improves boundary sensitivity, whereas larger kl = 7 can over-smooth details and mix neighborhoods.
  • DGPF Structures: Removing depth as the intermediate bridge, audio, or the Feature Calibrator weakens the progressive fusion design’s ability to align cues and focus on target regions.The only AV variant bypasses depth, w/o audio retains direct visual-depth fusion, and w/o FC reduces focus on potential target regions.
  • DGPF Structures: The full DGCM-AVS model focuses more stably on sounding objects and produces predictions with ground-truth-consistent coverage and clearer contours than the ablated fusion variant.Without structural constraints, attention can drift near sounding objects, causing coarse boundaries and unstable object recognition.
  • Depth Information: DA-V2 produces more stable and accurate depth than DP, especially in complex environments, but high-quality depth still requires DGCM-AVS to realize its full potential.The passage states that improved depth can further improve performance only when the framework effectively exploits the spatial cues.

V. CONCLUSION

The paper proposes DGCM-AVS, a depth-guided AVS framework that uses depth as a bridging cue for audio-visual alignment. Its Depth-Aware Dynamic Modulator combines depth boundary cues with visual semantic cues to improve feature consistency and object separation.

  • Conclusion: DGCM-AVS is proposed as a depth-guided framework for audio-visual segmentation.The authors describe it as the first AVS framework to explicitly use depth as a bridging cue between audio and visual modalities.
  • Conclusion: The framework explicitly uses depth as a bridging cue to guide audio-visual alignment.This bridges the audio and visual modalities through depth information.
  • Conclusion: The Depth-Aware Dynamic Modulator combines boundary-sensitive high-frequency depth cues with semantically informative low-frequency visual cues.This design improves intra-object feature consistency while enhancing inter-object separation.

VI. FUTURE WORK

Future work should address current failure modes through both expanded data and improved temporal modeling. Key directions include training on complex audio-visual scenes and separating synchronized current-frame cues from residual signals.

  • Data: More training videos with complex audio-visual scenes could improve generalization to long-tail cases.This addresses failure modes from the data perspective.
  • Methodology: Explicit temporal decoupling could help models focus on sound cues synchronized with the current frame.This is proposed as a methodological direction for addressing observed failure modes.
  • Methodology: Contrastive learning could suppress residual signals from previous frames while emphasizing current-frame sound cues.The passage identifies this as another methodological strategy for improving temporal focus.
Loading 2608.16285v1…