PaperlayerView exact revision on arXiv

Source-linked AI summary

GAANet: Global-guided Asymmetric Attention Network for Audio-Visual Speech Separation

Zhiyuan Zhang, Jingyuan Xu, Yiming Tang, Liu Liu, Dan Guo

arXiv:2610.02752v1cs.SD

TL;DR

AVSS seeks to recover a target speaker’s voice from a mixture using acoustic signals and visual cues, while existing multi-scale approaches often fuse modalities at matched temporal resolutions despite their distinct temporal behavior. GAANet combines asymmetric multi-scale fusion with global-guided attention, and experiments report competitive performance on LRS2 and VoxCeleb2 with a lightweight computational profile.

  • Problem

    AVSS methods commonly fuse audio and visual features at matched temporal resolutions, although audio and visual streams have different temporal behaviors and speech separation also requires utterance-level semantic context.

    Source: §I · p. 1

  • Method

    GAANet builds an audio temporal hierarchy while preserving visual temporal resolution, and uses compact global tokens to guide intra- and inter-modal fusion across scales.

    Source: §I · p. 2

  • Results

    GAANet achieves competitive performance on LRS2 and VoxCeleb2 with a lightweight computational profile.

    Source: §I · p. 2

  • Takeaways & Limitations

    The paper concludes that GAANet offers superior separation performance with a lightweight computational footprint for multimodal speech separation.

    Source: §VI · p. 6

Abstract

from arXiv · show

Multi-scale design is crucial for efficient audio-visual speech separation, yet effectively modeling multi-scale information for audio-visual feature fusion remains challenging. We argue that the limited capacity of existing approaches primarily arises from: 1) treating features from different modalities in the same manner, and 2) overlooking the role of global features. To address these issues, we propose a Global-guided Asymmetric Attention Network (GAANet). Our model introduces two core innovations: first, an asymmetric multi-scale fusion framework that allows audio and visual streams to extract and interact with features at their respective optimal temporal resolutions, removing the need for symmetric temporal downsampling; second, a global-guided attention mechanism that compresses each modality into a compact global token with a temporal dimension of one, which then provides high-level semantic cues to guide both intra- and inter-modal fusion across scales. Experiments on LRS2 and VoxCeleb2 demonstrate that GAANet achieves state-of-the-art performance, reaching 16.5 dB SI-SNRi on LRS2 and 14.0 dB on VoxCeleb2, while maintaining a lightweight computational profile with only 3.3M parameters and 19.8G MACs. These results highlight the strong potential of asymmetric temporal modeling and global guidance for efficient and robust multimodal fusion. The source code is publicly accessible at https://github.com/redizzy/GAANet

I. INTRODUCTION

AVSS uses acoustic signals and visual cues such as lip movements to separate a target speaker, but existing multi-scale methods typically fuse modalities at matched temporal resolutions despite their different temporal behavior. GAANet pairs modality-specific temporal processing with compact global tokens that guide fusion across scales.

Motivation

  • Audio and visual streams have distinct temporal behaviors, but most existing multi-scale AVSS methods fuse them at matched temporal resolutions, [2].Audio contains rapidly varying fine-grained patterns, whereas lip movements evolve more slowly and convey relatively stable semantic cues [2].Source: §I · p. 1

Contributions

  • Global tokens with temporal dimension T=1 summarize each modality into time-invariant semantic descriptors that guide intra-modal refinement and inter-modal fusion across scales.The tokens provide consistent cross-modal context rather than modeling temporal dynamics.Source: §I · pp. 1–2
  • GAANet combines asymmetric multi-scale fusion with global-guided attention to exploit audio-visual temporal asymmetry in AVSS.The audio stream forms a temporal hierarchy while the visual stream preserves its temporal resolution; compact global tokens provide guidance across scales.Source: §I · p. 2
  • GAANet achieves competitive performance on LRS2 and VoxCeleb2 with a lightweight computational profile.Source: §I · p. 2

II. RELATED WORK

AVSS research has progressed from modality-specific encoders to attention-based alignment and multi-scale architectures. GAANet extends this work with modality-specific temporal structures and compact global guidance for cross-scale fusion.

AVSS Research

  • Recent AVSS research explores multi-scale architectures to capture interactions between fast acoustic dynamics and slower visual motions, [2], [10].Other directions include simultaneous multi-speaker separation robust to missing visual cues, diffusion-based denoising with cross-attention, and Transformer-style alignment methods [13].Source: §II · p. 2

Attention-based Fusion

  • Attention-based AVSS systems use dual-path Transformers and variants to model long-range dependencies, refine hierarchical representations, and align audio and visual features –, –.Time-frequency modeling with global attention and early audio-visual fusion is also explored.Source: §II · p. 2

Positioning

  • GAANet combines modality-specific temporal structures with compact global guidance, departing from conventional symmetric temporal hierarchies and standard attention formulations.Source: §II · p. 2

III. PROPOSED MODEL

GAANet encodes mixture audio and target-speaker lip video, fuses their features through a repeated A-V Fusion Block, then refines and decodes the auditory representation. Its multi-scale framework builds an audio temporal hierarchy while preserving visual temporal resolution.

A. Overall Pipeline

  • The A-V Fusion Block is repeated N1 times with shared parameters to perform multi-scale audio-visual fusion and compute global semantic tokens that modulate both modalities.The pipeline then applies a lightweight auditory refinement module N2 times, produces a mask with ReLU, and decodes the masked embedding into the target waveform.Source: A. Overall Pipeline · p. 2

B. Asymmetric Multi-Scale Fusion Framework

  • The asymmetric framework builds modality-specific temporal hierarchies from encoder features and aggregates them into cumulative multi-scale audio and visual representations.Audio features are processed along time to form a hierarchy; the visual pathway maintains fixed temporal resolution without downsampling.Source: B. Asymmetric Multi-Scale Fusion… · pp. 2–3

A. Overall Pipeline

C. Global-Guided Attention Mechanism

The global-guided attention module takes cumulative multi-scale audio and visual features, pools them into compact descriptors, and uses cross-modal conditioning and feed-forward networks to produce modality-specific global tokens.

Global Descriptors

Cross-Modal Conditioning

Global Tokens

  • Modality-specific feed-forward networks produce the global audio and visual tokens; each network uses three convolutional layers followed by global layer normalization.The networks provide nonlinear transformation and stabilization for the global representations.Source: C. Global-Guided Attention Mechanism · p. 3

D. Multi-Scale Fusion and Reconstruction

The fusion block uses modality-specific global tokens to guide features at every scale, then consolidates the refined audio and visual representations for the next iteration.

D. Multi-Scale Fusion and Reconstruction

  • The global-guided attention mechanism produces modality-specific global tokens from cumulative multi-scale features, which inject global semantic priors into all fusion stages.The inputs are cumulative representations SA and SV; the outputs are GA and GV.Source: D. Multi-Scale Fusion and Reconstruction · p. 3
  • At each scale, corresponding global tokens modulate audio and visual features through a gated residual formulation.The modulation enriches each scale with global semantics and makes fused representations more consistent across modalities.Source: D. Multi-Scale Fusion and Reconstruction · p. 3
  • The refined multi-scale audio features are consolidated at the finest resolution for the next fusion iteration.The visual pathway similarly consolidates its refined features into its finest-resolution representation for the next iteration.Source: D. Multi-Scale Fusion and Reconstruction · p. 3

IV. EXPERIMENT CONFIGURATIONS

Experiments follow prior work on LRS2 and VoxCeleb2, using prepared two-speaker audio mixtures with synchronized lip frames. The accompanying figure describes an iteration of the A-V Fusion Block.

IV. EXPERIMENT CONFIGURATIONS

  • Each fusion-block iteration processes multi-scale audio and visual features, generates modality-specific global tokens, modulates features across scales, and consolidates them for the next iteration.The figure illustrates D = 3.Source: Figure 3 · p. 4
  • Audio segments are 2 seconds at 16 kHz, mixed from two speakers at SNRs from -5 dB to 5 dB, with synchronized 88×88 grayscale lip frames at 25 FPS.The experiments use the standard dataset split.Source: A. Datasets · pp. 3–4

B. Model configurations

The model configuration specifies shared fusion-block iterations and auditory refinement alongside training choices. Evaluation uses separation-improvement metrics and PESQ to assess separated-speech quality.

B. Model configurations

  • Training uses Adam with initial learning rate 0.001, SI-SNR objective, learning-rate reductions after 20 unimproved validation epochs, early stopping after 30 epochs, and gradient clipping at L2-norm 5.Source: B. Model configurations · p. 4
  • GAANet uses a depth-4 hierarchy, four shared A-V fusion-block iterations, and 12 auditory-refinement iterations.Experiments use batch size 3 on 8 GeForce RTX 4090 GPUs.Source: B. Model configurations · p. 4

C. Evaluation metrics

  • SI-SNRi and SDRi evaluate separation performance, and both measure improvement relative to the original mixture.Source: C. Evaluation metrics · p. 4
  • PESQ assesses the perceived quality and intelligibility of separated speech as an additional evaluation measure.Source: C. Evaluation metrics · p. 4

V. RESULTS

GAANet reports the strongest separation results on LRS2 and VoxCeleb2, with a compact model profile. Ablations report gains from GGA and AMF individually and together.

A. Comparisons with State-of-the-Art Methods

  • GAANet contains 3.3M parameters and 19.8G MACs, providing a compact architecture with manageable computational requirements.The passage also describes the model as running efficiently in practice and suitable for deployment.Source: A. Comparisons with State-of-the-Art… · p. 4

B. Ablation Study

  • On LRS2, SI-SNRi is 15.5 dB without GGA or AMF, 16.2 dB with only GGA, 15.9 dB with only AMF, and 16.5 dB with both components.The authors describe GGA and AMF as complementary and jointly enhancing separation performance; GGA contributes a relatively stronger improvement in isolation.Source: B. Ablation Study · pp. 4–5, Table II · p. 5

A. Comparisons with State-of-the-Art Methods

C. Qualitative Analysis

Qualitative comparisons show more target-consistent GAANet spectrograms and speaker-related structure in its global audio tokens; the ablations select N1=4, D=4 as the quality–efficiency setting.

C. Qualitative Analysis

D. Impact of Iteration Number and Multi-Scale Depth

VI. CONCLUSION AND DISCUSSION

GAANet combines asymmetric multi-scale fusion with global semantic guidance. The paper reports superior separation performance with a lightweight computational footprint.

VI. CONCLUSION AND DISCUSSION

  • GAANet uses an asymmetric multi-scale design that respects the distinct temporal characteristics of audio and visual signals.Source: §VI · p. 6
  • A compact global token provides coherent semantic guidance across fusion stages, supporting audiovisual alignment and stable feature integration.Source: §VI · p. 6
  • Experiments demonstrate superior separation performance with a lightweight computational footprint, which the authors describe as practical for real-world multimodal speech separation.Source: §VI · p. 6
Loading 2610.02752v1…