Source-linked AI summary
SG-Mamba: Sparse Graph-Guided Mamba for Audio-Visual Speech Enhancement
Guo-Ruei Tseng, Hung-Shin Lee, Hsin-Min Wang, Berlin Chen
TL;DR
Lightweight AVSE must balance computational cost with accurate cross-modal alignment, which simple concatenation and dense attention do not jointly provide. SG-Mamba combines sparse graph-guided fusion with Mamba temporal modeling; on LRS3 it improves SI-SDR over lightweight baselines and retains competitive performance in multi-interferer conditions.
Problem
Lightweight AVSE fusion must balance computational efficiency with expressive cross-modal relations, while concatenation lacks structural constraints and dense attention can be costly and unreliable under severe interference.
Source: §I · p. 1Method
SG-Mamba uses a sparse heterogeneous graph with content-adaptive audio-audio and visual-audio attention, local visual-to-audio connections, and Mamba for long-range temporal modeling.
Source: §I · p. 1, B. Heterogeneous Graph Construction · p. 2, C. Hierarchical Graph-Structured… · p. 3Results
13.091 dB SI-SDR was achieved by SG-Mamba under non-speech noise, exceeding RAVEN by 0.443 dB and Mamba by 0.628 dB.
Source: A. Performance on Non-Speech Noise · p. 4Takeaways & Limitations
Across LRS3 and VoxCeleb2, the authors report that explicit structural priors improve robustness and generalizability across diverse noise and multi-interferer conditions without substantially increasing computational cost.
Source: §V · p. 6The current framework relies on a fixed graph topology and assumes uncorrupted visual inputs.
Source: §V · p. 6
Abstract
from arXiv · showhide
Lightweight audio-visual speech enhancement (AVSE) models face a critical trade-off between computational efficiency and cross-modal alignment accuracy. While simple concatenation lacks relational expressiveness, dense cross-attention incurs computational overhead and is prone to unreliable cross-modal correspondence under strong acoustic interference. We propose Sparse Graph-Guided Mamba (SG-Mamba), a lightweight AVSE framework that integrates a sparse heterogeneous graph with a linear-complexity Mamba backbone. The graph explicitly models modality-specific relations through content-adaptive attention and cross-frame audio-visual connections, while Mamba captures long-range temporal context. We further introduce an audio skip connection to preserve spectral detail without sacrificing noise suppression. Evaluated on LRS3, SG-Mamba achieves competitive or superior performance against strong lightweight baselines and reaches 13.091 dB SI-SDR under noise-only condition. It also remains robust in cluttered multi-speaker conditions with a competitive cost of 3.45 G MACs (or 6.90 G FLOPs). Results on VoxCeleb2 further suggest that explicit structural priors improve robustness, generalizability, and computational efficiency in lightweight AVSE.
I. INTRODUCTION
Lightweight AVSE must balance computational efficiency with expressive cross-modal relational modeling. SG-Mamba addresses this with sparse graph-guided interactions and Mamba temporal modeling.
I. INTRODUCTION
- Mamba offers linear-complexity sequence modeling with competitive long-range dependency modeling for lightweight AVSE.The introduction identifies it as an attractive backbone for improving temporal sequence modeling while maintaining computational efficiency.Source: §I · p. 1
- Existing AVSE fusion faces a trade-off between computational efficiency and the expressive capacity needed for cross-modal relations.Concatenation-based designs lack explicit structural constraints, while dense global cross-attention can be costly and unreliable under severe acoustic interference.Source: §I · p. 1
- SG-Mamba combines sparse graph-guided cross-modal interaction with a Mamba backbone for lightweight AVSE.The graph models modality-specific interactions and neighboring visual-to-audio connections; Mamba handles long-range temporal modeling.Source: §I · pp. 1–2
- The proposed design explicitly connects neighboring visual frames to corresponding audio nodes to support local alignment before temporal modeling.This is intended to accommodate temporal offsets while limiting erroneous alignments from propagating globally.Source: §I · pp. 1–2
II. METHODOLOGY
SG-Mamba uses a modular pipeline that extracts audio and visual features, embeds them, and passes them into heterogeneous graph construction. Audio features retain phase information for waveform reconstruction.
II. METHODOLOGY
- SG-Mamba follows four stages: feature extraction and node embedding, graph construction, hierarchical graph-structured fusion, and Mamba modeling with speech reconstruction.The framework extends RAVEN with structurally graph-guided fusion and a Mamba temporal backbone.Source: §II · p. 2
A. Feature Extraction and Node Embedding
- Audio magnitude spectrograms pass through five CNN layers and an FCN to produce audio features, while phase is retained for reconstruction.Each CNN layer is followed by BatchNorm and ReLU, and the temporal receptive field is 5.Source: A. Feature Extraction and Node Embedding · p. 2
- A pre-trained visual encoder processes frames, which are temporally upsampled to the audio frame rate and projected by an FCN into lip-reading features.The resulting visual features have shape T × D, matching the audio feature sequence.Source: A. Feature Extraction and Node Embedding · p. 2
- Sinusoidal positional encoding is added to both modalities because graph message passing does not inherently encode temporal order.The resulting modality embeddings serve as inputs to heterogeneous graph construction.Source: A. Feature Extraction and Node Embedding · p. 2
B. Heterogeneous Graph Construction
The heterogeneous graph represents audio and visual frames as separate node sets and uses sparse local connections to model temporal continuity and cross-modal relations. Cross-modal edges run from nearby visual nodes to each audio node.
B. Heterogeneous Graph Construction
- Cross-modal edges connect each audio node to a local visual window Vt±3 through directed visual-to-audio links.The local window spans approximately ±30 ms and is intended to capture immediate phoneme-viseme correlations while limiting computational overhead.Source: B. Heterogeneous Graph Construction · p. 2, Figure 2 · p. 2
- Audio and visual nodes connect to neighboring frames from t−3 to t+3 within their respective modalities.The same temporal neighbor window is used for audio-audio, visual-audio, and visual-visual interactions.Source: B. Heterogeneous Graph Construction · p. 2
- Every node retains a self-loop to preserve its frame-level features during message passing.The graph contains separate audio and visual node sets, each initialized with modality-specific embeddings.Source: B. Heterogeneous Graph Construction · p. 2
- The graph uses two sparse neighbor index tables of shape T × 7 rather than materializing a full 2T × 2T adjacency matrix.One table is shared by audio-audio and visual-visual interactions, and the other handles visual-audio interactions.Source: B. Heterogeneous Graph Construction · p. 2
C. Hierarchical Graph-Structured Feature Fusion
HGFF performs content-adaptive graph-based alignment using distinct operations for audio-audio, visual-audio, and visual-visual relations. Its sparse asymmetric structure constrains cross-modal information flow while retaining dynamic representations.
C. Hierarchical Graph-Structured Feature Fusion
- Dynamic graph attention aggregates audio-audio and visual-audio messages using learned neighbor weights and a temperature parameter.The weights are computed independently for the two interaction types, with a joint nonlinear query-key transformation before softmax normalization.Source: C. Hierarchical Graph-Structured… · p. 3
- Visual-visual messages use GCN mean aggregation, which stabilizes visual anchors without attention overhead.The paper motivates averaging by the high correlation of temporally upsampled lip features.Source: C. Hierarchical Graph-Structured… · p. 3
- HGFF updates audio and visual node features through projected residual connections followed by layer normalization.The message matrices are formed by stacking each interaction type’s aggregated messages across all T frames.Source: C. Hierarchical Graph-Structured… · p. 3
- HGFF stacks four graph-fusion layers in a U-Net backbone, with skip additions preserving lower-level cues and an effective receptive field of ±12 frames.Only the aggregated audio representations are passed to the subsequent Mamba block.Source: C. Hierarchical Graph-Structured… · p. 3
- The sparse asymmetric adjacency routes cross-modal information dynamically and avoids the quadratic O(T^2) complexity of Transformer cross-attention.The design combines graph attention for dynamic routing with lightweight GCN mean pooling for redundant visual features.Source: C. Hierarchical Graph-Structured… · p. 3
D. Mamba Modeling and Speech Reconstruction
The unidirectional Mamba block models residual long-range dependencies in HGFF-aggregated audio features. A skip connection preserves fine spectral detail through mask-based reconstruction, while a composite spectral loss supervises phase consistency and spectral fidelity.
D. Mamba Modeling and Speech Reconstruction
- Mamba selectively scans aggregated audio features to model residual long-range temporal dependencies after HGFF has aggregated local bidirectional context.The unidirectional scan avoids redundant temporal modeling while keeping the backend lightweight.Source: D. Mamba Modeling and Speech… · p. 3
- The discretized parameters Ā_t and B̄_t dynamically gate information flow and are described as improving robustness.They are derived from the base transition A, with Ā_t = exp(∆tA) and B̄_t ≈ ∆tB_t.Source: D. Mamba Modeling and Speech… · p. 3
- A skip connection from F_A is added to the Mamba output before layer normalization, and three FCN layers generate a sigmoid mask for magnitude refinement.The refined magnitude is combined with noisy phase and reconstructed by ISTFT.Source: D. Mamba Modeling and Speech… · p. 3
- The composite spectral loss combines complex-domain and magnitude reconstruction terms to supervise phase consistency and spectral fidelity.Ŝ and S denote the estimated and clean power-compressed spectrograms.Source: D. Mamba Modeling and Speech… · pp. 3–4
A. Dataset and Evaluation
Evaluation uses LRS3 speech and visual data mixed with DNS noise and reverberation, with models also tested zero-shot on VoxCeleb2. The setup standardizes the visual frontend and backend configurations for lightweight comparisons.
A. Dataset and Evaluation
- LRS3-trained models were evaluated on the official VoxCeleb2 test set without fine-tuning, using the same mixing protocol and SNR range.LRS3 data were split by the official pretrain, trainval, and test sets with no speaker overlap; target speech was mixed with LRS3 interferers or DNS noise at uniformly sampled −10 to 10 dB SNR.Source: A. Dataset and Evaluation · p. 4
B. Implementation Setup
- The frozen AVHuBERT and TalkNet visual frontend was identical across methods, so “lightweight” refers to the trainable fusion and enhancement backend.Comparisons used a 4-layer HGFF backbone and unidirectional Mamba, without architecture-specific hyperparameter tuning.Source: B. Implementation Setup · p. 4
A. Dataset and Evaluation
- Table I covers performance comparisons in a noise-only scenario with mixed SNRs from −10 to 10 dB.Evaluation metrics were PESQ, ESTOI, and SI-SDR; SI-SDR alone was used in Sections IV-C and IV-D.Source: Table I · p. 4, A. Dataset and Evaluation · p. 4
C. Baseline Configuration
SG-Mamba leads the reported noise-only SI-SDR comparisons while retaining a compact backend cost. The evaluation also compares baseline fusion and backbone configurations and covers one- and three-interferer scenarios.
C. Baseline Configuration
- The baselines compare LSTM and Mamba backbones, bidirectional Mamba, FiLM, linear attention, cross-attention, and graph fusion under shared encoder and hidden-dimension settings.SG-RAVEN retains the heterogeneous graph front-end while replacing Mamba with LSTM.Source: C. Baseline Configuration · p. 4
A. Performance on Non-Speech Noise
- 13.091 dB SI-SDR was SG-Mamba’s highest noise-only result, exceeding RAVEN by 0.443 dB and Mamba by 0.628 dB, at 5.3 M parameters and 3.45 G MACs.SG-Mamba’s SI-SDR was 0.234 dB higher than SG-RAVEN’s.Source: A. Performance on Non-Speech Noise · pp. 4–5
- Tables II and III cover performance comparisons across SNR levels with one and three off-screen interferers, respectively.The supplied table captions specify the scenario scope but not the table values.Source: Table II · p. 5, Table III · p. 5
B. Performance on Interfering Speakers
SG-Mamba performs best or competitively across single- and three-interferer conditions, with strong perceptual quality under dense mixtures. Its SI-SDR advantage narrows as interferers increase, and enhancement can reduce SI-SDR at 10 dB relative to unprocessed noisy input.
B. Performance on Interfering Speakers:
- At 10 dB, all methods had lower SI-SDR than the unprocessed noisy input, as reconstruction artifacts could outweigh suppressed interference.Across SNRs and interferer counts, SG-Mamba remained best or competitive.Source: B. Performance on Interfering Speakers · p. 5
1) One Interfering Speaker:
- With one interferer, SG-Mamba achieved the best performance across metrics and SNR levels, including the highest PESQ and ESTOI.At 0 dB, it reached 2.752 dB SI-SDR, 0.576 dB above RAVEN; the SI-SDR margin narrowed at higher SNRs.Source: 1) One Interfering Speaker · p. 5
2) Three Interfering Speakers:
- With three interferers, SG-Mamba achieved the best PESQ and ESTOI at every SNR and competitive SI-SDR, with no performance collapse.Its SI-SDR margins over non-graph baselines narrowed relative to the single-interferer condition; the paper attributes this to fixed local graph connectivity.Source: 2) Three Interfering Speakers · p. 5
C. Ablation Studies
The ablations compare fusion designs and graph settings: sparse graph-guided fusion performs robustly under interference, while a moderate symmetric graph window gives the best reported hyperparameter setting.
1) Impact of Fusion Strategy:
- Under a similar parameter budget, SG-Mamba achieved the best SI-SDR across conditions, while attention-based methods degraded at 0 dB and by approximately 0.2 PESQ across interference conditions.The authors attribute the attention degradation to negative transfer from unconstrained cross-modal fusion; SG-Mamba restricts interactions to a sparse local window.Source: 1) Impact of Fusion Strategy · p. 6, Table IV · p. 6
2) Impact of Graph Hyperparameters:
- The best graph setting was k = 3, δ = 0: increasing k from 1 to 3 improved results, whereas k = 5 and offsets δ = ±2 underperformed.The authors explain that narrow windows miss temporal variation, oversized windows admit irrelevant distant frames, and forced offsets can misalign phonemes.Source: 2) Impact of Graph Hyperparameters · p. 6, Table V · p. 6
D. Cross-Dataset Generalization
Zero-shot transfer to VoxCeleb2 shows promising cross-dataset generalization, although the graph-based model does not lead every interferer-condition average. The conclusion identifies fixed graph topology and uncorrupted visual inputs as current assumptions.
D. Cross-Dataset Generalization:
- SG-Mamba reached 8.222 dB noise-only SI-SDR and the best Overall Avg. of 5.567 dB on VoxCeleb2, while Mamba slightly led Interferer Avg. at 2.940 dB versus 2.911 dB.The transfer evaluation was zero-shot, without fine-tuning.Source: D. Cross-Dataset Generalization · p. 6
V. CONCLUSION:
- The conclusion reports improved robustness and generalizability across noise and multi-interferer conditions without substantially increasing computational cost, but assumes fixed graph topology and uncorrupted visual inputs.Future work includes learnable sparse graph construction, visual degradation and audio-visual misalignment robustness, phase-aware reconstruction, and real-time streaming evaluation.Source: §V · p. 6