PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and Multi-Scale Global-Local Attention

Kai Li, Kejun Gao, Xiaolin Hu

arXiv:2509.23610v2cs.SDcs.CV

TL;DR

AVSS systems can be computationally expensive, while lightweight visual encoders may struggle to retain semantic alignment. Dolphin combines discrete audio-aligned visual tokens with a single-iteration global-local separator; across three benchmarks it outperformed IIANet on all separation metrics while reducing resource use.

  • Problem

    Many AVSS methods are computationally intensive, and lightweight visual encoders can lose semantic representation capability, leaving efficiency and separation quality difficult to balance.

    Source: §1 · pp. 1–2

  • Method

    Dolphin uses DP-LipCoder to produce discrete audio-aligned visual tokens and a single-iteration encoder-decoder separator with global-local attention.

    Source: §1 · p. 2

  • Results

    Across three public AVSS benchmarks, Dolphin outperformed IIANet on all separation metrics, with over 50% fewer parameters, over 2.4× lower computational cost, and over 6× faster GPU inference.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that Dolphin offers a viable solution for AVSS on edge devices.

    Source: §1 · p. 2

    Dolphin relies on relatively clean, synchronized lip videos; robustness to large head-pose variations, occlusions, and extreme lighting remains future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

Audio-visual speech separation (AVSS) methods leverage visual cues to extract target speech and have demonstrated strong separation quality in noisy acoustic environments. However, these methods usually involve a large number of parameters and require high computational cost, which is unacceptable in many applications where speech separation serves as only a preprocessing step for further speech processing. To address this issue, we propose an efficient AVSS method, named Dolphin. For visual feature extraction, we develop DP-LipCoder, a dual-path lightweight video encoder that transforms lip-motion into discrete audio-aligned semantic tokens. For audio separation, we construct a lightweight encoder-decoder separator, in which each layer incorporates a global-local attention (GLA) block to efficiently capture multi-scale dependencies. Experiments on three benchmark datasets showed that Dolphin not only surpassed the current state-of-the-art (SOTA) model in separation quality but also achieved remarkable improvements in efficiency: over 50% fewer parameters, more than 2.4x reduction in MACs, and over 6x faster GPU inference speed. These results indicate that Dolphin offers a practical and deployable solution for high-performance AVSS in real-world scenarios. Our code and demo page are publicly available at http://cslikai.cn/Dolphin/.

1 INTRODUCTION

Dolphin addresses the joint challenge of computationally expensive AVSS and lightweight visual encoders that can lose semantic alignment. It combines discrete audio-aligned visual tokens with a single-iteration global-local separator, and outperforms IIANet on three benchmarks while reducing resource use.

1 INTRODUCTION

  • AVSS methods often incur high computational cost, and balancing separation quality with computational cost remains a major challenge.Lightweight iterative separators can also incur substantial overhead from their iterative process.Source: §1 · p. 1
  • DP-LipCoder uses vector quantization and AV-HuBERT distillation to map video into discrete semantic tokens highly aligned with audio.Its dual paths produce compressed abstract visual features that preserve spatio-temporal structure and audio-aligned visual features.Source: §1 · p. 2
  • A single-forward-pass separator uses global-local attention at each layer to capture long-range dependencies and smooth local features while preserving details.Coarse-grained self-attention models global dependencies; heat-diffusion attention models local features.Source: §1 · p. 2
  • Over 50% fewer parameters, over 2.4× lower computational cost, and over 6× faster GPU inference; Dolphin also outperformed IIANet on all separation metrics across three public AVSS benchmarks.The authors describe these results as a viable solution for AVSS on edge devices.Source: §1 · p. 2

2 RELATED WORKS

Related work describes computationally costly AVSS systems and lightweight iterative alternatives, then highlights a parallel efficiency–semantic-alignment challenge in video encoders. Dolphin’s approach is framed as addressing both challenges.

2.1 AUDIO-VISUAL SPEECH SEPARATION

  • End-to-end and time-frequency AVSS systems have improved separation quality, but computational expense or recurrent inference latency limits their use in resource-constrained or latency-sensitive settings.Dolphin instead uses a single-iteration encoder-decoder modeling multi-scale global and local features within each layer.Source: §2.1 · p. 2

2.2 PRETRAINED VIDEO ENCODER

  • Large pretrained lip-reading backbones provide semantically aligned visual features at substantial computational cost, while lightweight compression or reconstruction-focused designs can lose semantic capability and separation quality.The section presents this as a performance–efficiency trade-off for video encoders.Source: §2.2 · p. 3
  • Jointly optimizing semantic extraction and video reconstruction is proposed as a way to retain a lightweight encoder while generating higher-semantic-density features for speech separation.Source: §2.2 · p. 3

3 METHODS

Dolphin combines a dual-path visual encoder and audio-visual fusion with a single-iteration encoder-decoder separator. Its global-local attention blocks and multi-scale pathways are designed to model audio features efficiently.

3.1 OVERALL PIPELINE

  • The pipeline encodes video into reconstruction-related and semantic-related features, encodes mixture audio, fuses the streams, separates target-speaker features, and decodes them into a waveform.Figure 1 marks frozen parameters with a snowflake.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 PRETRAINED VIDEO ENCODER

  • DP-LipCoder uses separate reconstruction and semantic encoder paths, a VQ module on the semantic path, and a mirrored decoder to reconstruct video.Reconstruction, AV-HuBERT distillation, and VQ commitment losses train the dual-path network.Source: §3.2 · p. 4

3.3 AUDIO-VISUAL FUSION (AVF) MODULE

  • The AVF module combines video-guided gated fusion and attention-based fusion across visual feature spaces, upsamples visual features temporally, then sums both pathways.Source: §3.3 · p. 5

3.4 SEPARATOR

  • The separator retains one TDANet iteration and uses Q encoder layers with two GLA blocks each, alongside Q decoder layers with TDA and three GLA blocks each.The design jointly models global and local attention to compensate for removing iterative refinement.Source: §3.4 · p. 5, Figure 3 · p. 5

3.4.1 GLA BLOCK

  • Each GLA block pairs global attention using downsampled temporal self-attention with local attention using heat-diffusion filtering and a feed-forward network.The heat-diffusion filter is described as a learnable multi-scale mechanism that avoids a limited convolutional receptive field without adding many parameters.Source: §3.4.1 · p. 6

3.4.2 ENCODER OF SEPARATOR

  • The encoder downsamples across temporal resolutions, sums features at the lowest resolution into a global representation, then uses it to guide local features through TDA blocks.Source: §3.4.2 · p. 7

3.4.3 DECODER OF SEPARATOR

  • The decoder reconstructs target-speaker features through top-down TDA and GLA connections, then outputs them directly rather than masking mixture features.A 1 × 1 convolution, GLU, and another 1 × 1 convolution produce the final representation.Source: §3.4.3 · p. 7

4 EXPERIMENT CONFIGURATIONS

The evaluation uses three established AVSS benchmarks with synchronized lip-video and two-speaker mixture inputs. Training and evaluation settings specify the optimization schedule, separation metrics, and resource-efficiency measurements.

4 EXPERIMENT CONFIGURATIONS

  • Experiments use LRS2, LRS3, and VoxCeleb2, with two-second, 16 kHz mixtures of two speakers and synchronized 25 FPS, 88 × 88 grayscale lip images by default.Source: §4 · p. 7
  • Training uses Adam with an initial learning rate of 1 × 10−3, halves it after 15 plateau epochs, and stops early after more than 30 stalled epochs.Training used eight RTX 5090 GPUs, batch size 48, SI-SNR optimization, and L2 gradient clipping at 5.Source: §4 · p. 7
  • Evaluation reports SDRi, SI-SNRi, and PESQ, plus parameters, MACs, CPU/GPU inference latency, and memory usage measured on one-second audio.Source: §4 · p. 7

5 RESULTS

Dolphin outperforms competing methods across benchmark separation metrics while reducing computational cost. Ablations support the contributions of discrete visual features, global-local attention, and the HDA layer.

5.1 EFFECTIVENESS OF THE PRETRAINED VIDEO ENCODER

  • Compared with the continuous autoencoder, LipCoder and DP-LipCoder improve SI-SNRi by at least 1.0 dB, while DP-LipCoder stays within 0.2 dB of 3D ResNet-18 with 93% fewer parameters and 70% fewer MACs.The comparison is on LRS2; the parameter and MAC figures concern the pretrained video encoder.Source: §5.1 · p. 8, Table 1 · p. 8
  • Replacing existing AVSS models’ video encoders with DP-LipCoder causes a slight performance drop, while yielding substantial efficiency gains.The models were retrained from scratch with the frozen DP-LipCoder; resource-constrained deployment is identified as a relevant setting.Source: §5.1 · p. 8, Table 2 · p. 8

5.2 COMPARISON WITH STATE-OF-THE-ART METHODS

  • Dolphin consistently surpasses competing approaches across all evaluation metrics on three benchmark datasets.Table 3 reports the comparison; baseline scores were taken from original papers or reproduced using official implementations.Source: §5.2 · p. 8, Table 3 · p. 9
  • Dolphin uses 46% of IIANet’s MACs without the pretrained video encoder and reduces its CPU inference time to 66%.It also reduces inference time by 18% versus Swift-Net and 47% versus RTFS-Net; with the video encoder included, it has the lowest MACs among the compared methods.Source: §5.2 · p. 9, Table 4 · p. 9

5.3 ABLATION STUDY

  • Removing either global or local attention from GLA degrades performance, and removing both produces the worst result, supporting their complementary roles.The omitted modules are replaced by a single depthwise convolution in the ablations.Source: §5.3 · p. 9, Table 5 · p. 10
  • The HDA layer improves all three metrics over a large-kernel convolution alternative while reducing learnable parameters.The alternative models local features purely through convolution; HDA incorporates physical priors through heat diffusion.Source: §5.3 · p. 10, Table 6 · p. 10
  • Single-iteration Dolphin achieves higher separation accuracy than iterative refinement at similar computational cost, while a deeper decoder and direct feature regression improve reconstruction quality and mitigate masking distortions.Early-stage audio-visual fusion improves SI-SNRi by 0.6 dB over later fusion.Source: §5.3 · p. 10

6 CONCLUSIONS

Dolphin pairs DP-LipCoder’s discrete audio-aligned visual semantics with a lightweight separator using global–local attention. The authors report a favorable performance–efficiency balance, while noting limits in video robustness, extreme-edge deployment, and fine-grained articulatory cues.

6 CONCLUSIONS

  • Across three benchmark datasets, Dolphin combines discrete audio-aligned DP-LipCoder features with a lightweight encoder–decoder separator and GLA blocks for a favorable performance–efficiency balance.The separator captures global and local features in a single iteration.Source: §6 · p. 10
  • Dolphin still relies on relatively clean, synchronized lip videos, and deployment on extremely resource-constrained edge devices remains challenging.The authors also note that discrete tokens may discard fine-grained articulatory cues.Source: §6 · p. 10

A.1 VIDEO AUTOENCODER ARCHITECTURE

  • The video encoder uses 3D residual blocks, spatial attention, and progressive spatial compression, then mirrors compression with sub-pixel upsampling for reconstruction.Spatial attention models dependencies between positions within each frame; residual units combine 3D convolutions, SE attention, and a residual connection.Source: §A.1 · pp. 14–15

A.2 OBJECTIVE FUNCTION OF THE PRE-TRAINING STAGE

  • Pretraining combines reconstruction, quantization-commitment, and AV-HuBERT semantic-distillation objectives to learn discrete representations while reconstructing the lip video.The distillation head predicts features aligned with the teacher’s outputs.Source: §A.2 · p. 16

A.3.1 DATASETS

  • The unified dataset combines LRS2, LRS3, and VoxCeleb2, using 2.0-second clips at 25 Hz and preserving each corpus’s official data splits.Each clip contains approximately 50 frames.Source: §A.3.1 · p. 16

A.3.2 MODEL CONFIGURATION

  • The visual codec uses grayscale 88 × 88 video, expands channel width from 4 to 32, and configures attention with 8 heads; its default codebook has 256 entries of dimension 64.Codebook entries are initialized with k-means.Source: §A.3.2 · p. 16

A.3.3 TRAINING DETAILS

  • Training uses Adam at 1 × 10^-3 for up to 500 epochs, with early stopping after 20 consecutive epochs without validation-loss improvement.Experiments used four NVIDIA RTX 3090 GPUs and a global batch size of 32.Source: §A.3.3 · p. 17

A.4 RESULTS

DP-LipCoder achieved the lowest reconstruction error among the compared methods, and its visualizations show stronger structural fidelity and detail. The AVF module combines two visual–audio fusion pathways into one output representation.

A.4 RESULTS

  • Reconstruction error fell by about one order of magnitude versus AE and more than two orders versus unaligned VQ.DP-LipCoder converged with lower variance to the smallest error.Source: Figure 7 · p. 17, §A.4 · p. 17
  • DP-LipCoder preserves spatial structure with sharper, more realistic textures; LipCoder’s reconstruction remains suboptimal despite effective separation as a pretrained encoder.The AE produces more coherent shapes than LipCoder but can blur contours or fragment structures.Source: §A.4 · pp. 17–18, Figure 8 · p. 18

B DETAILS OF AVF MODULE

  • AVF combines video-guided gated fusion and multi-space attention fusion by summing their outputs after integrating visual and audio features.The attention pathway partitions visual features into sub-features that guide fusion along different dimensions.Source: B DETAILS OF AVF MODULE · pp. 18–19

C DATASET DETAILS

This section describes the three benchmark datasets and the experimental conventions used to evaluate Dolphin. It also outlines the AVSS training objectives and key model configurations.

C DATASET DETAILS

  • Evaluation uses the established dataset partitions, and the datasets’ speaker-organized directories support speaker-independent evaluation.The experiments use LRS2, LRS3, and VoxCeleb2; preprocessing resamples video to 25 FPS and audio to 16 kHz mono, crops 96 × 96 grayscale lip sequences, and mixes two speakers.Source: C DATASET DETAILS · pp. 19–20
  • LRS2, LRS3, and VoxCeleb2 provide distinct acoustic settings, with approximately 11, 28, and 56 training hours, respectively, and 3 validation and 1.5 test hours each.LRS2 includes outdoor segments and authentic noise; LRS3 recordings are generally cleaner, while VoxCeleb2 features varied noise, reverberation, head poses, and illumination.Source: C DATASET DETAILS · p. 19

D AVSS OBJECTIVE FUNCTION

  • Dolphin trains with time- and frequency-domain SI-SNR objectives, comparing the ground-truth signal with the estimate and its STFT magnitude spectrum.The overall loss uses epoch-dependent weighting; the frequency-domain estimate is reconstructed from the separator’s decoder-layer output before its magnitude spectrum is computed.Source: D AVSS OBJECTIVE FUNCTION · p. 20

E MODEL HYPERPARAMETER CONFIGURATIONS

  • The model combines reconstruction and semantic visual tokens, a 256-dimensional audio representation, and a hierarchical separator with four downsampling layers.The visual tokens are summed into a 3872-dimensional feature; the audio encoder uses a 1D convolution with kernel 16 and stride 4.Source: E MODEL HYPERPARAMETER CONFIGURATIONS · p. 21
  • The visual-feature fusion network uses an 8-head attention module, while the cross-modal AVF module uses a 1 × 1 convolution and four upsampling stages.The fusion network has hidden dimension 64 and feed-forward size 128; AVF has hidden dimension 128 and uses Layer Normalization.Source: E MODEL HYPERPARAMETER CONFIGURATIONS · p. 21

F DETAILS OF DIFFERENT PRETRAINED VIDEO ENCODER

  • The compared pretrained video encoders include a 3D ResNet-18 backbone and a lightweight frame-wise 2D autoencoder.The 3D ResNet-18’s main computational cost comes from its backbone; the autoencoder follows the AVLiT-8 architecture.Source: F DETAILS OF DIFFERENT PRETRAINED… · p. 21

G ABLATION STUDY: EFFECT OF THE VECTOR QUANTIZATION MODULE IN THE DP-LIPCODER

The section reports Dolphin’s multi-speaker comparison with existing baselines and tests the contribution of vector quantization in DP-LipCoder.

G ABLATION STUDY: EFFECT OF THE VECTOR QUANTIZATION MODULE IN THE DP-LIPCODER

  • Removing VQ degrades every reported metric, including SI-SNRi by approximately 0.5 dB, supporting VQ’s role in separation quality.The ablation removes the quantizer and commitment loss while retaining reconstruction and distillation objectives; the visual encoder is frozen during separator training.Source: G ABLATION STUDY: EFFECT OF THE VECTOR… · p. 21, Table 7 · p. 21
  • The authors attribute VQ’s benefits to discrete semantic anchors that improve temporal alignment and stabilize distillation against distribution drift, while regularizing irrelevant variation.They further associate this regularization with improved noise robustness and generalization at negligible computational overhead.Source: G ABLATION STUDY: EFFECT OF THE VECTOR… · pp. 21–22

H MULTI-SPEAKER PERFORMANCE

  • Dolphin consistently outperforms the existing baselines across multi-speaker evaluation datasets and metrics, with its largest improvement over the state of the art on LRS2-4Mix.Baseline results were taken from IIANet; Table 8 reports the comparison.Source: Table 8 · p. 22, H MULTI-SPEAKER PERFORMANCE · p. 22

I ADDITIONAL EXPERIMENTS UNDER DIVERSE ACOUSTIC CONDITIONS

These experiments evaluate Dolphin under added noise and interfering speakers, then assess its output on naturally overlapping speech. The reported comparisons favor Dolphin across both evaluations.

I ADDITIONAL EXPERIMENTS UNDER DIVERSE ACOUSTIC CONDITIONS

  • Four VoxCeleb2 test scenarios add environmental or music noise, either alone or with two interfering speakers; each scenario contains 1,000 samples.Dolphin, IIANet, and AV-Mossformer2 were evaluated in all four scenarios.Source: I ADDITIONAL EXPERIMENTS UNDER DIVERSE… · p. 22
  • Dolphin outperforms IIANet and AV-Mossformer2 on SI-SNRi, SDRi, and PESQ across the four scenarios, including the three-speaker noisy conditions.The authors describe these tests as additional evidence of performance under realistic and more adverse acoustic environments.Source: I ADDITIONAL EXPERIMENTS UNDER DIVERSE… · p. 22, Table 9 · p. 23

J SUBJECTIVE HUMAN EVALUATION ON REAL OVERLAPPING SPEECH

  • On 18 short debate-video clips rated by 50 listeners in a double-blind MOS study, Dolphin receives higher scores across all reported criteria than both baselines.The clips contain natural overlapping speech, background noise, and room reverberation; ratings used a five-point scale.Source: J SUBJECTIVE HUMAN EVALUATION ON REAL… · p. 23

K VISUAL RESULTS FOR SPEECH SEPARATION

The section uses visual examples and ablations to examine separation quality, iteration cost, GLA allocation, and separator output formulation. The reported comparisons favor Dolphin’s visual results and direct mapping, while identifying an encoder-decoder allocation that performs best.

K VISUAL RESULTS FOR SPEECH SEPARATION

  • Across four LRS2 examples, Dolphin most closely matches ground truth, with less noise and more complete harmonic structure than IIANet and AV-Mossformer2.The examples describe cleaner output in the first sample, better preservation of low-frequency detail in the second, fewer residual high-frequency artifacts in the third, and a coherent harmonic pattern in the fourth.Source: Figure 9 · p. 24, K VISUAL RESULTS FOR SPEECH SEPARATION · pp. 24–25

L IMPACT OF THE NUMBER OF SEPARATOR ITERATIONS

  • TDANet’s MACs rise from 2.99 G/s at one iteration to 7.04 G/s at eight and 11.68 G/s at 16, while Dolphin achieves comparable results with lower overhead.The authors attribute Dolphin’s efficiency-performance trade-off to GLA capturing both global and local dependencies.Source: L IMPACT OF THE NUMBER OF SEPARATOR… · p. 25, Table 11 · p. 25

M ANALYSIS OF ENCODER AND DECODER ARCHITECTURE

  • With five GLA modules fixed, allocating two to the encoder and three to the decoder achieves the best performance across all evaluation metrics.The authors interpret the result as evidence that a relatively deeper decoder helps reconstruct target speech from high-level semantic representations.Source: M ANALYSIS OF ENCODER AND DECODER… · p. 25, Table 12 · p. 25

N ABLATION STUDY: OUTPUT FEATURE FORMULATION OF THE SEPARATOR

  • On LRS2, direct mapping outperforms masking by 0.5 dB in both SI-SNRi and SDRi and by 0.09 in PESQ; the final model uses direct mapping.Direct mapping regresses the target-speaker embedding, whereas mask-based approaches apply a feature mask to the mixture representation.Source: N ABLATION STUDY: OUTPUT FEATURE… · p. 26, Table 13 · p. 26

O ABLATION STUDY: EFFECT OF FUSION LOCATION

On LRS2, the earliest visual-audio fusion position outperformed the latest across all three reported separation metrics.

O ABLATION STUDY: EFFECT OF FUSION LOCATION

P LLM USAGE

  • The authors used an LLM only to polish wording and English clarity, not for research design, data, analysis, or scientific ideas.They attribute all substantive content, methods, and conclusions to the authors.Source: P LLM USAGE · p. 26

Related papers

Loading 2509.23610v2…