PaperlayerView exact revision on arXiv

Source-linked AI summary

CF-AVSS: A Coarse-to-Fine Audio-Visual Speech Separation Framework with Target-Speaker Refinement and Post-Fusion for Real-World AVSE Challenge

Longjie Luo, Zhongshu Hou, Yuepeng Li, Gang Yang, Shidong Shang, Fan Zhou

arXiv:2610.21773v1cs.SDcs.AI

TL;DR

Real-world AVSE must handle overlap, reverberation, noise, and unreliable visual cues, conditions that challenge direct enhancement systems. This paper uses progressive coarse separation followed by mixture-conditioned target refinement and calibrated fusion. The system placed first on Track 1 and second on Track 2, with the best CER on both.

  • Problem

    Direct AVSE systems can struggle with natural overlap, reverberation, and degraded or desynchronized visual cues, while risking suppression of weak target phonemes.

    Source: §1 · p. 1

  • Method

    Progressive visually conditioned separation creates two ordered coarse estimates, which a mixture-conditioned TFGridNet refinement further processes before development-set-calibrated fusion.

    Source: §1 · p. 1, §3.3 · p. 2, §3.4 · p. 3

  • Results

    The system ranked joint first on Track 1 and second on Track 2, achieving the lowest CER at 12.31% and 14.79%, respectively.

    Source: §5.1 · p. 4, §6 · p. 5

  • Takeaways & Limitations

    The results show strong performance across speech quality, intelligibility, speaker preservation, and recognition accuracy in the challenge evaluation.

    Source: §5.1 · p. 4

    Because background noise is shared by both targets, the coarse estimates are not constrained to sum to the mixture.

    Source: §3.3 · p. 2

Abstract

from arXiv · show

Audio-visual speech enhancement (AVSE) leverages visual cues to recover target speech from mixtures corrupted by reverberation, noise, and overlapping speakers. We propose a coarse-to-fine two-stage AVSE framework. In the first stage, a progressive SPAV-TFGridNet is extended to two synchronized visual streams to produce visually ordered coarse estimates by progressively suppressing interfering speech while preserving target reverberation and shared background noise. In the second stage, a TFGridNet-based target speaker refinement network takes the original mixture and both coarse estimates as input to further recover target speech and improve signal quality. The coarse and refined estimates are finally combined using development-set-calibrated fusion. On the ISCSLP 2026 Real-World AVSE Challenge, our system tied for first place on Track 1 and ranked second on Track 2 in OVRL, while achieving the lowest CER on both tracks at 12.31% and 14.79%, respectively.

1. Introduction

The paper addresses weaknesses of direct AVSE systems in real recordings with a coarse-to-fine framework. The system tied for first on Track 1 and placed second on Track 2, with the best CER on both tracks.

1. Introduction

  • AVSE combines acoustic observations with visual speech cues to recover desired speech when reverberation, noise, or concurrent speakers impair audio-only processing [3] [6].Source: §1 · p. 1
  • Direct one-step AVSE can struggle with non-additive overlap, reverberation, imperfect or desynchronized video, and weak target phonemes that may be suppressed with interference.These issues create tension among perceptual quality, intelligibility, and identity preservation.Source: §1 · p. 1
  • Track 1 tests natural overlap, reverberation, noise, and device responses; Track 2 adds visual degradation and 3 m far-field audio across real mixes and reference-available remixes.The challenge combines fidelity, quality, recognition, and speaker-similarity measures to emphasize balanced performance.Source: §1 · p. 1
  • The system tied for first on Track 1 and ranked second on Track 2, while achieving the best CER on both tracks at 12.31% and 14.79%, respectively.It also obtained an official OVRL of 2.86 on both tracks.Source: §1 · p. 1
  • Progressive SPAV-TFGridNet produces two visually ordered coarse outputs, suppressing non-target speakers while retaining target reverberation and shared background noise.It extends the single visually conditioned output to two jointly estimated outputs.Source: §1 · p. 1
  • A mixture-conditioned TFGridNet refinement uses both coarse outputs, asymmetric over-suppression and quality-oriented losses, followed by development-set-calibrated linear fusion.The fusion coefficient is selected on the development set and held fixed for test inference.Source: §1 · p. 1

2. Preliminary

The preliminary formulation models a single-channel noisy, reverberant mixture of at least two speakers. It defines the component signals and presents Figure 1 as an overview of the proposed two-stage method.

2. Preliminary

  • The setup considers a single-channel noisy and reverberant mixture containing M ≥2 speakers.Source: §2.1 · p. 1
  • Each speaker's clean speech s_m is convolved with room response h_m to form reverberant speech u_m, and n denotes background noise.Source: §2.1 · p. 1
  • Figure 1 provides an overview of the proposed two-stage AVSE method.Source: Figure 1 · p. 2

3. Method

The method uses synchronized visual streams to produce visually ordered coarse speech estimates, then refines them using the original mixture and both coarse estimates. A development-set-calibrated linear fusion combines the coarse and refined outputs.

3.1. System overview

  • The system processes the mixture and two synchronized video streams into visually ordered coarse estimates, refined estimates, and final outputs.Visual features are extracted separately for each stream and remain aligned with their corresponding targets.Source: §3.1 · p. 2, §3.2 · p. 2

3.2. Visual representation

  • The visual representation uses AV-HuBERT Base features extracted from mouth-centered regions at 25 frames per second for each video stream.The preprocessing detects 68 facial landmarks and aligns each face to a canonical mean shape.Source: §3.2 · p. 2

3.3. Progressive coarse separation

  • The coarse AVSS stage jointly estimates two visually ordered targets, progressively suppressing competing speech while preserving target reverberation and shared background noise.Its AVTFGrid blocks use feature-wise audio-visual attention conditioned on both visual streams; earlier stages retain more interference, while the final level reaches the background-preserving coarse target.Source: §3.3 · p. 2

3.4. Contextual target speaker refinement

  • A TFGridNet-based refinement stage jointly uses the original mixture and both coarse estimates to recover target components suppressed during coarse separation while retaining their identity cues.It concatenates contextual representations before TFGrid blocks and decoding, and uses an asymmetric loss that penalizes missing target magnitude in active bins without directly encouraging excess energy.Source: §3.4 · p. 3

3.5. Post-fusion

  • The final output linearly combines coarse and refined estimates using a global fusion weight selected on the development set and fixed for all test samples.The coarse estimate preserves reverberation and background ambience, while the refined estimate favors cleaner target extraction.Source: §3.5 · p. 3

4. Experimental Setup

The experiments use simulated mixtures built from synchronized speech and video, then validate and tune on official development sets before organizer-scored challenge testing. Evaluation combines intrusive and non-intrusive quality measures, recognition accuracy, and speaker similarity.

4.1. Datasets

  • Training mixtures combine 2–4 speakers, two visible targets, reverberant sources, and environmental noise in synchronized 4.0-second segments.The source utterances are convolved with 16-kHz room impulse responses; remaining speakers are off-screen interferers.Source: §4.1 · p. 3
  • Official testing uses 4,257 Track 1 clips and 4,941 Track 2 clips, with hidden references and organizer-computed scores.Track 2 adds degraded visual conditions and far-field recordings, increasing visual corruption and acoustic mismatch relative to Track 1.Source: §4.1 · p. 3

4.2. Evaluation metrics

  • SI-SDR, PESQ, and STOI use the remix subset with clean references, while UTMOSv2 and DNSMOS cover both mix and remix.CER comes from normalized FunASR hypotheses, and SPK-SIM is cosine similarity between enhanced-speech and enrollment-voiceprint embeddings.Source: §4.2 · pp. 3–4

4.3. Implementation details

  • Both networks train for 50 epochs, and the development-set fusion coefficient α = 0.34 is fixed for both tracks at test time.Track 2 training includes random visual degradation and drops the visual modality with probability 0.1.Source: §4.3 · p. 4

5. Results and Discussion

The system ties for first on Track 1 and ranks second on Track 2, with the lowest CER on both tracks. The Track 2 ablation indicates that post-fusion balances complementary coarse and refined estimates across metrics.

5.1. Official test-set results

  • 12.31% CER on Track 1 and 14.79% on Track 2 are the lowest reported values, alongside a shared first-place finish and second-place finish, respectively.The system shows only moderate degradation in SI-SDR, STOI, and SPK-SIM under Track 2’s less reliable visual evidence.Source: §5.1 · p. 4

5.2. Development-set ablation study

  • DNS-OVRL rises from 1.56 to 2.20 with TSR, while SI-SDR, PESQ, and SPK-SIM decrease and CER increases.The passage attributes this trade-off to perceptual enhancement potentially discarding target-speech components.Source: Table 2 · p. 4, §5.2 · p. 4
  • Post-fusion achieves the best SI-SDR, PESQ, CER, and SPK-SIM while retaining a clear DNS-OVRL gain over coarse separation alone.The coarse estimate preserves speaker identity and acoustic context, while refinement emphasizes perceptual quality.Source: Table 2 · p. 4, §5.2 · p. 4

5.1. Official test-set results

  • The system remains strong across speech quality, intelligibility, speaker preservation, and recognition, while other teams lead on some individual metrics.Team 1 has better intrusive metrics, while several other systems score higher on individual non-intrusive quality estimators.Source: §5.1 · p. 4

6. Conclusion

The conclusion presents a two-stage real-world AVSE method: visually conditioned coarse separation preserves target room responses and shared noise, followed by mixture-conditioned refinement and development-calibrated fusion. The system ranks joint first on Track 1 and second on Track 2, with the best CER on both.

6. Conclusion

  • Progressive visually conditioned coarse separation establishes two target identities while retaining room responses and shared background noise.Mixture-conditioned TFGridNet-based refinement produces cleaner target signals while mitigating over-suppression.Source: §6 · pp. 4–5
  • The system ranks joint first on Track 1 and second on Track 2 and achieves the best CER on both tracks.Coarse and refined outputs are linearly fused using a coefficient selected on the development set.Source: §6 · p. 5

Related papers

Loading 2610.21773v1…