PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Personalized Speech Enhancement for Low-Latency Audio-Visual Target-Speaker Extraction

Rayhan Rashed, Senja Filipi, Ross Cutler

arXiv:2609.30631v1eess.AScs.CVcs.SD

TL;DR

Online target-speaker extraction must preserve a requested voice while removing competing speech, and existing online systems described here were trained and evaluated on simulated mixtures. AV-PVQE adapts personalized speech enhancement by adding mouth features to PVQE’s speaker conditioning and fine-tuning its reconstruction network. It improves target recovery over PVQE while retaining similar mean quality ratings, and exceeds AVASE in target recovery and listener ratings, with larger separation gains on recorded meetings.

  • Problem

    Prior online extraction work described here was trained and evaluated on simulated two-speaker mixtures, while useful extraction must preserve a lone target and reject speech when the target is silent.

    Source: §1 · p. 1

  • Method

    AV-PVQE adds mouth features alongside enrollment at PVQE’s speaker-conditioning input and fine-tunes its reconstruction network using current and earlier video frames.

    Source: §1 · p. 1, §2 · p. 2

  • Results

    AV-PVQE improves target recovery over PVQE with similar mean perceptual-quality ratings, and exceeds AVASE in target recovery and listener ratings, with larger separation gains on recorded meetings.

    Source: §1 · p. 2, §5 · p. 4

  • Takeaways & Limitations

    Pretrained personalized speech enhancement can be adapted for online audiovisual target-speaker extraction with no lookahead.

    Source: §5 · p. 4

    Device processing time and capture-to-output delay remain to be measured.

    Source: §5 · p. 4

Abstract

from arXiv · show

Online audio-visual target-speaker extraction aims to remove competing voices while preserving speech quality and bounding lookahead. Existing extractors are built and evaluated for separation on synthetic mixtures, leaving listening quality and meeting behavior largely untested. We introduce Audio-Visual Personalized Voice Quality Enhancement (AV-PVQE), which approaches these requirements from the other direction. We start from a personalized speech enhancement model that reconstructs a requested voice at high quality but confuses the target in 46% of two-speaker mixtures despite clean enrollment. Adding mouth features at its speaker-conditioning input and jointly fine-tuning the visual and reconstruction networks reduces this rate to 1.6%, with no future frames and 20 ms of algorithmic delay. Compared with an online autoregressive audio-visual extractor, AV-PVQE yields separation gains on two synthetic benchmarks and larger gains on recorded meetings, and keeps its advantage on excerpts with more speakers than the fine-tuning mixtures. In personalized P.835 listening tests on two meeting corpora, it improves overall quality over this extractor by 0.57 and 0.63 MOS, with similar mean rating relative to the starting model. Preservation and rejection tests show that it keeps the target intact when no competing voice is present and suppresses competing speech when the target is absent.

1. INTRODUCTION

Online target-speaker extraction must preserve a requested voice while removing competing speech, but prior online extraction work described here was trained and evaluated on simulated mixtures. AV-PVQE adapts personalized enhancement for that task, improving target recovery over PVQE and AVASE while retaining similar mean quality to PVQE.

1. INTRODUCTION

  • Online extraction must preserve the participant’s voice during overlap and solo speech, then remove remaining voices when the participant is silent.Source: §1 · p. 1
  • PVQE selects the competing speaker in 46% of synthetic two-speaker LRS3 mixtures despite clean enrollment, while audio-only DNSMOS scores cannot identify which speaker was recovered.Source: §1 · p. 1
  • The evaluation compares AV-PVQE with AVASE and PVQE on two synthetic corpora and three meeting corpora, including personalized P.835 listening tests and target-preservation and rejection tests.Source: §1 · p. 1
  • AV-PVQE adds mouth features to PVQE’s speaker-conditioning input, combines them with enrollment, and fine-tunes the reconstruction network without future frames.Source: §1 · p. 1, Figure 1 · p. 2
  • AV-PVQE improves target recovery over PVQE with similar mean perceptual-quality ratings, and exceeds AVASE in target recovery and listener ratings, with larger separation gains on recorded meetings.Source: §1 · p. 2

2. AV-PVQE MODEL

AV-PVQE feeds aligned mouth features and enrollment information into PVQE’s speaker-conditioning input. Its training and streaming design use only current and past inputs, with 20 ms of algorithmic and buffering delay.

2. AV-PVQE MODEL

  • The model combines time-varying mouth features with a fixed enrollment voice reference through a gated enrollment residual at PVQE’s speaker-conditioning input.Source: §2 · p. 2
  • The visual cue is extracted from mouth features, projected from 512 dimensions to 176, and repeated to match the audio feature rate.Source: §2 · p. 2
  • Zero-initializing the residual’s final weights makes the initial conditioning equal the visual cue, after which the enrollment residual is learned during fine-tuning.Source: §2 · p. 2
  • Online fine-tuning replaces whole-excerpt cue aggregation with a running mean, remaps decoder filters to current and past inputs, and updates the visual, fusion, and acoustic networks together.Source: §2 · p. 2
  • AV-PVQE uses no future audio or video and has 20 ms of algorithmic and buffering delay; chunked and full-waveform outputs agree above 128 dB SI-SNR.Source: §2 · p. 2

3. EXPERIMENTAL SETUP

The evaluation combines synthetic two-speaker mixtures with meeting excerpts tested without further fine-tuning, using PVQE and AVASE as baselines. Objective measures, listening tests, and target-present/absent tests assess recovery, quality, and suppression behavior.

3. EXPERIMENTAL SETUP

  • Training uses 40,000 same-corpus mixture pairs, target-to-interferer ratios from −10 to 10 dB, and test speakers unseen during training and validation.Each test set contains 3,000 fixed pairs; results are grouped by quieter, equal (within ±2 dB), and louder targets.Source: §3 · p. 2
  • PVQE provides the no-video baseline with enrollment, while AVASE uses video and its own previous output as an acoustic cue, without enrollment.All systems process the same locally generated test mixtures, so their absolute scores are not comparable with AVASE’s published results.Source: §3 · p. 2
  • Meeting tests use AMI headset-channel mixtures, MTM participant-channel sums grouped by speakers in each excerpt, and MCoRec central-microphone recordings.MCoRec’s worn-microphone recording follows a different acoustic path and is not a separation reference.Source: §3 · p. 3
  • Objective measures include SI-SNRi, target preference, speaker similarity, and word accuracy; DNSMOS estimates output quality without enrollment.MCoRec uses corrected reference transcripts, while the other corpora use Whisper transcripts of target references.Source: §3 · p. 3
  • Preservation tests count target-speaker suppression events lasting over one second and measure the fraction of target-only outputs within ±3 dB of input level.For target-absent tests, the target audio is removed and video is either a repeated still or cyclically shifted within the excerpt.Source: §3 · p. 3
  • Personalized P.835 tests compare three systems on AMI and MCoRec using 395 screened listeners and 107 excerpts per corpus.Listeners rate target speech, background, and overall quality on five-point scales after hearing a clean target-speech reference.Source: §3 · p. 3, Table 4 · p. 4

4. RESULTS AND DISCUSSION

AV-PVQE’s advantage over AVASE grows from synthetic mixtures to recorded meetings and extends across speaker counts. It also improves listener-rated quality over AVASE while preserving target speech and suppressing competitors in separate tests.

4.1. Extraction on synthetic mixtures

  • AV-PVQE selects the target at least 97% of the time at every target level, while PVQE selects it in only 6.25% of mixtures where the target is quieter.On both synthetic benchmarks, AV-PVQE also improves SI-SNRi, word accuracy, and speaker similarity over AVASE.Source: §4.1 · p. 4

4.2. Generalization to meetings

  • AV-PVQE’s SI-SNRi margin over AVASE grows from 0.31 dB on LRS3 and 1.84 dB on VoxCeleb2 to 4.43 dB on AMI and 6.58 dB on MTM.Its lead over AVASE holds across MTM speaker-count groups; with four speakers, target preference is 75.00% for AV-PVQE versus 56.82% for AVASE.Source: §4.2 · p. 4

4.3. Target preservation and rejection

  • Across 6,000 target-only LRS3 inputs, AV-PVQE produces seven long target-suppression events, versus 142 for PVQE, and keeps 96.93% of outputs within ±3 dB of input level.When target audio is removed, AV-PVQE attenuates competing speech by 32.83 dB with a still image and 18.28 dB with shifted video.Source: §4.3 · p. 4

4.4. Listener-rated quality

  • Overall quality improves by 0.57 MOS on AMI and 0.63 MOS on MCoRec over AVASE; AV-PVQE’s overall-quality rating does not measurably differ from PVQE’s on either corpus.The AV-PVQE–PVQE OVRL differences are −0.06 MOS on AMI and +0.17 MOS on MCoRec.Source: §4.4 · p. 4

5. CONCLUSION

AV-PVQE adapts a pretrained personalized speech enhancer for online audiovisual target-speaker extraction with no lookahead. It improves target recovery over PVQE while retaining similar mean overall-quality ratings, and outperforms AVASE in target recovery and listener ratings, with larger separation gains on recorded meetings.

5. CONCLUSION

  • AV-PVQE improves target recovery over PVQE while retaining similar mean overall-quality ratings, and outperforms AVASE in target recovery and listener ratings, with larger separation gains on recorded meetings.Future work will measure device processing time and capture-to-output delay, test joint enhancement and extraction training, and compare models without video and from random initialization.Source: §5 · p. 4

7. COMPLIANCE WITH ETHICAL STANDARDS

The listening study was conducted as industry research through Prolific, with voluntary compensated participation and informed consent.

7. COMPLIANCE WITH ETHICAL STANDARDS

  • Participants gave informed consent, and ratings were recorded under pseudonymous participant IDs.Participation was voluntary and compensated; the study was conducted as industry research through Prolific.Source: §7 · p. 5
Loading 2609.30631v1…