PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring a Single Autoregressive LLM for Unified Target Speech Extraction across Synchronous and Asynchronous Cues

Wenxuan Wu, Shuhan Zhang, Shuai Wang, Haizhou Li

arXiv:2609.29238v1cs.SDcs.MMeess.AS

TL;DR

Target speech extraction commonly uses separate cue-specific systems, and visual-cue systems may require corruption-matched training to handle missing frames. TSE-Omni uses one autoregressive LLM for synchronous and asynchronous cues, reusing predicted speech-token history for visual compensation. It matches or surpasses baselines across most scenarios and leads under visual corruption, speaker switching, and sparse overlap.

  • Problem

    TSE systems are commonly optimized at the signal level, while existing autoregressive LLM extractors handle only one cue type and can lack a fallback for corrupted visuals or mid-mixture target switches.

    Source: §I · p. 1

  • Method

    TSE-Omni uses one autoregressive LLM for synchronous and asynchronous cues, feeding its accumulated predicted speech tokens back as self-enrollment and fusing them with synchronized visual cues when available.

    Source: §I · pp. 1–2, C. Self-enrollment and Cross-modality… · p. 4

  • Results

    TSE-Omni matches or surpasses state-of-the-art baselines across most scenarios and leads under visual corruption, speaker switching, and sparse overlap.

    Source: §VII · p. 11

  • Takeaways & Limitations

    A shared extractor reduces per-modality deployment cost and supports extraction without corruption-specific training or a visual-recovery module.

    Source: §VII · p. 11

Abstract

from arXiv · show

Target speech extraction (TSE) typically trains a separate extractor per cue, and visual-cue systems often need corruption-matched training to remain robust under visual frame corruption. We show that one autoregressive LLM backbone, TSE-Omni, can serve both temporally synchronous cues (lip movements, co-speech gestures) and asynchronous cues (enrollment audio, text). TSE-Omni is driven by next-token prediction: each step predicts target speech semantic tokens from its own past outputs, which we term self-enrollment, forming a continuous target-speech context initialized by the enrollment cue (asynchronous audio or text, or a short visual prefix). This enables audio-visual compensation: the model uses synchronized visuals when intact and its token history when visual frames are missing. Under clean visuals, TSE-Omni matches strong discriminative and generative baselines (SpeechBERTScore 0.81 on VoxCeleb2 and 0.89 on LRS3 zero-shot) with higher DNSMOS. On the same VoxCeleb2 test set, after a 2 s clean visual start, removing the remaining visual frames leaves SpeechBERTScore at 0.81. It remains usable under sparse overlap and multi-speaker interference, and supports streaming inference. Project page: https://alexwxwu.github.io/tseomni-main/.

I. INTRODUCTION

Existing TSE systems are often cue-specific and signal-focused, while visual systems can lack a fallback when frames are corrupted. TSE-Omni unifies synchronous and asynchronous cues in one autoregressive extractor and reuses its predicted speech-token history for compensation.

I. INTRODUCTION

  • Most TSE systems are optimized mainly at the signal level and do not leverage speech’s phonetic and linguistic knowledge; prior LLM approaches add textual guidance but are not purely semantic-driven.The paper identifies accumulated autoregressive speech tokens as a reusable source of speaker identity and linguistic context when external cues degrade.Source: §I · p. 1
  • Existing autoregressive LLM extractors accept either audio or visual cues, leaving no token-history fallback for missing visuals and making mid-mixture target switches ambiguous for audio-only systems.Errors at a switch point can accumulate over time.Source: §I · p. 1
  • TSE-Omni uses one autoregressive extractor for synchronous visual cues and asynchronous audio or text cues, rather than a separate extractor for each cue.Its experiments report matching strong baselines on clean-visual semantics while improving perceptual quality, with unchanged SpeechBERTScore when later visual frames are zero-filled.Source: §I · p. 2
  • Self-enrollment feeds accumulated predicted target-speech tokens back as a running cue, enabling compensation when visual frames are corrupted or missing without corruption-matched training or a recovery module.The method fuses the token history with synchronized visual tokens when available.Source: §I · p. 2, Figure 2 · p. 2
  • Experiments span VoxCeleb2, zero-shot LRS3, visual corruption, target switching, sparse overlap, multi-interferer mixtures, and streaming, with clean-visual semantic quality matching strong baselines and perceptual quality improving.SpeechBERTScore remains unchanged on the core test set when later visual frames are zero-filled.Source: §I · p. 2

II. RELATED WORK

Prior TSE work uses audio or visual cues in distinct approaches, while LLM-based systems have largely focused on audio-only extraction. TSE-Omni extends a single LLM-based extractor to audio and visual enrollment cues.

A. Audio-Based and Visual-Based TSE Models

  • Audio-based systems commonly use a short pre-recorded target utterance as an enrollment cue, with newer approaches also capturing contextual information between cue and mixture speech.Visual-based systems use synchronized lip movements aligned with the target speaker’s phonemes.Source: A. Audio-Based and Visual-Based TSE… · p. 2

2) Visual-Based TSE Models:

  • Visual corruption degrades synchronized-cue extraction; interleaved speech-and-visual refinement has been used to address severe degradation, while TSE-Omni shares one backbone across audio and visual cues.A shared backbone avoids storing a separate extractor for each cue.Source: 2) Visual-Based TSE Models · p. 2

B. Target Speech Extraction with LLMs

  • LLM-based TSE methods either inject textual linguistic guidance into traditional pipelines or fine-tune LLMs for speech tasks, but prior LLM-based TSE remains restricted to audio-only scenarios.TSE-Omni is presented as the first LLM-based TSE system to accept visual as well as audio cues in one extractor backbone.Source: B. Target Speech Extraction with LLMs · p. 2

C. Omni-LLMs

  • Omni-style multimodal LLMs unify diverse modalities; inspired by them, TSE-Omni extends LLM-based TSE to audio and visual enrollment cues.Its training framework predicts speech semantic tokens autoregressively and reconstructs waveform in a non-autoregressive stage.Source: C. Omni-LLMs · pp. 2–3, Figure 3 · p. 3

III. METHODS

TSE-Omni shares one autoregressive LLM across audio- and visual-cue extraction. It predicts target-speech semantic tokens and then reconstructs the waveform, with mixture, cue, and token-history embeddings forming the autoregressive input.

A. TSE-Omni Overview

  • TSE-Omni replaces separate audio- and visual-cue models with one AR-LLM that predicts target-speech semantic tokens and a NAR stage that generates the waveform.The NAR stage conditions on the predicted tokens, mixture, and cue.Source: A. TSE-Omni Overview · p. 3

B. Mixture and Cue Encoders

  • At each autoregressive step, the input sequence combines the mixture embedding, cue embedding, and embeddings of previously predicted speech tokens.The mixture embedding comes from a Conformer encoder; modality-specific cue front ends project their features into a shared continuous cue space.Source: B. Mixture and Cue Encoders · p. 3
  • Cue front ends cover pre-recorded audio, lip movement, co-speech gesture, and text, with missing modalities represented by zero tensors.Gesture embeddings are upsampled to the speech-token frame rate, while text embeddings are derived from partial target-speech transcripts.Source: B. Mixture and Cue Encoders · p. 3
  • All cues identify the target speaker at cold start, using asynchronous cues directly or only a short initial segment for synchronous cues.Synchronous cues can track over time by fusing aligned visual input with each newly predicted speech token; asynchronous cues instead rely on prior predicted tokens.Source: B. Mixture and Cue Encoders · p. 4

C. Self-enrollment and Cross-modality Compensation

TSE-Omni predicts target-speech semantic tokens autoregressively, using its own token history as context and fusing synchronized visual tokens when available. This lets token history continue conditioning predictions when later visual frames are corrupted or missing.

C. Self-enrollment and Cross-modality Compensation

  • Next-token prediction produces target-speech semantic tokens from the model’s own history, with each step predicting the first two of 32 RVQ layers.The paper calls the reuse of past predicted speech tokens self-enrollment.Source: C. Self-enrollment and Cross-modality… · p. 4
  • For visual-cue extraction, each prediction uses a fused representation of the previous speech token and its synchronized visual frame; audio-cue extraction uses only historical speech tokens.The visual tokens start at v2 because the first speech token has no preceding token to fuse with.Source: C. Self-enrollment and Cross-modality… · p. 4
  • When later visual frames are corrupted or missing, the model replaces them with zeros but continues conditioning on its predicted-token history through the residual speech-token path.The visual module stays on; the residual design retains the speech token with weight α while fusing the projected audio-visual token with weight β.Source: C. Self-enrollment and Cross-modality… · p. 4
  • The end-of-sequence token is predicted using a zero embedding for its corresponding visual token, avoiding an extra special token.Source: C. Self-enrollment and Cross-modality… · p. 4

D. Training

Omni-training mixes audio- and visual-cue extraction, with text and gesture cues trained separately; the implementation and dataset protocols cover core, cross-dataset, corruption, switching, and interference evaluations.

D. Training

  • Half of each Omni-training batch uses audio cues and half visual cues; text and co-speech gesture cues are trained later on their own mixtures.The default loss combines cross-entropy on discrete speech semantic tokens with an MSE term on full target-speech embeddings.Source: D. Training · p. 4

IV. IMPLEMENTATION

  • The speech and visual tokens share dimension 128 and are sampled at 25 Hz; the visual front end uses an LRS3-pretrained VSR ResNet and the default model has no visual prediction head.The optional visual head is used only in the visual-token ablation.Source: §IV · p. 4
  • The audio and visual cue encoders are aligned with MSE before Omni-training; skipping alignment degrades visual-cue performance, and training uses validation-based learning-rate halving and stopping.The learning rate starts at 10^-3, halves after three unimproved validation epochs, and training stops after six.Source: §IV · p. 4

A. Datasets

  • VoxCeleb2 supplies the core two-speaker dataset, with 400,000/2,000/2,000 train/validation/test utterances and randomly sampled 3–6-second clips and −5 to 5 dB SIR.It includes audio, lip movements, and transcripts for the same utterances; the default audio and visual cue duration is 2 seconds.Source: A. Datasets · pp. 4–5
  • LRS3 tests the VoxCeleb2 core checkpoint zero-shot for audio and visual cues, while Libri2mix tests a zero-shot VoxCeleb2 checkpoint and a LauraTSE-initialized, VoxCeleb2-finetuned variant.LRS3 visual baselines are trained in-domain; Libri2mix audio baselines are trained in-domain.Source: A. Datasets · p. 5
  • The VoxCeleb2 visual-corruption test keeps the first 2 seconds clean, then zeros later frames for full-missing, partial-occlusion, or low-resolution conditions.The full-missing and three-type-average results use the same 2,000-utterance core test set.Source: A. Datasets · p. 5, Figure 4 · p. 5
  • Additional evaluations use LRS3 target switches at 3–3.5 seconds, zero-shot three-speaker VoxCeleb2 mixtures, and fine-tuned IEMOCAP sparse-overlap mixtures.Switching is evaluated with visual cues, not audio-only dual enrollment; sparse-overlap inputs are 6 seconds for inference.Source: A. Datasets · p. 5
  • Cue-combination data add VoxCeleb2 and YGD cues, including transcripts and pre-recorded audio; YGD-2mix uses lifted 3D poses and 200,000/5,000/3,000 mixtures.Partial transcriptions mask 20% of the full transcription; VoxCeleb2 and YGD cue-combination models are trained from scratch on their respective data.Source: A. Datasets · p. 5

B. Baselines

The benchmark compares TSE-Omni with generative and discriminative extractors across audio- and visual-cue settings. The table spans LRS3, VoxCeleb2, and Libri2mix, with training and zero-shot status varying by dataset.

B. Baselines

  • Audio-cue baselines include flow-matching SoloSpeech, STFT-based NeMo, token-based non-autoregressive LLaSE-G1, and decoder-only LLM LauraTSE.Source: B. Baselines · p. 5
  • Visual-cue comparisons include RNN, Transformer, and state-space models, plus generative two-stage AVDiffuSS and its flow-matching variant FlowAVSE.Source: B. Baselines · pp. 5–6
  • Table I benchmarks LRS3, VoxCeleb2, and Libri2mix; LRS3 and VoxCeleb2 identify best visual-cue values per metric, while Libri2mix identifies best values per metric.LRS3 visual baselines and Libri2mix audio baselines are in-domain; TSE-Omni includes zero-shot rows, with A+Pretrain initialized from LauraTSE and fine-tuned on VoxCeleb2.Source: Table I · p. 6, B. Baselines · p. 6

VI. RESULTS AND ANALYSIS

On clean visuals, TSE-Omni matches leading systems on semantic similarity while reporting higher speech quality. With later visual frames missing or impaired, its speech-token history supports continued extraction and limits semantic degradation.

A. Comparison with Other Backbones

  • On VoxCeleb2, TSE-Omni (V) matches AV-Sepformer at SBS 0.81 and reports NISQA 3.24 versus 2.55/2.08/2.50 for USEV/AV-Sepformer/AV-Mamba; on zero-shot LRS3, it ties AV-Mamba at SBS 0.89 with NISQA 3.84.Its VoxCeleb2 audio-cue SBS is 0.79.Source: A. Comparison with Other Backbones · p. 6
  • On zero-shot Libri2mix audio extraction, TSE-Omni (A) reaches SBS 0.83 versus LauraTSE’s 0.90 but has higher OVL, 3.29 versus 3.21.LauraTSE initialization and VoxCeleb2 fine-tuning raises SBS to 0.88, with OVL 3.28; this is no longer zero-shot on Libri2mix.Source: A. Comparison with Other Backbones · p. 6

B. Speech Token Compensation

  • With only the first 2 s of visuals available, TSE-Omni (V) maintains SBS 0.81, SIM 0.95, and NISQA 3.06, while USEV and AV-Sepformer fall to SBS 0.71 and 0.70.This is the same 2,000-utterance VoxCeleb2 test set used for the clean evaluation.Source: B. Speech Token Compensation · p. 7, Table II · p. 7
  • Across clean and impaired visuals, TSE-Omni exceeds USEV on SBS and NISQA; its SBS changes only from 0.810 to 0.808, while NISQA changes from 3.24 to 3.08.Impaired results average full missing, partial occlusion, and low resolution after a 2 s clean start; SBS is effectively unchanged when later frames are zero-filled, but NISQA drops.Source: B. Speech Token Compensation · p. 7, Figure 6 · p. 7
  • For visual compensation, later frames are zero-filled while the visual module remains enabled, and self-enrollment uses prior predicted speech tokens without corruption-matched training or a recovery module.The study evaluates three representative degradations, although real-world visual degradation is open-ended.Source: B. Speech Token Compensation · pp. 6–7

C. Challenging Scenarios

A switch token improves target-speaker extraction when the target changes mid-utterance. The described mechanism resets accumulated speech-token history when a new visual cue conflicts with it.

1) Target Speaker Switch:

  • With SWS, SBS rises from 0.774 to 0.812 under clean visuals and from 0.766 to 0.809 under occlusion; the model also outperforms USEV across quality metrics in both conditions.Under occlusion, SBS is 0.809 versus 0.736 for USEV; under clean visuals USEV has higher SIM (0.949), but NISQA is 2.635.Source: Table III · p. 7, 1) Target Speaker Switch · p. 8
  • At a detected conflict between the new visual cue and historical speech-token prediction, the model predicts SWS, resets its autoregressive context, and follows the new target.The reported mechanism is described as mitigating inter-speaker interference during transitions.Source: 1) Target Speaker Switch · p. 8, Figure 7 · p. 8

2) Three-Speaker:

TSE-Omni is evaluated under three-speaker interference, sparse-overlap conversation, and streaming conditions. Results show different trade-offs across cue type, speech quality, and latency.

2) Three-Speaker:

  • Under three-speaker interference, TSE-Omni (A) is best on every reported metric; TSE-Omni (V) matches USEV on SBS (0.629 vs. 0.626) and exceeds it on NISQA (2.820 vs. 1.774), but has lower SIM.Audio enrollment remains the stronger identity cue in this three-speaker setting.Source: Table IV · p. 8, 2) Three-Speaker · p. 8

3) Sparse Overlap:

  • On IEMOCAP sparse overlap, TSE-Omni (V) and (A) are slightly below USEV on SBS and SIM but exceed it on NISQA (1.52 vs. 1.30) and OVL (2.26 vs. 1.98).The mixtures alternate between two talkers with long silences, and visual cues are clean.Source: Table V · p. 8, 3) Sparse Overlap · p. 8

4) Streaming Inference:

  • With 1 s chunks, MeMo has the lowest RTF (0.03) but requires a 2-second warm-up; TSE-Omni avoids that pseudo-AR warm-up, while its RTF is 0.64 with LauraGPT and 0.23 with Qwen2.5.The Qwen2.5 latency reduction accompanies an audio-cue SBS drop from 0.75 to 0.64; MeMo has slightly better SBS and SIM, while TSE-Omni has higher speech quality, including NISQA 2.73 versus 2.02 with LauraGPT.Source: 4) Streaming Inference · p. 8
  • The streaming performance drop relative to offline inference is likely because the NAR-stage vocoder benefits from broader context than a single 1-second chunk provides.Source: 4) Streaming Inference · p. 8

D. Potential of Modality Extension and Combination

Cue-combination experiments show gains in speaker similarity for audio plus lip cues, but weak modalities do not reliably improve extraction. Text-only and gesture-only results also expose limitations in cue strength and alignment.

D. Potential of Modality Extension and Combination

E. Ablation Studies

The ablations favor the compact LauraGPT-style backbone and a separate visual-integration path, while showing that speech-token history must remain balanced with visual information.

1) LLM Backbones and Initialization:

  • Visual-cue SBS is 0.806 for LauraGPT-style versus 0.710 for downscaled Qwen2.5 at a comparable 77M parameter budget.The study attributes LauraGPT's better generalization to its simpler, speech-adapted design under limited training data.Source: 1) LLM Backbones and Initialization · pp. 9–10, Table VIII · p. 9
  • Pretrained Qwen2.5-0.5B raises SBS from below 0.50 to 0.67–0.69, while LauraTSE initialization raises audio-cue SBS from 0.783 to 0.807.LauraTSE initialization slightly lowers visual-cue SBS and NISQA; the authors attribute this to the LibriSpeech/Libri2Mix-to-VoxCeleb2 domain gap.Source: 1) LLM Backbones and Initialization · p. 10

2) Audio-Visual Token Integration Strategy:

  • Dual-Path outperforms Uni-Path on speech quality, with NISQA 3.89 vs. 3.72 for audio cues and 3.84 vs. 3.79 for visual cues.Uni-Path routes audio-cue extraction through the visual module with zero-padded visual tokens, introducing an unnecessary distribution shift.Source: Table IX · p. 10, 2) Audio-Visual Token Integration… · p. 10, Figure 9 · p. 10
  • Equal residual scales perform best, while removing the speech residual drops SBS to 0.79 and setting α = 10 yields the largest quality drop, with NISQA 3.64.The speech-token residual preserves self-enrollment when later visual frames are zeroed, but its scale must be balanced against the visual branch.Source: Table X · p. 10, 2) Audio-Visual Token Integration… · p. 10

3) Stronger Visual Semantic Tokens:

The paper reports little benefit from stronger visual modeling, gains from enrolling cues at both extraction stages, and a unified model that remains usable in several challenging settings.

3) Stronger Visual Semantic Tokens:

  • Neither replacing ResNet with AV-HuBERT nor adding a visual-token prediction head improves extraction; the default keeps ResNet and omits the visual head.On LRS3, all configurations are within ±0.011 SBS of the table's best value, 0.895, a statistically negligible difference.Source: 3) Stronger Visual Semantic Tokens · p. 10, Table XI · p. 11

4) Cue Robustness:

  • Enrolling the cue at both AR and NAR stages consistently outperforms AR-only enrollment across metrics for both cues.The NAR Conformer and codec decoder are trained with the additional cue input.Source: 4) Cue Robustness · p. 10, Table XII · p. 11

VII. CONCLUSION

  • TSE-Omni matches or surpasses baselines across most scenarios and leads under visual corruption, speaker switching, and sparse overlap, while supporting streaming inference and additional cues.The conclusion identifies stronger backbones, more cue modalities, and lower streaming latency as future work.Source: §VII · p. 11
Loading 2609.29238v1…