Source-linked AI summary
Dialogue-Based Streaming Audio-Visual Target Speaker Extraction with Predictive Dialogue Information
Shuhan Zhang, Wenxuan Wu, Haizhou Li
TL;DR
Online AV-TSE must track a target through natural turn-taking amid cross-talk, while most studies use simulated mixtures that lack this structure. The paper introduces a real-conversation benchmark and an LLM-based predictor that guides extraction from overlapping audio. TS-VAP improves extraction across backbones, and combining predictive, historical, and synchronous context yields the best reported performance.
Problem
Most target speaker extraction studies use simulated mixtures that lack the turn-taking structure of natural conversations, although online AV-TSE must operate through pauses and overlapping speech.
Source: §1 · p. 1Method
An LLM-based TS-VAP forecasts target and partner activity directly from overlapping audio to guide a low-latency separator, alongside historical and synchronous context.
Source: §1 · p. 1Results
TS-VAP improves extraction across all three backbones; all three context types together reach 10.03 dB average, a 10.9% relative improvement over the context-free baseline.
Source: §4.1 · p. 4, §4.2 · p. 4, §5 · p. 4Takeaways & Limitations
The authors identify fully semantic-driven separation as a future step beyond predictive conditioning.
Source: §5 · p. 4The formulation assumes that the target and partner belong to the same dyadic dialogue.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
In face-to-face, real-time communication, a talking agent must track the target speaker through natural pauses, turn-taking, and backchannels, often amid background cross-talk. Most target speaker extraction (TSE) studies, however, rely on simulated mixtures with full or sparse overlap and ignore the turn-taking of real conversations. We therefore introduce, to our knowledge, the first benchmark for online audio-visual TSE (AV-TSE), built from intact dyadic interactions with independent third-party interference. Observing that anticipating upcoming activity from semantic, acoustic, and facial cues benefits online AV-TSE, we propose an LLM-based target-speaker voice activity projection (TS-VAP) module. Unlike conventional VAP with separated speaker channels, it forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-latency separator. We further combine this predictive context with historical and synchronous speaker context. Experiments show that TS-VAP consistently improves streaming extraction across multiple AV-TSE backbones, and that further combining historical, synchronous, and predictive context yields nearly 1 dB gain on real AV conversations. Project page: https://jjjjiaozi.github.io/TS-VAP/.
1. INTRODUCTION
The paper addresses online audio-visual target speaker extraction in natural dyadic conversation with unrelated interference, where turn-taking makes streaming extraction difficult. It introduces a benchmark and an LLM-based module that forecasts activity from overlapping audio to guide extraction.
1. INTRODUCTION
- In the target setting, a face track identifies the participant without a separate enrollment utterance and remains reliable when audio is noisy or heavily overlapped.The system uses the mixture audio and target face track to retain the target while suppressing the conversational partner and unrelated speaker.Source: §1 · p. 1, Figure 1 · p. 1
- Most online AV-TSE studies use fully overlapped or manually simulated sparse-overlap mixtures, which lack the turn-taking structure that governs target silences and difficult extraction moments.Streaming models lack future frames, making turn boundaries and similar-sounding overlapping voices especially ambiguous.Source: §1 · p. 1
- LLM-based TS-VAP forecasts near-future target and interferer activity directly from overlapping mixtures, using the prediction to guide a low-latency separator.It draws on a speech-LLM’s linguistic and conversational knowledge and is considered alongside historical and synchronous context.Source: §1 · pp. 1–2
- The paper introduces an online AV-TSE benchmark built from real dyadic dialogues with third-party interference, preserving natural turn-taking rather than independently assembled utterances.The benchmark is built from IEMOCAP and a RealTalk-based test set.Source: §1 · p. 1
2. METHOD
The method formulates chunk-wise extraction from mixture audio and a target face track, then equips a streaming separator with predictive, historical, and synchronous context. It also adds target-activity supervision and inference-time attenuation.
2.1. Problem Formulation
- Chunk-wise extraction maps mixture audio and the target face track, together with predictive context from observations through step k−1, to a reconstructed target waveform.The target and conversational partner belong to the same dyadic dialogue; the mixture also includes independent third-party speech.Source: §2.1 · p. 2
2.2. LLM-based TS-VAP
- TS-VAP produces 256-dimensional predictive representations at 25 Hz, with each frame predicting joint target–partner activity over the next 2 seconds.The 256 joint states represent four binary future-activity bins per speaker; the frozen representations construct the separator’s predictive context.Source: §2.2 · p. 2
2.3. Online AV-TSE with Historical, Synchronous, and Predictive Knowledge
- Historical context summarizes past separator latents, synchronous context combines ASD and visual features, and predictive context encodes anticipated target–partner activity.The table categorizes these as self-enrollment, ASD, and LLM-based TS-VAP prediction, respectively.Source: §2.3 · p. 2, Table 1 · p. 3
2.4. Training Objective and Evaluation
- A personal VAD head supervises target-absent regions; inference thresholds its probabilities into attenuation, while SI-SNR is reported on target-present portions grouped by overlap ratio.The objective combines negative SI-SNR over target-active samples, scaled by their temporal proportion, with binary cross-entropy for pVAD.Source: §2.4 · p. 3
3. EXPERIMENTAL SETUP
The experiments use intact dyadic conversations mixed with independent third-party interference, alongside a RealTalk test set for cross-corpus evaluation. Backbones share an online training and inference protocol, and Table 2 evaluates suppression when the target is silent.
3.1. Datasets
- IEMOCAP-Dialog3Mix preserves pauses, overlaps, and turn transitions in 6-s mixtures with independent third-speaker interference; its train, validation, and test sets contain 23,522, 8,120, and 3,000 mixtures.RealTalk-Dialog3Mix adds a 1,000-mixture test set from unscripted RealTalk conversations for cross-corpus generalization.Source: §3.1 · p. 3
3.2. Implementation Details
- Five AV-TSE backbones are adapted to the same causal online setting and Vox2Mix pretraining; TS-VAP is tested on AV-SepFormer, TDSE, and USEV, with context combinations evaluated on USEV.Every system uses a pVAD head.Source: §3.2 · pp. 3–4
- Table 2 evaluates correct suppression rate (CSR, %) under non-target interference when the target is silent; higher values are better.The supplied caption does not give table cell values.Source: Table 2 · p. 4
- TS-VAP is trained separately to predict the subsequent 2 s from 10-s IEMOCAP observations, then frozen during AV-TSE training; online inference uses a 2-s window and 600-ms hop.Acoustic VAP uses a TalkNet-based audio-only architecture with the same training protocol.Source: §3.2 · p. 4
4. RESULTS
TS-VAP improves online extraction across three backbones, while combining predictive, historical, and synchronous context gives the best reported context configuration. The gains also appear in target-silent interference suppression and zero-shot transfer to RealTalk.
4.1. Core Results
- TS-VAP raises average TP SI-SNR by 5.7%, 4.6%, and 5.8% for AV-SepFormer, TDSE, and USEV, respectively; USEV reaches 9.56 dB, 0.15 dB above Acoustic VAP.These results use 3,000 IEMOCAP-Dialog3Mix test mixtures and a 2-s processing window with a 600-ms hop.Source: §4.1 · p. 4
4.2. Effectiveness of Historical, Synchronous, and Predictive Context
- Combining historical, synchronous, and predictive context yields 10.03 dB average TP SI-SNR, a 10.9% relative improvement over the context-free baseline; the full configuration gives no gain in the (80, 100] % overlap bin.History+Pred. is the best pairwise combination, slightly outperforming History+Sync. and Sync.+Pred.Source: §4.2 · p. 4
4.4. Zero-shot Transfer to RealTalk
- On RealTalk, TS-VAP improves all tested backbones; TDSE rises from 3.35 to 5.48 dB, and full-context USEV reaches 8.03 dB.The figure reports TP SI-SNR in dB for configurations evaluated with and without TS-VAP.Source: Figure 3 · p. 4, §4.4 · p. 4
4.3. Interference Suppression Guided by TS-VAP
- Target-silent active non-target speech risks leakage; LLM-VAP adds 2.03 and 3.27 percentage points in CSR over Acoustic VAP for partner and third-party interference, respectively.The evaluation conditions may overlap when both interferers are active.Source: §4.3 · p. 4
5. CONCLUSION
The paper studies streaming AV-TSE in natural conversations with third-party interference and proposes TS-VAP for predictive conversational context. It reports benefits from predictive context, especially alongside historical and synchronous cues, while identifying deeper semantic integration as future work.
5. CONCLUSION
- The paper studies streaming AV-TSE in natural conversations with third-party interference and proposes LLM-based TS-VAP to provide predictive conversational context.The conclusion frames deeper LLM integration, toward fully semantic-driven separation, as future work.Source: §5 · p. 4
- Predictive context improves extraction and complements historical and synchronous cues; LLM-based TS-VAP also outperforms its acoustic counterpart.Source: §5 · p. 4
- Jointly leveraging predictive, historical, and synchronous context yields the best performance among the reported configurations.Source: §5 · p. 4