Source-linked AI summary
Tap2Listen: Instant Target Switching for Streaming Audio-Visual Speaker Extraction
Hansol Park, Minkyung Song, Zinu Kim, Yujun Lee, Hoseong Ahn, Hyunjin Choi, Kyuhong Shim
TL;DR
Streaming AV-TSE systems generally fix a target speaker even though users may want immediate switches during conversation. Tap2Listen keeps audio processing target-agnostic and conditions decoding on the selected face; it improves extraction quality and produces full-quality output from the first frame after a known-face switch.
Problem
Streaming AV-TSE systems typically fix the target speaker for a clip, while conversational users may switch targets repeatedly and expect immediate response.
Source: §1 · p. 1Method
Tap2Listen processes the mixture with a target-agnostic causal encoder and uses selected-face visual cues only in the decoder, re-projecting cached memory on a switch.
Source: §1 · pp. 1–2Results
9.68 dB SI-SNRi versus the baseline’s 8.85 dB at comparable parameter count, with full quality from the first frame after a known-face switch while the baseline takes about one second to recover.
Source: §1 · p. 2Takeaways & Limitations
Tap2Listen evaluates switching through response delay and post-switch recovery and is reported to run in real time on a single mobile CPU core.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Streaming audio-visual target speaker extraction (AV-TSE) uses a target speaker's face to extract the speaker's speech from an input audio mixture in real time. However, existing systems fix the target speaker for an entire clip, while conversation demands frequent and immediate target switches. To address this problem, we propose Tap2Listen, a streaming AV-TSE model designed for instant response to target switches. The model separates computation by modality: the target-agnostic audio encoder processes only the mixture, while the target-specific visual cues of the selected face condition only the decoder. A switch requires only a 0.12ms re-projection and takes effect immediately. We also design component-wise losses: a prediction loss for the encoder and a supervised extraction loss for the decoder. For evaluation, we measure how quickly output quality recovers after a switch. On the standard LRS3-2mix test set, Tap2Listen improves SI-SNRi from 8.85 to 9.68 dB at a comparable parameter count. After a switch to a known face, the model produces full-quality output from the first frame, whereas the baseline needs about 1 second to recover. Finally, we show that the proposed streaming AV-TSE model runs in real time on a single CPU core.
1 Introduction
Interactive target switching remains little studied in streaming AV-TSE, where existing systems generally fix a speaker for an entire clip. Tap2Listen separates target-agnostic audio processing from selected-face conditioning and is reported to switch immediately while retaining strong extraction quality.
1 Introduction
- Existing streaming AV-TSE systems typically fix the target for a clip, although conversational users may switch speakers repeatedly and expect each switch to take effect immediately.Interactive target switching has received little attention.Source: §1 · p. 1
- Tightly coupled audio-visual processing and target-dependent streaming state can make a visual target change entangled with ongoing computation and delay the transition.Previous-speaker information may persist in recurrent, autoregressive, or speaker-tracking states.Source: §1 · p. 1
- Tap2Listen keeps the causal audio encoder target-agnostic and conditions only the decoder on the selected face, so a tap re-projects cached memory without invalidating the encoder cache.The re-projection immediately produces output for the newly selected speaker.Source: §1 · pp. 1–2
- 9.68 dB SI-SNRi versus the baseline’s 8.85 dB, with full quality from the first frame after a switch to a known face while the baseline takes about one second to recover.The comparison is at a comparable parameter count.Source: §1 · p. 2
- The proposed contributions include component-wise extraction and prediction losses, evaluation of response delay and post-switch recovery, and real-time execution on a single mobile CPU core.The evaluation supplements utterance-level extraction quality with transition measurements.Source: §1 · p. 2
2 Tap2Listen: Instant Target Switching for Streaming AV-TSE
Tap2Listen separates persistent target-agnostic audio processing from selected-face decoding, then trains the components with extraction and mixture-only prediction objectives. At inference, target switches recompute target-specific memory from cached audio rather than rerunning the encoder.
2.1 Architecture
- An 8.06 M-parameter target-agnostic causal encoder processes the mixture, while the selected face conditions the decoder; the encoder cache remains valid across switches.The encoder uses 16 ms windows, 8 ms hops, and local causal attention over 128 frames; the rolling lip-feature buffer spans 4 seconds.Source: §2.1 · p. 2
- The decoder combines temporally aligned lip features with cached audio representations to form attention memory, and the face identity embedding supplies its query.The fixed identity embedding appears only in the decoder query, while time-varying lip features guide cross-attention.Source: §2.1 · p. 2
- Four causal cross-attention and feed-forward blocks produce complex ratio masks that are converted to waveform output at each 8 ms hop, without autoregressive feedback.Causality is preserved because decoder memory contains only frames up to the current time.Source: §2.1 · p. 3
2.2 Training Strategy and Loss
- Training treats each mixture speaker in turn as target, runs the encoder once per mixture, and recomputes memory projection for each speaker under deep supervision.The shared mask head produces target estimates at each decoder block.Source: §2.2 · p. 3
- A mixture-only auxiliary head predicts the change in log magnitude five frames ahead, requiring no target-speaker labels.The final objective combines extraction and prediction losses with prediction-loss weight λ = 0.1.Source: §2.2 · p. 3
2.3 Inference and Target Switching
- At inference, a switch recomputes target-specific projection over cached frames with the new face’s buffered lip features and replaces the identity term without rerunning the audio encoder.Returning to a previously selected face costs the same target-specific recomputation as any other switch.Source: §2.3 · p. 4
3 Experimental Results
Tap2Listen improves fixed-target extraction and responds to target switches more quickly than the baseline. The evaluation also reports its on-device latency on a Galaxy S22.
3.1 Setup
- The LRS3-2mix evaluation uses 40k/5k/3k train/validation/test rows, with only the test split speaker-independent, and compares models on reconstructed switching mixtures.The released checkpoint reproduces no-switch and switch-face means within 0.10 and 0.56 dB, respectively.Source: §3.1 · p. 4
3.2 Fixed-Target Extraction
- Tap2Listen reaches 9.68 dB SI-SNRi, 0.71 PESQi, and 0.20 STOIi, exceeding the baseline under identical conditions.Its SI-SNRi improvement is +0.83 dB at matched parameter count; algorithmic latency is 16 ms versus approximately 1 ms for AV-SkiM.Source: §3.2 · p. 4
3.3 Target Switching
- After switching to a known face, Tap2Listen leads by 2.59 dB in the first 200 ms window with no recovery period, while AV-SkiM takes about 1 s to close the gap.For first-appearing faces, the gap is 8.08 dB over the first second and peaks at 11.9 dB at 80 ms; the baseline still trails by 3.0 dB at 1.6 s.Source: §3.3 · pp. 4–5, Figure 3 · p. 5
3.4 On-device Latency
- On a Galaxy S22, each 8 ms frame takes 3.2 ms at p95, and tap-triggered re-projection takes 5.2 ms at p95 for 8.5 ms transition latency including current-frame computation.No frame exceeds the 8 ms budget; larger blocks reduce computation but increase buffering delay.Source: §3.4 · p. 5
4 Conclusion
Tap2Listen separates persistent target-agnostic audio processing from target-conditioned decoding. It improves extraction quality and switching behavior over AV-SkiM, while operating within a single mobile CPU core’s real-time budget.
4 Conclusion
- Tap2Listen confines target-dependent computation to one memory projection, enabling each switch as a 5.2 ms p95 re-projection on a Galaxy S22 without a state-driven recovery period.Its output matches extraction conditioned on the new speaker from the start to approximately −70 dB.Source: §4 · p. 5
A.1 Non-Streaming Methods
Prior non-streaming AV-TSE work uses visual cues and varied fusion architectures to separate speech associated with visible speakers. These methods evaluate fixed-target separation rather than user-initiated switching during processing.
A.1 Non-Streaming Methods
- Looking to Listen separates speech for each detected face using the visual stream and mixed audio, but is not real-time and is better suited to video post-processing.Source: §A.1 · p. 8
- VisualVoice combines lip motion and facial appearance, jointly learning speech separation and cross-modal face-voice representations; MuSE progressively refines target speech using self-enrolled embeddings across extractor blocks.Source: §A.1 · p. 8
- CTCNet repeatedly fuses multi-resolution audio and visual features, while RTFS-Net uses attention and reduced-resolution representations; both evaluate fixed-target, non-streaming separation.Source: §A.1 · p. 8
A.2 Streaming Methods
Streaming AV-TSE methods limit future-input dependence for real-time operation, and some reuse past information to guide extraction. Prior switching evaluations do not fully characterize the immediate post-selection response.
A.2 Streaming Methods
- RT-LA-VocE, Online AV-CrossNet, RAVEN, and Swift-Net use causal or limited-look-ahead processing to support real-time operation.RT-LA-VocE uses one video frame (40 ms) of algorithmic latency, while RAVEN runs in real time on a CPU.Source: §A.2 · p. 8
- MoMuSE stores target-speaker voiceprints to help track the speaker when visual cues are unreliable, but evaluates fixed-target extraction rather than streaming target switches.Source: §A.2 · p. 8
- Pan et al. measure SI-SNRi before and after arbitrary switches, but averages do not show immediate recovery; the proactive hearing assistant tracks intervals after a partner starts speaking without explicit target selection as reference.This paper instead uses target-selection time to measure response delay and post-switch performance evolution.Source: §A.2 · p. 8