Source-linked AI summary
RePlay: Retrieval-Based Voice Playback for Multi-Turn spoken dialogue
Sathvik Udupa, Naveen Kumar, Ryan Folmsbee
TL;DR
Full-duplex models are low-latency but cannot ensure exact response content and delivery, while cascades can select predefined lines with additional latency. RePlay uses a probed hidden state in PersonaPlex to retrieve and play pre-recorded lines. In simulated interviews, it achieves 383 ms median latency, 3 to 7 times faster than cascades of comparable dialogue quality, while exact line choice trails a large LLM.
Problem
Full-duplex spoken-dialogue models cannot guarantee exact content or reproduce a recorded performance, while cascades constrain responses to predefined content at higher latency.
Source: §1 · p. 1Method
RePlay adapts PersonaPlex by probing for the frame and layer where the upcoming response is recoverable, then uses that hidden state with turn-taking and retrieval heads to select a pre-recorded line.
Source: §1 · p. 1, §3.2 · p. 2Results
Takeaways & Limitations
RePlay generalizes to new lines within seen intents without retraining, while unseen intents benefit from in-domain data.
Source: §6 · p. 4The probing setup targets casual conversation with general, keyword-free answers, using 200 training and 100 test questions.
Source: §3.2 · p. 2
Abstract
from arXiv · showhide
Many voice interaction applications require exact control over both the content and delivery of responses, typically using pre-recorded lines. Recent full-duplex models respond with low latency but cannot guarantee exact content or reproduce a specific recorded performance, while cascaded systems can be constrained to predefined responses at the cost of additional latency. We propose RePlay, a spoken dialogue system adapted from PersonaPlex that handles multi-turn conversations by retrieving and playing pre-recorded lines. Using probing, we identify the layer and frame at which the upcoming response becomes recoverable, and use this hidden state as the retrieval query. RePlay retains only the layers up to that point and replaces text and speech generation with lightweight turn-taking and retrieval heads. In simulated multi-turn interviews, RePlay reaches a median latency of 383 ms, 3 to 7 times lower than ASR-LLM cascades of comparable dialogue quality, at the cost of lower exact-line accuracy. In a user study, participants preferred RePlay in 63% of ratings versus 12% for a fast cascade with a small LLM (p = 0.008), and showed a non-significant preference (46% vs. 21%) over a slower cascade with a stronger LLM.
1. INTRODUCTION
Full-duplex models offer low-latency speech interaction but cannot guarantee fixed content and delivery; cascades can select predefined responses with added latency. RePlay probes for a response representation in PersonaPlex and uses it to retrieve recorded lines, combining turn-taking and selection without ASR or speech synthesis.
Motivation
- Exact content and recorded delivery are not guaranteed by open-ended full-duplex generation, limiting its use in applications that need predefined responses.The paper names interactive characters, venue guides, and customer service agents as examples.Source: §1 · p. 1
Background
- Cascaded systems select from known responses after detecting turn end and transcribing speech; LLM-based selection has higher recall but higher latency.These systems separate turn detection, transcription, and response selection.Source: §1 · p. 1
RePlay
- RePlay adapts PersonaPlex by probing for the earliest frame and layer where the upcoming response is recoverable, then uses that hidden state to retrieve a line.Lightweight turn-taking and retrieval heads replace text and speech generation, jointly handling turn-end detection and response selection.Source: §1 · p. 1
Contributions
- RePlay’s contributions include a probing analysis, a truncated retrieval-based dialogue system, and simulated multi-turn and user-study evaluations.The introduction reports a 383 ms median latency, 3 to 7 times lower than cascades of similar dialogue quality.Source: §1 · p. 1
2. RELATED WORK
Prior spoken-dialogue retrieval systems retrieve knowledge or tools, then generate a response; RePlay instead retrieves the response line itself at the turn boundary for immediate playback.
Retrieval in spoken dialogue
- RePlay retrieves the response itself at the turn boundary, so the selected pre-recorded line can play immediately rather than following further response generation.Prior retrieval approaches use retrieved information to ground generation, so the user hears it only after generation.Source: §2 · p. 1
Retrieval from speech
- Speech retrieval methods include aligning a speech encoder to a frozen text retriever and retrieving from spoken queries in dialogue or long-form audio.These approaches retrieve knowledge after the user has finished speaking.Source: §2 · p. 1
- Figure 1 reports per-layer probing with QA accuracy and clustering accuracy, marking layer 15 with a dotted line.The caption identifies the plotted measures and the marked layer, but does not state their values.Source: Figure 1 · p. 2
3. PROPOSED APPROACH
The proposed approach adapts PersonaPlex for retrieval by identifying when its hidden state encodes an upcoming response, retaining the layers up to that point, and replacing generation with turn-taking and retrieval heads.
3.1. PersonaPlex
- PersonaPlex combines a Mimi speech encoder, a 32-layer Transformer LLM, and a Depth Transformer that predicts system speech audio codebooks.Its LLM predicts a text stream, with the first <EPAD> after a user turn marking the system response start.Source: §3.1 · p. 2
3.2. Identifying the layer and frame for retrieval
- Probing uses ridge regression at candidate response-start frames and each layer to map hidden states to response embeddings, measuring top-5 retrieval and balanced cluster accuracy.The test uses general, keyword-free answers, with 200 training and 100 test questions.Source: §3.2 · p. 2
- The response becomes recoverable only when <EPAD> is fed back as input; the signal rises around layers 13–14, so RePlay uses layer 15 and retains the first 16 layers.Keeping those layers halves LLM compute while preserving a clear response signal.Source: §3.2 · p. 2
3.3. RePlay model
- RePlay discards the remaining PersonaPlex layers and speech generator, using two linear heads for turn-taking and line retrieval; new lines can be added by embedding them.The turn-start <EPAD> identifies the response boundary, and the following frame supplies the retrieval query.Source: §3.3 · p. 2
3.4. Retrieval training
- Retrieval training aligns projected hidden states with frozen line embeddings using a one-directional SigLIP loss and a cosine-alignment term.The candidate set includes the target line and hard negatives mined by embedding similarity, excluding near-paraphrases.Source: §3.4 · pp. 2–3
4. EXPERIMENTAL SETUP
The experiments use synthetic multi-character dialogue and Nobu interview data, train RePlay with D1 pretraining and D2 fine-tuning, and compare it with streaming-ASR/LLM cascades. Evaluation combines simulated interviews with a small real-user study.
4.1. Datasets
- D1 comprises 52,768 synthetic dialogues across 100 characters and 29,225 lines; D2 contains 300 Nobu lines across 30 interview intents.D2 is used for simulated job interviews, with evaluation splits holding out lines or intents.Source: §4.1 · p. 3
4.2. Training details
- Training uses synthetic D1 pretraining followed by fine-tuning on 2 hours of simulated D2 interviews; PersonaPlex supplies turn-taking robustness from training that includes real conversations.To preserve robustness while training on synthetic data, LoRA is applied to layers 0–9 and layers 10–15 are fully fine-tuned.Source: §4.1 · p. 3, §4.2 · p. 3
4.3. Baselines
- The baselines are streaming-ASR/LLM cascades that select and immediately play a pre-recorded line after detecting the user-turn end.Cascade A uses a commercial LLM; B and C use local LLMs, with C trading accuracy for latency through a small model and embedding-based line mapping.Source: §4.3 · p. 3
4.4. Evaluation
- Evaluation simulates 100 interviews of up to 90 seconds, using an LLM judge to assess intent agreement, recall at k, engagement, and responsiveness.Latency is measured from the end of user speech to system reply; early replies are reported separately as cutoff rate.Source: §4.4 · pp. 3–4
4.5. User study
- Fourteen participants interviewed RePlay and one baseline in counterbalanced order, then rated conversation quality and responsiveness.Sessions lasted 78–253 seconds, with a median of 149 seconds.Source: §4.5 · p. 4
5. RESULTS
In simulated multi-turn interviews, RePlay substantially reduces latency relative to cascades while maintaining similar dialogue quality to the stronger baselines. Its exact-line selection and generalization to unseen intents remain weaker, while user-study preferences favor it over the fast cascade.
Simulated multi-turn evaluation
- 383 ms median latency made RePlay 6.8× and 3.2× faster than Cascades A and B, with similar Q to both and intent agreement near Cascade A.RePlay’s Q was 3.76 versus 3.90 and 3.77; intent agreement was 0.83 versus Cascade A’s 0.85.Source: §5 · p. 4
Ablation
- Combining D1 and D2 training performs best on every metric, raising R@1 from 0.23 to 0.35; D1-only training selects a line from the correct intent in 45% of turns.The training-data ablation compares D1 pretraining with D2 fine-tuning conditions.Source: Table 3 · p. 4, §5 · p. 4
Simulated multi-turn evaluation
- RePlay’s remaining gap to Cascade A is mainly exact-line choice, with R@1 of 0.35 versus 0.52.The reported result suggests RePlay often retrieves an appropriate response even when it differs from the judge’s preferred line.Source: §5 · p. 4
Ablation
- Unseen lines within seen intents retain near all-seen dialogue quality, while unseen intents lower intent agreement to 0.65 and R@10 to 0.35.The reported Q is 3.74 for unseen lines versus 3.76 for all-seen lines; the unseen-intent model still exceeds D1-only intent agreement and Q.Source: §5 · p. 4
User study
- Against Cascade C, participants preferred RePlay in 63% of ratings versus 12% (p=0.008); against Cascade B, preference was 46% versus 21% and not significant (p=0.50).Participants favored RePlay’s shorter waits and pace, but found Cascade B’s replies slightly more appropriate.Source: §5 · p. 4
6. CONCLUSION
RePlay retrieves and plays pre-recorded lines using the hidden state of a full-duplex speech model. It achieved low latency in simulated interviews and was preferred to a fast cascade in a small user study, while evaluation scope and exact-line accuracy remain limitations.
Conclusion
- RePlay selects pre-recorded lines from a full-duplex speech model’s hidden state, retaining its first 16 layers and replacing text and speech generation with turn-taking and retrieval heads.Probing identified the turn-start token feedback point as when the upcoming response becomes recoverable.Source: §6 · p. 4
- In simulated interviews, RePlay reached 383 ms median latency, 3 to 7 times faster than cascades of comparable dialogue quality, and participants preferred it over a fast cascade.It generalizes to new lines within seen intents without retraining, while unseen intents benefit from in-domain data and exact-line selection trails a large LLM.Source: §6 · p. 4
- Evaluation relies on synthetic training data, an LLM judge, and a small user study.The paper identifies retrieval with finer-grained response metadata as future work.Source: §6 · p. 4