Source-linked AI summary
PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models
Rajarshi Roy, Jonathan Raiman, Sang-gil Lee, Teodor-Dumitru Ene, Robert Kirby, Sungwon Kim, Jaehyeon Kim, Bryan Catanzaro
TL;DR
Existing duplex speech models generally use fixed voices and assistant roles, limiting structured and personalized interactions. PersonaPlex combines text role prompts with audio voice prompts, trains on synthetic dialogs and speech, and extends evaluation to multi-role customer service. It reports state-of-the-art role adherence, voice similarity, and dialog naturalness while maintaining duplex responsiveness and turn taking.
Problem
Existing duplex speech systems are generally limited to fixed voices and roles, restricting structured, role-driven, and personalized applications.
Method
PersonaPlex uses hybrid system prompts that combine textual role conditioning with audio voice examples, trained on synthetic dialogs and speech and evaluated with a multi-role service benchmark.
Results
PersonaPlex achieves state-of-the-art role adherence, voice similarity, and dialog naturalness while maintaining duplex responsiveness and turn-taking abilities.
Takeaways & Limitations
Hybrid prompt conditioning integrates role and voice control into duplex speech without altering the underlying architecture.
Abstract
from arXiv · showhide
Recent advances in duplex speech models have enabled natural, low-latency speech-to-speech interactions. However, existing models are restricted to a fixed role and voice, limiting their ability to support structured, role-driven real-world applications and personalized interactions. In this work, we introduce PersonaPlex, a duplex conversational speech model that incorporates hybrid system prompts, combining role conditioning with text prompts and voice cloning with speech samples. PersonaPlex is trained on a large-scale synthetic dataset of paired prompts and user-agent conversations, generated with open-source large language models (LLM) and text-to-speech (TTS) models. To evaluate role conditioning in real-world settings, we extend the Full-Duplex-Bench benchmark beyond a single assistant role to multi-role customer service scenarios. Experiments show that PersonaPlex achieves strong role-conditioned behavior, voice-conditioned speech, and natural conversational responsiveness, surpassing state-of-the-art duplex speech models and hybrid large language model-based speech systems in role adherence, speaker similarity, latency, and naturalness.
NVIDIA
The paper lists NVIDIA-affiliated authors and indexes the work under conversational speech, duplex spoken language, role conditioning, and voice cloning.
- The author contact list uses NVIDIA’s nvidia.com domain.
- The work is indexed by conversational speech model, duplex spoken language model, role conditioning, and voice cloning.
1. INTRODUCTION
Duplex speech systems provide natural, low-latency interaction but typically lack flexible role and voice conditioning. PersonaPlex addresses this gap with hybrid prompts, synthetic training data, and a multi-role customer-service benchmark.
- Existing duplex systems offer low-latency speech interaction but generally remain fixed to one voice and role, limiting structured applications.
- Voice-conditioned TTS and instruction-following LLMs provide relevant capabilities, but duplex speech systems have not fully combined them under latency constraints.
- PersonaPlex combines text-based role conditioning with audio-based voice cloning through hybrid system prompts.
- The paper adds 350 service questions across specific roles to the 400 questions in Full-Duplex-Bench and reports state-of-the-art role adherence, voice similarity, and dialog naturalness.
2. RELATED WORK
Prior systems trade off duplex responsiveness, voice flexibility, and role conditioning. PersonaPlex combines these capabilities within a duplex speech model to broaden real-world applicability.
- Cascaded ASR-LLM-TTS systems preserve reasoning and TTS naturalness but lose paralinguistic information, while streaming TTS mainly reduces latency and supports voice cloning.
- Half-duplex models retain paralinguistics and lower latency but rely on external turn-taking, cannot listen while speaking, and use fixed voices.
- Full-duplex models support natural turn taking and responsiveness but remain limited to fixed voices and assistant-style roles.
- PersonaPlex integrates duplex modeling, zero-shot voice cloning, and instruction-based role conditioning to address these limitations.
3. PERSONAPLEX
PersonaPlex uses a Moshi-based duplex architecture with hybrid text–audio prompts, synthetic dialogs and speech, and benchmarks designed to test naturalness, voice similarity, and service-role behavior.
- 3.1. Architecture: PersonaPlex receives user audio, agent text, and agent audio, while its Hybrid System Prompt combines textual role descriptions with audio voice examples.
- 3.1. Architecture: The text segment conditions the agent role, and a short speech sample in the voice segment enables subsequent utterances in the cloned voice.
- 3.1. Architecture: Voice prompts can precede text prompts to enable inference prefilling and reduce latency when zero-shot voice cloning is unnecessary.
- 3.2.1. Dialog Transcripts and Text Prompt Generation: Synthetic service dialogs are generated hierarchically from domains, scenarios, descriptions, and two-speaker transcripts using Qwen-3-32B and GPT-OSS-120B.
- 3.2.2. Dialog Speech and Voice Prompt Generation: Training speech uses 26,296 single-speaker samples and multispeaker TTS for service dialogs, with separate voice prompting and turn-taking simulation procedures.
- 3.3. Service-Duplex-Bench: Service-Duplex-Bench contains 50 service-role scenarios with 7 questions each, probing context adherence, request handling, and customer-rudeness management.
4. EXPERIMENTS AND RESULTS
PersonaPlex is evaluated for voice control, naturalness, interactivity, role adherence, and instruction following across duplex speech benchmarks. Results show strong voice similarity, state-of-the-art human-like interactivity, and improved role adherence with more synthetic data.
- Training Data: The evaluation uses 1,840 hours of customer-service dialogs across 105,410 dialogs and 410 hours of general question-answering dialogs across 39,322 dialogs.
- Dialog Naturalness and Voice Cloning: Human evaluators rate dialog naturalness using DMOS scores from 1 to 5 across randomized audio samples from five models.The study polls 202 evaluators for Service-Duplex-Bench and 152 for Full-Duplex-Bench User Interruption.
- Dialog Naturalness and Voice Cloning: PersonaPlex achieves consistently higher speaker similarity than baseline models on Full-Duplex-Bench, indicating effective voice control.Similarity is measured as cosine similarity between voice-prompt and synthesized-agent speech embeddings using WavLM-TDNN.
- Full-Duplex-Bench & Service-Duplex-Bench: PersonaPlex shows state-of-the-art performance on metrics related to human-like user interactivity across the evaluated duplex benchmarks.
- Full-Duplex-Bench & Service-Duplex-Bench: On Service-Duplex-Bench, PersonaPlex outperforms all models except Gemini Live on role adherence and instruction following.
- Dataset Scale: Adding synthetic data greatly enhances voice cloning and role adherence versus the Moshi baseline, with role adherence improving steadily as Service-Duplex-Bench data increases.Strong Full-Duplex-Bench performance is achieved with limited data.
5. CONCLUSION
PersonaPlex integrates zero-shot voice cloning and fine-grained role conditioning into a full-duplex speech-to-speech model through hybrid text–audio prompts. It outperforms prior duplex baselines on speaker similarity, role adherence, and dialog naturalness.
- Contribution: PersonaPlex enables zero-shot voice cloning and fine-grained role conditioning through hybrid text–audio system prompts.
- Contribution: Conditioning integrates into duplex speech systems without altering their underlying architecture.
- Results: PersonaPlex outperforms prior duplex baselines in speaker similarity, role adherence, and dialog naturalness.
- Implications: The authors describe hybrid prompt conditioning as a scalable path toward personalized, role-conditioned conversational agents.
A. RELEASED CHECKPOINT
The released PersonaPlex checkpoint adds real conversational data and changes the synthetic voice-generation pipeline while retaining the paper’s hybrid prompting and role-conditioning architecture.
- Real Conversational Data: The released checkpoint adds 7,303 Fisher English conversations totaling 1,217 hours to improve natural backchanneling, expressions, and emotional responses.
- Real Conversational Data: Added conversations receive minimal, topic-specific, or highly detailed prompts to balance generalization capability with instruction-following precision.
- Synthetic Voice Generation: Synthetic dialogs use TortoiseTTS voices rather than real voice datasets, with pitch and formant augmentation to cover varied timbres.
- Synthetic Voice Generation: The released pipeline uses ChatterboxTTS for both assistant and service scenarios, replacing the mixed Dia/Chatterbox approach.
A.1. Evaluation of Released Checkpoint
The released checkpoint improves naturalness and conversational dynamics while preserving hybrid prompting and role conditioning. A separate evaluation finds competitive naturalness relative to baseline models.
- Released Checkpoint: The released checkpoint demonstrates improved naturalness and conversational dynamics while maintaining hybrid prompting and role-conditioning capabilities.It shows significantly increased backchannel frequency and improved pause handling.
- Evaluation: The released checkpoint maintains competitive naturalness relative to baseline models on Full-Duplex-Bench User Interruption.This evaluation uses a separate annotator pool.
- Evaluation: Table 7 reports dialog naturalness MOS with 95% confidence intervals, using scores that are relative within this study and not directly comparable to Table 1.