PaperlayerView exact revision on arXiv

Source-linked AI summary

PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models

Rajarshi Roy, Jonathan Raiman, Sang-gil Lee, Teodor-Dumitru Ene, Robert Kirby, Sungwon Kim, Jaehyeon Kim, Bryan Catanzaro

arXiv:2602.06053v1cs.CL

TL;DR

Duplex speech models are generally limited to fixed roles and voices, despite the need for structured and personalized interactions. PersonaPlex combines text-based role prompts with audio voice examples, and outperforms prior duplex baselines in speaker similarity, role adherence, and dialog naturalness.

  • Problem

    Duplex systems generally use fixed roles and voices, while integrating role and voice conditioning under latency constraints remains challenging for structured and personalized applications.

    Source: §1 · p. 1

  • Method

    PersonaPlex uses hybrid system prompts that combine textual role descriptions with audio voice examples in a duplex conversational model.

    Source: §3.1 · p. 2

  • Results

    PersonaPlex outperforms prior duplex baselines in speaker similarity, role adherence, and dialog naturalness, and on Service-Duplex-Bench exceeds all models except Gemini Live.

    Source: §5 · p. 4, §4.2 · p. 4

  • Takeaways & Limitations

    Hybrid prompt conditioning offers a scalable path toward personalized, role-conditioned conversational agents.

    Source: §5 · p. 4

Abstract

from arXiv · show

Recent advances in duplex speech models have enabled natural, low-latency speech-to-speech interactions. However, existing models are restricted to a fixed role and voice, limiting their ability to support structured, role-driven real-world applications and personalized interactions. In this work, we introduce PersonaPlex, a duplex conversational speech model that incorporates hybrid system prompts, combining role conditioning with text prompts and voice cloning with speech samples. PersonaPlex is trained on a large-scale synthetic dataset of paired prompts and user-agent conversations, generated with open-source large language models (LLM) and text-to-speech (TTS) models. To evaluate role conditioning in real-world settings, we extend the Full-Duplex-Bench benchmark beyond a single assistant role to multi-role customer service scenarios. Experiments show that PersonaPlex achieves strong role-conditioned behavior, voice-conditioned speech, and natural conversational responsiveness, surpassing state-of-the-art duplex speech models and hybrid large language model-based speech systems in role adherence, speaker similarity, latency, and naturalness.

1. INTRODUCTION

PersonaPlex addresses the fixed-role, fixed-voice limitations of duplex speech with hybrid role and voice conditioning, supported by synthetic training data and a multi-role customer-service benchmark extension. The paper reports state-of-the-art role adherence, voice similarity, and dialog naturalness while maintaining duplex responsiveness and turn-taking.

1. INTRODUCTION

  • Existing duplex systems generally support only a fixed voice and role, limiting their use in structured applications such as customer service, multi-character interactions, and personalized assistants.Although voice-conditioned TTS and instruction-following LLMs have advanced, latency constraints and coupled speech-text dynamics make combining role and voice conditioning in duplex systems challenging.Source: §1 · p. 1
  • PersonaPlex combines text-based role conditioning with audio-based voice cloning in a full-duplex speech model, enabling zero-shot voice cloning and fine-grained role control.The work also introduces a large-scale synthetic corpus of paired prompts and user-agent conversations.Source: §1 · p. 1
  • PersonaPlex achieves state-of-the-art role adherence, voice similarity, and dialog naturalness while maintaining duplex responsiveness and turn-taking on Full-Duplex-Bench and Service-Duplex-Bench.Service-Duplex-Bench adds 350 customer-service evaluation questions, each tied to a service role, to Full-Duplex-Bench's 400 questions.Source: §1 · p. 1

2. RELATED WORK

Prior conversational speech systems trade off latency, paralinguistic information, interaction style, role control, and voice flexibility. PersonaPlex combines duplex modeling, voice cloning, and role conditioning, while its benchmark extension targets real-world customer-service roles.

RELATED WORK

  • Cascaded ASR-LLM-TTS systems lose paralinguistic information, while streaming TTS reduces latency and supports voice cloning, and HumeAI encodes expressive information in text.Half-duplex approaches stream speech tokens to lower latency and retain paralinguistics, but rely on external turn-taking, cannot listen while speaking, and use fixed voices.Source: §2 · pp. 1–2
  • Full-duplex models enable responsive real-time speech interaction but retain fixed voices and assistant roles, while commercial systems condition roles but use fixed voices.PersonaPlex integrates duplex modeling, zero-shot voice cloning, and instruction-based role conditioning to address these limitations.Source: §2 · p. 2

Duplex Speech Model Evaluation

  • Most conversational speech benchmarks [15] [17] [18] evaluate single-turn, half-duplex responses, whereas Full-Duplex-Bench streams user audio and evaluates generated audio and conversational dynamics.Its tasks assess knowledge and reasoning through QA, as well as turn-taking, responsiveness, interruption, and backchannel handling.Source: §2 · p. 2
  • PersonaPlex extends Full-Duplex-Bench with customer-service scenarios to evaluate fine-grained role conditioning in real-world applications.Source: §2 · p. 2

3. PERSONAPLEX

PersonaPlex combines text-based role conditioning and speech-sample voice prompting in a duplex model, trained on synthetic conversations and evaluated with single-turn probes across service roles.

3.1. Architecture

  • PersonaPlex follows the Moshi architecture and uses a Hybrid System Prompt to combine textual role descriptions with audio voice examples for duplex speech generation.The prompt concatenates a role-conditioning text segment with a voice sample segment; subsequent agent utterances use the prompted voice, enabling zero-shot voice cloning. Voice-first ordering supports inference prefilling when cloning is not required, reducing latency.Source: §3.1 · p. 2

3.2.1. Dialog Transcripts and Text Prompt Generation

  • Synthetic service dialogues are generated hierarchically from sampled domains and scenarios, while a separate question-answering dataset uses a fixed wise-and-friendly-teacher role.Qwen-3-32B and GPT-OSS-120B generate transcripts; service training scenarios are distinct from evaluation scenarios.Source: §3.2.1 · pp. 2–3

3.2.2. Dialog Speech and Voice Prompt Generation

  • The speech data pipeline uses 26,296 voice samples and Dia for multispeaker service dialogue audio, while Chatterbox TTS generates question-answering speech with stitched turns.A separate set of 2,630 voice samples is reserved for speaker-similarity measurements; silence padding can simulate turn-taking, and negative-duration silence can simulate interruptions.Source: §3.2.2 · p. 3

3.3. Service-Duplex-Bench

  • Service-Duplex-Bench extends Full-Duplex-Bench with 50 service-role scenarios and seven single-turn questions per scenario probing context-specific capabilities.Probes test proper-noun recall, context adherence, handling unfulfillable requests, and managing customer rudeness; the authors plan to release the dataset.Source: §3.3 · pp. 3–4

4. EXPERIMENTS AND RESULTS

PersonaPlex is evaluated for voice similarity, conversational interactivity, and service-role adherence, with results showing strong benchmark performance and steadily improving role adherence as training data increases.

4. EXPERIMENTS AND RESULTS

  • PersonaPlex is fine-tuned from Moshi on synthetic conversations comprising 1,840 hours of customer-service dialogs and 410 hours of general question-answering dialogs.The dataset contains 105,410 customer-service dialogs and 39,322 question-answering dialogs.Source: §4 · p. 4

4.1. Dialog Naturalness and Voice Cloning

  • Human evaluators rate eight randomized audio samples from five models on a 1-to-5 DMOS scale to assess naturalness.The evaluation includes 1,616 Service-Duplex-Bench samples and 1,216 Full-Duplex-Bench User Interruption samples.Source: §4.1 · p. 4
  • PersonaPlex achieves consistently higher speaker similarity than baseline models, demonstrating effective voice control.Similarity is measured as cosine similarity between provided voice-prompt and synthesized-agent-speech embeddings using WavLM-TDNN.Source: §4.1 · p. 4

4.2. Full-Duplex-Bench & Service-Duplex-Bench

  • PersonaPlex achieves state-of-the-art human-like interactivity on Full-Duplex-Bench and outperforms all models except Gemini Live on Service-Duplex-Bench role adherence and instruction following.The Service-Duplex-Bench comparison is reported in Table 4.Source: §4.2 · p. 4, Table 4 · p. 4
  • PersonaPlex combines full-duplex models’ human-like interactivity with non-duplex architectures’ instruction-following ability.This conclusion incorporates the human-rated naturalness evaluation reported in Table 1.Source: §4.2 · p. 4

4.3. Dataset Scale

  • Adding synthetic data improves voice cloning and role adherence over the Moshi baseline; role adherence on Service-Duplex-Bench improves steadily with more data.Full-Duplex-Bench performance is strong even with limited data.Source: §4.3 · p. 4

5. CONCLUSION

PersonaPlex enables zero-shot voice cloning and fine-grained role conditioning through hybrid text–audio prompts, outperforming prior duplex baselines and reaching naturalness comparable to closed commercial systems.

5. CONCLUSION

  • PersonaPlex enables zero-shot voice cloning and fine-grained role conditioning through hybrid text–audio system prompts.Source: §5 · p. 4
  • Hybrid conditioning integrates into duplex speech systems without altering their underlying architecture.Source: §5 · p. 4
  • The findings suggest hybrid prompt conditioning offers a scalable path toward personalized, role-conditioned conversational agents.Source: §5 · p. 4
  • Future work will explore post-training alignment and integration with external tools.Source: §5 · p. 4
  • PersonaPlex outperforms prior duplex baselines in speaker similarity, role adherence, and dialog naturalness, with naturalness comparable to closed commercial systems.The released-checkpoint Full Duplex Bench results are reported in Table 6.Source: §5 · p. 4, Table 6 · p. 5

A. RELEASED CHECKPOINT

The released PersonaPlex checkpoint adds Fisher conversations with prompts of varying detail and standardizes synthetic voice generation. The release reports improved speaker similarity over the paper’s Table 1 result.

A. RELEASED CHECKPOINT

  • The checkpoint adds 7,303 Fisher English conversations totaling 1,217 hours to improve natural backchanneling, expressions, and emotional responses.GPT-OSS-120B annotated conversations with prompts at varying detail levels to balance generalization and instruction-following precision.Source: A. RELEASED CHECKPOINT · p. 5
  • Fisher conversation prompts range from minimal and topic-specific to highly detailed, including biographical context and preferences.Examples progress from a general invitation to converse, to a specified discussion topic, to a reflective scenario with personal history and preferences.Source: A. RELEASED CHECKPOINT · p. 5
  • 0.65 speaker similarity was reported for the released model, compared with 0.57 in Table 1.All synthetic dialogs use ChatterboxTTS, replacing the mixed Dia/Chatterbox approach; the release attributes the improvement to Chatterbox’s superior speaker consistency.Source: A. RELEASED CHECKPOINT · p. 5
  • For privacy, synthetic dialogs use TortoiseTTS voices augmented with Praat pitch and formant changes rather than real voice datasets.ChatterboxTTS generates speech for both assistant and service scenarios in the released checkpoint.Source: A. RELEASED CHECKPOINT · p. 5

A.1. Evaluation of Released Checkpoint

The released checkpoint improves naturalness and conversational dynamics while retaining the hybrid prompting architecture and role-conditioning capabilities. On Full-Duplex-Bench’s User Interruption category, it maintains competitive naturalness relative to baseline models.

A.1. Evaluation of Released Checkpoint

  • The released checkpoint shows increased backchannel frequency and improved pause handling compared with the experimental setup.These changes are reported as evidence of improved naturalness and conversational dynamics.Source: §A.1 · p. 5
  • The released checkpoint retains the core hybrid prompting architecture and role-conditioning capabilities.Source: §A.1 · p. 5
  • In the User Interruption category, the released checkpoint maintains competitive naturalness relative to baseline models.The DMOS evaluation used a separate annotator pool; Table 7 reports dialog naturalness MOS with 95% confidence intervals, relative within this study and not directly comparable to Table 1.Source: Table 7 · p. 5, §A.1 · p. 5

Related papers

Loading 2602.06053v1…