PaperlayerView exact revision on arXiv

Source-linked AI summary

CharDuplex: Building Character-Consistent Full-Duplex Spoken Dialogue Models

Donghang Wu, Yisi Liu, Chen Chen, Hexin Liu, Eng Siong Chng

arXiv:2609.34461v1eess.AS

TL;DR

Full-duplex models enable concurrent speech, but natural conversation also requires responses that reflect a character and remain consistent across evolving interactions. CharDuplex combines full-duplex adaptation, character-conditioned training data, and response-dependent reinforcement learning; it achieves the highest average SpeechRole-Eval score among evaluated open-source models while remaining competitive with closed-source systems.

  • Problem

    Full-duplex models enable concurrent listening and speaking, but character-consistent dialogue must also reflect a character’s knowledge and personality and persist across multi-turn exchanges.

    Source: §1 · p. 1

  • Method

    CharDuplex adapts GLM-4-Voice for full-duplex conversation, uses automated character-conditioned data for SFT, and applies FDGym with an LLM user simulator for interactive reinforcement learning.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    CharDuplex achieves the highest average SpeechRole-Eval score among evaluated open-source models while remaining competitive with closed-source systems.

    Source: §6 · p. 9

  • Takeaways & Limitations

    FDGym improves all four character-related metrics while general speech intelligence and full-duplex interaction performance remain broadly stable.

    Source: §6 · p. 9

Abstract

from arXiv · show

Full-duplex speech models are moving voice interaction beyond conventional turn-taking, yet natural conversation is shaped not only by when an agent speaks, but also by how it behaves as a conversational character. We present CharDuplex, a character-driven full-duplex speech model that combines real-time spoken interaction with persona-conditioned behavior. We first adapt GLM-4-Voice to an always-on dual-stream architecture and train the model for full-duplex conversation. Then a fully automated pipeline constructs character-conditioned dialogue data from open-source character descriptions for character-conditioned supervised fine-tuning. The model is further refined with the proposed FDGym, where an LLM-simulated user dynamically interacts with the model, enabling reinforcement learning over evolving multi-turn interactions. On SpeechRole-Eval, CharDuplex achieves the highest average score among the evaluated open-source models, while remaining competitive with closed-source systems. It also demonstrates competitive general speech intelligence and strong full-duplex interaction capabilities. CharDuplex demonstrates a practical training recipe for building full-duplex voice assistants that are not only interactive, but also character-consistent.

1 INTRODUCTION

Full-duplex interaction enables concurrent listening and speaking, but natural dialogue also depends on responses that consistently reflect a character’s knowledge, personality, and perspective. CharDuplex combines character-conditioned data and supervised training with response-dependent interactive reinforcement learning to strengthen that consistency across evolving conversations.

1 INTRODUCTION

  • Character-consistent dialogue must respect a character’s knowledge, personality, and background while remaining coherent through follow-ups, corrections, and topic shifts.Responses should remain grounded in the prescribed character and consistent with the dialogue history.Source: §1 · p. 1
  • Training data must connect character descriptions to multi-turn spoken conversations, but fixed supervised continuations do not capture response-dependent user reactions or interruptions.Users may question a claim, request an explanation, follow up on the model’s actual reply, or interrupt while it speaks.Source: §1 · pp. 1–2
  • CharDuplex outperforms full-duplex SDLMs, especially PersonaPlex, across all four SpeechRole-Eval dimensions while maintaining strong general speech intelligence and full-duplex interaction capabilities.Its training combines full-duplex adaptation, character-conditioned SFT, and FDGym reinforcement learning.Source: §1 · p. 2
  • The paper introduces an automated character-conditioned dialogue-data pipeline and FDGym, where online simulated user speech enables closed-loop multi-turn reinforcement learning.Its stated contributions also include combining character-consistent response generation with general speech intelligence and full-duplex interaction capabilities.Source: §1 · p. 2

2 RELATED WORK

Prior work advances full-duplex interaction, spoken character conditioning, and reinforcement learning, but FDGym targets character consistency in evolving conversations by conditioning simulated user turns on the model’s unfolding replies.

2.1 FULL-DUPLEX SPOKEN DIALOGUE MODELS

  • Full-duplex models support concurrent listening and speaking, while prior work explores parallel speech streams, alternative architectures, and temporally structured or latent reasoning.CharDuplex focuses on character consistency while preserving general interaction and reasoning capabilities.Source: §2.1 · p. 2

2.2 CHARACTER-CONDITIONED SPOKEN INTERACTION

  • Spoken character modeling spans linguistic personality and vocal traits, role-playing data, text-and-voice-conditioned full-duplex dialogue, and broader persona steerability.CharDuplex focuses on multi-turn character-consistent responses supported by automated data construction and interactive post-training.Source: §2.2 · p. 2

2.3 REINFORCEMENT LEARNING FOR FULL-DUPLEX SDLM

  • Earlier full-duplex RL methods primarily use prerecorded speech or predetermined contexts, whereas FDGym generates simulated user turns in response to the model’s own behavior.This lets character consistency be optimized under conversational contexts that emerge from the model’s replies.Source: §2.3 · p. 3

3 METHOD

CharDuplex combines GLM-4-Voice full-duplex adaptation, character-conditioned speech supervision, and interactive reinforcement learning. Its method builds synchronized character dialogue data and trains over response-dependent user interactions.

3.1 CHARACTER-CONDITIONED DATA CONSTRUCTION

  • An automated generator–verifier pipeline creates character-conditioned dialogues, controls synthesized-speech quality, and aligns speech into synchronized two-channel audio with interaction events.Verification checks character grounding and turn-to-turn consistency; alignment and scheduling encode pauses, backchannels, and interruptions.Source: §3.1 · pp. 3–4

3.2 FULL-DUPLEX MODEL ARCHITECTURE

  • Two synchronous, always-on streams combine incoming user audio and response tokens framewise over a shared causal backbone, with autoregressive replies and continuously supplied user speech.Turn-level alignment lets the model end a response on interruption without waiting for its remaining text or speech.Source: §3.2 · pp. 4–5

3.3 SUPERVISED FINE-TUNING

  • Supervised training proceeds through general full-duplex training, character-conditioned SFT, and separate speech-generation training.The first two stages train the response stream conditioned on character context, incoming speech, and preceding response tokens; the speech module is then trained with the backbone frozen.Source: §3.3 · p. 5

3.4 REAL-TIME INTERACTIVE REINFORCEMENT LEARNING

  • FDGym samples response-dependent user reactions on 160-ms ticks, scores interaction, character adherence, response timing, and yielding, then updates the policy with GRPO.The simulator receives only the portion of the model response available by each tick, allowing it to react or interrupt as the reply unfolds.Source: §3.4 · pp. 5–6, Figure 3 · p. 6

4 EXPERIMENTAL SETUP

The experiments use a character-conditioned corpus and staged training setup, then evaluate role-playing, general speech intelligence, and full-duplex conversational behavior.

4.1 DATA AND TRAINING DETAILS

  • The character-conditioned corpus contains 70 characters, 40,608 dialogues, and 756.04 hours of speech, with training character descriptions disjoint from evaluation descriptions.RL starts from CharDuplex-SFT and samples 32 trajectories per policy iteration.Source: §4.1 · p. 7

4.2 BENCHMARKS AND EVALUATION

  • SpeechRole-Eval measures instruction adherence, conversational coherence, personality consistency, and knowledge consistency in role-conditioned dialogues.The evaluation dialogues use their associated character descriptions.Source: §4.2 · pp. 7–8
  • Table 2 reports results across general speech benchmarks, marks full-duplex support, and identifies the best score separately for full-duplex and non-full-duplex models.The table caption identifies LLaMA-Questions, WebQuestions, OpenBookQA, AdvBench, AlpacaEval, CommonEval, and WildVoice among its columns.Source: Table 2 · p. 8
  • General speech intelligence is assessed across spoken QA, reasoning, open-ended response quality, and safety, while Full-Duplex-Bench evaluates turn-taking and interruption handling.The general-speech suite includes VoiceBench datasets plus LLaMA-Questions and WebQuestions.Source: §4.2 · p. 8

5 RESULTS

CharDuplex leads evaluated open-source models on character consistency while remaining competitive with closed-source systems. Its character gains coexist with broadly stable general speech intelligence and full-duplex interaction performance.

5.1 CHARACTER-RELATED CAPABILITIES

  • CharDuplex scores 0.785 average on SpeechRole-Eval, +0.134 absolute over PersonaPlex, highest among evaluated open-source models.It leads on Conversational Coherence, Personality Consistency, and Knowledge Consistency, and ties for best Instruction Adherence.Source: §5.1 · p. 8
  • CharDuplex remains competitive with closed-source systems; FDGym improves all four SpeechRole-Eval dimensions over an already strong CharDuplex-SFT baseline.It exceeds Grok on four dimensions and Gemini on Instruction Adherence and Conversational Coherence, while Gemini is stronger on Personality and Knowledge Consistency.Source: §5.1 · p. 8

5.2 SPEECH INTELLIGENCE

  • After FDGym, eight of ten general speech-intelligence metrics change by no more than 1.11% relative; SD-QA improves 3.80% and BBH 2.40%.CharDuplex also exceeds its GLM-4-Voice foundation on all five factual and multiple-choice QA tasks.Source: §5.2 · p. 9

5.3 FULL-DUPLEX INTERACTION CAPABILITY

  • CharDuplex shows strong backchannel behavior and interruption handling, while nine of ten Full-Duplex-Bench metrics change by at most 2.26% relative after FDGym.The comparison is reported in Table 3; its caption defines TOR, ICC, and JSD and marks whether higher or lower values are better.Source: §5.3 · p. 9, Table 3 · p. 9

6 CONCLUSION

CharDuplex combines character-consistent response generation with general speech intelligence and full-duplex interaction. It achieves the highest average SpeechRole-Eval score among evaluated open-source models, while FDGym improves character-related metrics and the other capabilities remain broadly stable.

6 CONCLUSION

  • CharDuplex combines character-consistent response generation with general speech intelligence and full-duplex interaction capabilities.The model integrates an always-on dual-stream architecture, automated character-conditioned data construction, and FDGym for response-dependent multi-turn reinforcement learning with a simulated user.Source: §6 · p. 9
  • CharDuplex achieves the highest average SpeechRole-Eval score among evaluated open-source models while remaining competitive with closed-source systems.FDGym further improves all four character-related metrics.Source: §6 · p. 9
  • General speech intelligence and full-duplex interaction performance remain broadly stable as character consistency is strengthened.The paper concludes that character consistency can be strengthened without sacrificing the general capabilities required for full-duplex spoken interaction.Source: §6 · p. 9

AI USE STATEMENT

The authors report AI assistance in manuscript polishing and several research-method components. The character-conditioned data pipeline generates dialogue, verifies and corrects it, and annotates interaction events.

AI USE STATEMENT

  • AI tools assisted with manuscript polishing and were also used for dialogue generation and verification, user simulation, reward computation, and applicable automated evaluation.The authors state that they developed and verified the research design, experiments, analysis, and conclusions and take responsibility for the manuscript and results.Source: AI USE STATEMENT · p. 10

A CHARACTER-CONDITIONED DATA CONSTRUCTION

  • The automated generator–verifier pipeline creates dialogue from character descriptions and sampled conversational scenarios, conditioning replies on character traits and preceding dialogue.The character description remains the system prompt and is excluded from the text passed to TTS.Source: A CHARACTER-CONDITIONED DATA… · p. 13
  • Two verifier prompts check factual support and cross-turn consistency, with local edits for isolated errors and conversation regeneration when errors invalidate the scenario or affect much of it.The factual-grounding verifier checks claims against character background or established dialogue and returns localized repairs when possible.Source: A CHARACTER-CONDITIONED DATA… · pp. 13–14, Figure 5 · p. 14
  • After dialogue verification, candidate event locations are identified and an LLM selects contextually suitable backchannels, thinking pauses, and interruptions for verification.Event annotations remain separate from dialogue text, allowing audio construction to control pauses and overlaps without sending timing instructions to TTS.Source: A CHARACTER-CONDITIONED DATA… · p. 14

B.1 GENERAL FULL-DUPLEX SFT DATA

The general full-duplex training corpus combines speech-continuation, instruction-following QA, and ASR-QA data. Separately, character-conditioned speech data are generated, verified, synthesized, and aligned from character descriptions.

B.1 GENERAL FULL-DUPLEX SFT DATA

  • The general training corpus contains approximately 530K hours of speech-continuation data, 70K hours of instruction-following QA data, and 20K hours of ASR-QA data.The setup follows the corpus and construction procedure of Wu et al. (2026) and Artificial Analysis (2026).Source: §B.1 · p. 14
  • Speech-continuation data convert continuous text into alternating user–assistant turns; instruction-following QA includes generated single- and multi-turn conversations, while ASR-QA uses questions grounded in real-speech transcriptions.The ASR-QA source is NeMo ASRSET.Source: §B.1 · p. 14
  • Synthetic subsets use GPT-OSS-120B, Qwen2.5-72B-Instruct, and Llama-3.1-70B-Instruct for text generation, and Chatterbox, Magpie-TTS, and MoonCast for speech synthesis.Voice references are drawn from LibriTTS, YODAS, and HiFi-TTS, yielding more than 100K segments from over 20K speakers.Source: §B.1 · p. 15
  • The character-conditioned corpus contains 70 characters and 40,608 dialogues totaling 756.043 hours of speech; training character descriptions are disjoint from evaluation descriptions.The pipeline performs dialogue generation and event annotation, content verification and correction, event validation, speech synthesis, and word-level alignment. Training applies SpecAugment to user speech features and mixes background noise into the user stream.Source: §B.1 · p. 15

C TRAINING DETAILS

CharDuplex is trained through supervised full-duplex and character-conditioned stages, then refined with interactive rollouts and GRPO. FDGym uses a response-dependent simulator and separate interaction and character judges.

C TRAINING DETAILS

  • Supervised training initializes from GLM-4-Voice and uses general speech data followed by character-conditioned SFT.The general corpus includes 530K hours of speech continuation, 70K hours of instruction-following QA, and 20K hours of ASR-QA data.Source: C TRAINING DETAILS · p. 16
  • Each RL iteration samples 32 trajectories across four initial dialogue conditions, with each rollout limited to 260 ticks or 41.6 seconds.Only speech estimated to be audible by the current tick is shown to the simulator.Source: C TRAINING DETAILS · p. 16
  • The event-specific verifier checks backchannels and pauses for local pragmatic fit and validates interruptions against speech audible before interruption.This prevents later dialogue from relying on speech discarded by an interruption.Source: Figure 8 · p. 17

D PROMPTS USED IN FDGYM

  • FDGym’s simulator generates subsequent user utterances from dialogue history and the progressively available CharDuplex response.The user turns evolve in response to the model’s unfolding replies.Source: D PROMPTS USED IN FDGYM · p. 17
  • Two independent LLM judges score interaction from dialogue and timing statistics and character consistency from the character-conditioned prompt.The interaction judge evaluates coherence, responsiveness, and appropriateness independently of character consistency.Source: D PROMPTS USED IN FDGYM · p. 17

E GRPO FOR CHARDUPLEX

  • GRPO collects trajectories under a fixed old-policy snapshot, then updates the policy using a clipped loss over each complete response stream.The token context includes the character prompt, available user speech, and preceding model outputs; one optimization pass precedes the next rollout.Source: E GRPO FOR CHARDUPLEX · pp. 17–18

F COMPARISON BEFORE AND AFTER INTERACTIVE POST-TRAINING

After FDGym post-training, CharDuplex improves across SpeechRole-Eval metrics while largely preserving the general speech intelligence and full-duplex interaction capabilities established during supervised training.

F COMPARISON BEFORE AND AFTER INTERACTIVE POST-TRAINING

  • FDGym improves every SpeechRole-Eval metric, while general speech intelligence and full-duplex interaction remain largely preserved.The reported SpeechRole-Eval gains span instruction adherence, conversational coherence, personality consistency, and knowledge consistency; Backchannel TOR shows a larger relative decrease.Source: F COMPARISON BEFORE AND AFTER… · pp. 18–19

G ABLATION STUDIES AND ANALYSIS

The ablations support retaining both deterministic timing rewards during RL and verification in dialogue-data construction.

Effect of Timing Rewards

  • Removing timing rewards yields no meaningful SpeechRole-Eval improvement and slightly decreases full-duplex turn-taking performance.The deterministic rewards are retained because they help preserve SFT interaction behavior and add negligible computational cost.Source: G ABLATION STUDIES AND ANALYSIS · p. 19

Effect of Dialogue Verification

  • 93.7% of dialogues pass after generator–verifier processing, compared with 65.6% without verification.The full pipeline’s initial verification pass rate is 68.8%; the authors retain verification despite its added construction cost.Source: G ABLATION STUDIES AND ANALYSIS · p. 20
Loading 2609.34461v1…