PaperlayerView exact revision on arXiv

Source-linked AI summary

DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues

Takyoung Kim, Kang-wook Kim, Sang Hoon Woo, Julia Hirschberg, Gunhee Kim, Dilek Hakkani-Tür

arXiv:2607.26178v1cs.CL

TL;DR

Current full-duplex models do not adapt turn-taking to scenario-specific social roles, and existing data or synthesis methods lack human validation. DuplexGen calibrates LLM turn-taking predictions with a small set of human annotations, producing substantially better alignment with human judgments and scenario-specific behaviors preferred by humans.

  • Problem

    Current full-duplex models lack scenario-sensitive turn-taking, while existing corpora and synthesis methods do not validate behavior against human preferences in target scenarios.

  • Method

    DUPLEXGEN identifies candidate turn-taking slots and calibrates LLM predictions with a small set of scenario-specific human annotations during dialogue synthesis.

  • Results

    Small-budget calibration substantially improves alignment with human judgments over uncalibrated prompting or large generic corpora, while trained full-duplex models acquire human-preferred scenario-specific behaviors.

  • Takeaways & Limitations

    Human calibration is a practical and scalable path toward socially appropriate, scenario-specific turn-taking in spoken AI.

  • Takeaways & Limitations

    DUPLEXGEN uses incrementally revealed transcripts rather than prosody, pauses, intonation, speech rate, overlap, or visual cues, and text–audio preference agreement remains unestablished.

Abstract

from arXiv · show

Turn-taking is a central component of full-duplex interaction. Which turn-taking behaviors are appropriate varies with the scenario, yet current models apply a single norm regardless of context. This limitation originates in their training data: human-human speech corpora capture natural timing phenomena but provide little role grounding or scenario-specific norms, while heuristic or prompted synthesis methods inject turn-taking behaviors without basing them on human preferences. We introduce DuplexGen, a framework for generating dialogues with scenario-adaptive turn-taking by calibrating LLM predictions against a small set of slot-level human preference annotations. In six cooperative and competitive tasks, human turn-taking preferences differ systematically, and DuplexGen aligns substantially more closely with those preferences than uncalibrated prompting or training solely on generic human-human data; a full-duplex model trained on DuplexGen-generated data exhibits distinctive, human-preferred turn-taking behaviors. These results show that human calibration, not corpus scale or prompt design alone, is what allows turn-taking synthesis to be scenario-specific.

1 Introduction

DUPLEXGEN addresses the lack of scenario-sensitive turn-taking in full-duplex language models by calibrating LLM predictions with a small set of slot-level human annotations. Across six cooperative and competitive tasks, this calibration improves alignment with human preferences and yields models exhibiting scenario-specific behaviors.

  • 1 Introduction: Turn-taking strategies depend on social context and role structure, but current full-duplex language models do not reflect this scenario sensitivity.Different roles call for different behaviors, such as patient listening for tutors and more frequent interjections for negotiators.
  • 1 Introduction: Generic human conversation corpora capture natural turn-taking at scale but lack defined task roles and scenario structure.They commonly pair unfamiliar speakers on assigned topics, limiting their grounding in role-specific interaction norms.
  • 1 Introduction: DUPLEXGEN identifies candidate turn-taking slots and calibrates LLM action distributions with a small set of per-scenario human annotations.The framework converts text dialogue into spoken-style transcripts and predicts [LISTEN], [BACKCHANNEL], and [TAKE_FLOOR] actions.
  • 1 Introduction: DUPLEXGEN is evaluated on six cooperative and competitive tasks spanning tutoring, planning, interview, negotiation, persuasion, and social chat.The task set provides varied goal structures and relational dynamics for studying scenario-dependent timing preferences.
  • 1 Introduction: Human turn-taking preferences differ systematically across scenarios, while calibrated models align substantially better with human judgments than prompt-only generation or generic human–human training.These findings support scenario-adaptive modeling and identify human calibration as the key ingredient in scenario-specific synthesis.
  • 1 Introduction: A full-duplex model trained on DUPLEXGEN-generated data exhibits distinctive, scenario-specific turn-taking behaviors preferred by humans.This demonstrates that the synthesized corpus can transfer human-preferred timing patterns to a full-duplex model.

2 Related Work

Prior work explains conversational timing and provides speech resources or synthesis methods, but does not establish which listener action is socially appropriate for a given scenario. DUPLEXGEN addresses this gap through human-calibrated, scenario-adaptive turn-taking synthesis for full-duplex dialogue data.

  • Turn-taking foundations: Turn-taking research models conversation through transition points, anticipatory response preparation, and acoustic, visual, and incremental language cues.These approaches often target coordination, latency reduction, or turn-boundary prediction.
  • Scenario-specific norms: Existing timing models do not determine whether silence, backchanneling, or floor-taking is socially appropriate across assistant roles and scenarios.The same partial utterance may call for different listener actions when the assistant is a tutor, planner, interviewer, or negotiation partner.
  • Existing data resources: Generic human–human corpora provide natural timing phenomena but lack defined task roles, controlled scenarios, and slot-level listener-action supervision.Switchboard, Fisher, and Candor therefore offer conversational realism but weak supervision for scenario-specific behavior.
  • Synthetic dialogue generation: Prompting-, heuristic-, and TTS-based synthesis pipelines improve controllability and scale without grounding inserted turn-taking behaviors in target-scenario human preferences.DUPLEXGEN instead combines scenario-controlled synthesis, slot-level human judgments, and context-aware calibration before generation.
  • Full-duplex systems: Full-duplex benchmarks evaluate real-time behavior, but DUPLEXGEN targets the upstream need for training dialogues whose turn-taking is calibrated to human preferences in context.Its positioning is between generic speech corpora and prompted synthetic dialogue: human-calibrated, scenario-adaptive turn-taking synthesis.

3 DUPLEXGEN Framework

DuplexGen synthesizes scenario-adaptive turn-taking by converting text dialogues into spoken-style transcripts, locating candidate slots, calibrating LLM decisions against human preferences, and applying the calibration during generation. Its framework models floor-taking and backchanneling as distinct actions, preserving soft distributions that reflect subjective human judgments.

  • Turn-taking behaviors: The framework models floor-taking as claiming the conversational floor and backchanneling as acknowledging it without taking control.Both behaviors are text-expressible and actionable for LLM assistants.
  • Framework overview: DuplexGen operates in four stages: spoken-style conversion, candidate-slot identification, human-calibrated decision learning, and calibrated dialogue synthesis.The framework focuses on turn-taking behaviors given user utterances.
  • Candidate-slot identification: Candidate slots are identified from linguistic boundaries using both heuristic rules and an LLM, including clause endings, sentence completions, hesitations, and filler words.These slots mark natural locations where turn-taking actions could occur.
  • Distribution calibration: Human calibration collects scenario-specific annotations and maps LLM prediction distributions toward human judgments rather than applying one context-agnostic transformation.This design targets the situational nature of turn-taking.
  • Distribution calibration: Soft human annotation distributions are used instead of majority labels, preserving disagreement because multiple turn-taking actions may be appropriate at one slot.The calibration objective aligns predicted and human distributions using KL divergence.
  • Dialogue synthesis: During synthesis, [TAKE_FLOOR] truncates the current turn and starts the next, whereas [BACKCHANNEL] adds a brief acknowledgment while the turn continues.The calibrated model scans candidate slots and decides whether to insert either marker.

4 Experiments

Experiments show that turn-taking preferences vary systematically by interaction scenario, and human-calibrated DUPLEXGEN models better reproduce these preferences than prompting or generic human-human dialogue training. DUPLEXGEN-synthesized data also yields full-duplex behavior with stronger scenario-dependent variation and higher human-evaluated adherence.

  • Scenario-dependent preferences: Across six tasks, cooperative and competitive interactions exhibit distinct turn-taking norms, while tasks within the same scenario type show similar patterns.Pairwise chi-square tests with Holm correction find significant differences across nearly all scenario pairs (p < 0.05), except within competitive scenarios.
  • Scenario-dependent preferences: Cooperative settings show more backchanneling, whereas competitive scenarios show more floor-taking and cooperative scenarios less.
  • Calibration results: SWBD+DUPLEXGEN consistently achieves the lowest KL divergence against human annotations, except for a marginal difference at 1.7B, indicating the closest alignment with human preferences.PROMPT-ONLY can perform decently at 4B and 14B but fails to capture scenario-specific variation as well as calibrated models.
  • Calibration results: SWBD-ONLY is always less aligned than DUPLEXGEN-ONLY despite using a much larger human-human dialogue corpus.SWBD-ONLY overestimates silence and underestimates backchanneling relative to human annotations, while PROMPT-ONLY generates too many [TAKE_FLOOR] slots.
  • Full-duplex evaluation: PP-DG shows more pronounced scenario-dependent turn-taking variation than Moshi and PP, including stronger floor-taking in competitive scenarios.This pattern matches the human annotation findings from §4.1.
  • Full-duplex evaluation: PP-DG significantly outperforms PP (p = 0.034) and Moshi (p = 0.0026) at the 95% confidence level in human evaluation.The results indicate that DUPLEXGEN provides scenario-specific turn-taking guidance while improving adherence to the interaction setting.

5 Conclusion

DuplexGen addresses scenario-dependent turn-taking in full-duplex interaction by calibrating LLM predictions against a small set of human annotations. Its synthesized data produces scenario-specific behaviors preferred by humans, identifying human calibration as a practical path toward socially appropriate spoken AI.

  • DuplexGen synthesizes scenario-adaptive turn-taking dialogues by calibrating LLM turn-taking predictions against a small set of human annotations.The framework targets variable turn-taking norms across different scenarios.
  • Human turn-taking preferences differ systematically across cooperative and competitive scenarios, shaping when agents should hold, yield, or take the turn.
  • A full-duplex model trained on DuplexGen-synthesized data acquires scenario-specific behaviors preferred by humans.The findings identify human calibration as a practical and scalable path toward socially appropriate spoken AI.

Limitations … C.4 Human Turn-Taking Annotation Interfaces

The paper’s limitations concern data diversity, training sensitivity, and text-grounded supervision, while its appendices detail the synthesis pipeline, prompting, annotation procedures, slot heuristics, and interface examples. Human annotation supports scenario-adaptive choices among backchanneling, floor-taking, and continuing to listen.

  • Limitations: The synthesized datasets cover limited scenarios and interaction types, while model behavior remains sensitive to turn-switching latency, pause duration, and learning rate.The paper identifies broader scenario coverage and robust training recipes as future directions.
  • Limitations: Text-grounded calibration does not directly condition on prosody, pauses, intonation, speech rate, overlap, or visual cues.The paper calls for paired text/audio annotation and multimodal calibration because the listening study does not establish agreement between text-based and audio-conditioned preferences.
  • A Algorithm: DUPLEXGEN Pipeline: DUPLEXGEN calibrates turn-taking predictions before synthesizing utterances with TAKE_FLOOR or BACKCHANNEL markers.The pipeline has separate calibration and dialogue-synthesis phases and returns an utterance with turn-taking markers.
  • B DUPLEXGEN Prompts: The prompts convert polished text into spoken interactions by giving users fillers and self-repairs while requiring assistants to remain concise, fluent, and free of disfluencies.Both prompts preserve the original facts, goals, and speaker personas while adapting delivery to conversational flow.
  • C.1 Heuristics for Turn-Taking Slot Identification: Candidate slots are detected after punctuation, hesitation markers, or LLM-identified clause boundaries, then thinned and supplemented at 4-token intervals when gaps exceed 8 tokens.Slots are inserted only in user turns, excluding the final user token.
  • C.2 Human Turn-Taking Annotation Guidelines: Participants annotate timing and preferred action from three choices: Backchannel, Take Floor, or Continue, using partial dialogue context and incrementally revealed user utterances.The examples associate backchanneling with cooperative engagement, floor-taking with proactive or supportive responses, and continuing with silent listening.
  • C.3 Human Annotation Density Plots: The annotation materials include ternary action-density visualizations and interface examples showing backchannel selection at a prior slot and floor-taking that truncates the remaining user turn.The figures illustrate how participants select actions for each slot in a streaming utterance.
  • C.4 Human Turn-Taking Annotation Interfaces: The interface examples show slot-level action selection during streaming utterances, including backchanneling at one slot and floor-taking at another.The appendix also reports a table on how calibration-data amount affects test KL divergence across six averaged scenarios, without providing values in the supplied passage.

D Ablation on Calibration Data

A data-scaling ablation finds that calibration performance improves with more supervision but shows diminishing returns and gradual convergence, indicating that the current calibration set is sufficient.

  • Data-scaling ablation: The ablation trains a turn-taking predictor on 10%–100% of calibration dialogues, corresponding to 2–20 dialogues per scenario.Held-out test KL divergence is used for evaluation with the 4B model trained on SWBD+DUPLEXGEN as the turn-taking act.
  • Data-scaling ablation: Each training fraction below 100% is repeated with eight independent subsampling seeds, with mean and standard deviation reported.
  • Data-scaling ablation: Across all six scenarios, the macro-average shows diminishing returns as calibration data increases and performance gradually converges.The results indicate that the current calibration set is sufficient.

E Model Training Details · E.1 Turn-Taking Predictor Training

The turn-taking predictors share a common Qwen3-based architecture, soft-label objective, slot construction scheme, and optimization framework, while differing in their supervision corpora and curriculum. PROMPT-ONLY is evaluated separately without gradient updates.

  • E.1 Turn-Taking Predictor Training: Three trained variants—SWBD-ONLY, DUPLEXGEN-ONLY, and SWBD+DUPLEXGEN—share architecture, objective, and optimization, differing only in training corpora.PROMPT-ONLY performs no gradient updates and is described separately.
  • E.1 Turn-Taking Predictor Training: A token-level classification head on Qwen3 backbones maps each final hidden state to [LISTEN], [BACKCHANNEL], or [TAKE_FLOOR].The language-modeling head is replaced with a linear layer W ∈ R^H×3.
  • E.1 Turn-Taking Predictor Training: Models use 1.7B, 4B, and 14B parameter sizes, with candidate slots mapped to the final sub-word token of each boundary word and loss computed only there.Contexts use the Qwen3 chat template with thinking disabled.
  • E.1 Turn-Taking Predictor Training: Training minimizes per-slot KL divergence between predicted action probabilities and categorical target distributions averaged across valid example-slot pairs.Human and SWBD supervision share this soft-label objective.
  • E.1 Turn-Taking Predictor Training: DUPLEXGEN targets use empirical human-annotator vote distributions, whereas SWBD targets are one-hot vectors derived from inserted dialogue-act markers.The two corpora share the same boundary candidate scheme but differ in slot supervision.
  • E.1 Turn-Taking Predictor Training: Each user turn becomes one prefix-based example per boundary candidate, containing one supervised slot at the truncated utterance end with context limited to the last four turns.This construction mirrors streaming inference.
  • E.1 Turn-Taking Predictor Training: SWBD-ONLY trains for 1 epoch on SWBD, DUPLEXGEN-ONLY trains independently per scenario from off-the-shelf Qwen3 weights, and SWBD+DUPLEXGEN continues with per-scenario calibration fine-tuning.The combined variant first runs the SWBD stage, then fine-tunes its resulting LoRA adapter on DUPLEXGEN data.
  • E.1 Turn-Taking Predictor Training: LoRA uses rank 8 and α = 16 with AdamW at learning rate 2×10^-4, no weight decay, a constant schedule, and batch size 16 for each 1-epoch stage.Qwen3-1.7B and 4B train on 2×48 GB GPUs, while 14B trains on 4×48 GB GPUs.

E.2 Full-Duplex Model Training

PP-DG is trained separately for each scenario by fine-tuning PersonaPlex-7B on DuplexGen-generated dialogues. The pipeline filters role-swap artifacts, renders speech variants, conditions samples with hybrid prompts, and applies LoRA fine-tuning.

  • Model construction: PP-DG uses one PersonaPlex-7B checkpoint per scenario, fine-tuned on synthetic dialogues generated by DuplexGen.The source weights are public PersonaPlex-7B weights.
  • Data filtering: A GPT-4.1 zero-shot classifier flags role-swapped dialogues and offending turns, which are removed before text-to-speech rendering.Role swaps occur when user-labeled turns speak in the assistant’s voice, or vice versa.
  • Speech rendering: Each retained dialogue is rendered into ten two-channel speech variants using distinct LibriSpeech user speakers and Chatterbox-TTS assistant speech.User speakers require at least 5 s of reference audio; assistant voice prompting uses cumulative voice prompts.
  • Training conditioning: Every training sample uses a Hybrid System Prompt combining a reference voice prompt, a 0.5 s silence spacer, and a scenario-specific persona description.Loss on the prompt region is masked so the model is not penalized for reproducing its conditioning prefix.
  • Fine-tuning setup: LoRA adapters with rank 32 and scaling 2.0 are trained on Streaming-Transformer blocks while embedding tables remain frozen.Optimization uses AdamW, a peak learning rate of 1×10−5, one epoch, bfloat16, gradient checkpointing, and FSDP sharding.

F Source Dataset License … PROMPT-ONLY SWBD+DUPLEXGEN

The appendices document source-data licenses, restrict LLM use to manuscript language improvement, and compare PROMPT-ONLY with SWBD+DUPLEXGEN using matched dialogue-generation components. The examples illustrate turn-taking behavior in cooperative teaching and competitive negotiation scenarios.

  • F Source Dataset License: Apache-licensed source data include SocraticLM.
  • F Source Dataset License: MIT-licensed source data include MultiWOZ and CraigslistBargain.
  • G Use of AI Assistants: LLMs were used only to improve manuscript grammar, clarity, and expression, not for brainstorming, original drafting, or substantive contributions.
  • H Example: Generated Turn-Taking Dialogues: PROMPT-ONLY and SWBD+DUPLEXGEN are compared side by side using one synthesized dialogue per scenario.
  • PROMPT-ONLY SWBD+DUPLEXGEN: The teaching comparison uses the same GPT-4.1 dialogue writer and GPT-4.1-mini boundary detector, differing only in the turn-taking prediction head.
  • H.1 Cooperative Scenario: Teaching (TEA): In cooperative teaching, the dialogue repeatedly marks user continuation attempts with TAKE_FLOOR and backchannels with BC while the assistant provides stepwise guidance.
  • PROMPT-ONLY SWBD+DUPLEXGEN: The negotiation example shows the assistant prompting the user through incomplete turns before discussing the bike’s backstory and accepting a $900 pickup offer.

I Full-Duplex Model Evaluation · I.1 LLM Prompt for Evaluating Instruction Following · I.2 Human Guideline for Evaluating Turn-Taking Naturalness

The evaluation framework separately measures instruction following with an LLM rubric and turn-taking naturalness with human judgments. Human raters assess whether the AI’s turn-taking is natural and appropriate for its assigned role, focusing on timing, backchannels, and interruptions.

  • I.1 LLM Prompt for Evaluating Instruction Following: The LLM evaluator scores instruction following on a 1–5 scale based on whether the assistant fulfills the user’s goals across the conversation.The listed user goals are treated as the ground-truth goal sequence, and the evaluator returns a JSON score with a brief rationale.
  • I.1 LLM Prompt for Evaluating Instruction Following: Score 5 means all user goals are correctly addressed with accurate, coherent, and appropriately detailed responses.No goal is dropped or forgotten.
  • I.1 LLM Prompt for Evaluating Instruction Following: Scores 4–1 represent progressively larger failures, from minor gaps to irrelevant or contradictory responses that miss the conversation’s core goals.Score 3 indicates partial task success, while scores 2 and 1 indicate several missed goals or failure to address the core task.
  • I.2 Human Guideline for Evaluating Turn-Taking Naturalness: Human raters listen to synthesized human–AI dialogues and judge how natural the AI assistant’s turn-taking feels for its assigned role.They are instructed to focus on turn-taking rather than dialogue content or voice quality.
  • I.2 Human Guideline for Evaluating Turn-Taking Naturalness: The evaluated behaviors include speaker transitions, brief backchannels, and floor-taking or interruptions while the human is speaking.Optional comments may address timing issues, backchannel frequency, interruptions, or other turn-taking problems.
  • I.2 Human Guideline for Evaluating Turn-Taking Naturalness: Human judgments use a 5-point scale ranging from completely natural and role-appropriate to completely unnatural, disruptive, or inappropriate.Intermediate scores distinguish mostly natural behavior, acceptable but somewhat unnatural behavior, and often unnatural or poorly timed behavior.
Loading 2607.26178v1…