PaperlayerView exact revision on arXiv

Source-linked AI summary

DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues

Takyoung Kim, Kang-wook Kim, Sang Hoon Woo, Julia Hirschberg, Gunhee Kim, Dilek Hakkani-Tür

arXiv:2607.26178v1cs.CL

TL;DR

Current full-duplex models do not adapt turn-taking to scenario-specific social roles, and existing data or synthesis methods lack human validation. DuplexGen calibrates LLM turn-taking predictions with a small set of human annotations, producing substantially better alignment with human judgments and scenario-specific behaviors preferred by humans.

  • Problem

    Current full-duplex models lack scenario-sensitive turn-taking, while existing corpora and synthesis methods do not validate behavior against human preferences in target scenarios.

    Source: §1 · p. 1

  • Method

    DUPLEXGEN identifies candidate turn-taking slots and calibrates LLM predictions with a small set of scenario-specific human annotations during dialogue synthesis.

    Source: §1 · p. 2, §3.2 · p. 4, §3.2.2 · p. 5

  • Results

    Small-budget calibration substantially improves alignment with human judgments over uncalibrated prompting or large generic corpora, while trained full-duplex models acquire human-preferred scenario-specific behaviors.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Human calibration is a practical and scalable path toward socially appropriate, scenario-specific turn-taking in spoken AI.

    Source: §5 · p. 9

    DUPLEXGEN uses incrementally revealed transcripts rather than prosody, pauses, intonation, speech rate, overlap, or visual cues, and text–audio preference agreement remains unestablished.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Turn-taking is a central component of full-duplex interaction. Which turn-taking behaviors are appropriate varies with the scenario, yet current models apply a single norm regardless of context. This limitation originates in their training data: human-human speech corpora capture natural timing phenomena but provide little role grounding or scenario-specific norms, while heuristic or prompted synthesis methods inject turn-taking behaviors without basing them on human preferences. We introduce DuplexGen, a framework for generating dialogues with scenario-adaptive turn-taking by calibrating LLM predictions against a small set of slot-level human preference annotations. In six cooperative and competitive tasks, human turn-taking preferences differ systematically, and DuplexGen aligns substantially more closely with those preferences than uncalibrated prompting or training solely on generic human-human data; a full-duplex model trained on DuplexGen-generated data exhibits distinctive, human-preferred turn-taking behaviors. These results show that human calibration, not corpus scale or prompt design alone, is what allows turn-taking synthesis to be scenario-specific.

1 Introduction

DUPLEXGEN addresses the lack of scenario-sensitive turn-taking in full-duplex language models by calibrating LLM predictions with a small set of slot-level human annotations. Across six cooperative and competitive tasks, this calibration improves alignment with human preferences and yields models exhibiting scenario-specific behaviors.

1 Introduction

  • Turn-taking strategies depend on social context and role structure, but current full-duplex language models do not reflect this scenario sensitivity.Different roles call for different behaviors, such as patient listening for tutors and more frequent interjections for negotiators.Source: §1 · p. 1
  • Generic human conversation corpora capture natural turn-taking at scale but lack defined task roles and scenario structure.They commonly pair unfamiliar speakers on assigned topics, limiting their grounding in role-specific interaction norms.Source: §1 · p. 1
  • DUPLEXGEN identifies candidate turn-taking slots and calibrates LLM action distributions with a small set of per-scenario human annotations.The framework converts text dialogue into spoken-style transcripts and predicts [LISTEN], [BACKCHANNEL], and [TAKE_FLOOR] actions.Source: §1 · p. 2
  • DUPLEXGEN is evaluated on six cooperative and competitive tasks spanning tutoring, planning, interview, negotiation, persuasion, and social chat.The task set provides varied goal structures and relational dynamics for studying scenario-dependent timing preferences.Source: §1 · p. 2
  • Human turn-taking preferences differ systematically across scenarios, while calibrated models align substantially better with human judgments than prompt-only generation or generic human–human training.These findings support scenario-adaptive modeling and identify human calibration as the key ingredient in scenario-specific synthesis.Source: §1 · p. 2
  • A full-duplex model trained on DUPLEXGEN-generated data exhibits distinctive, scenario-specific turn-taking behaviors preferred by humans.This demonstrates that the synthesized corpus can transfer human-preferred timing patterns to a full-duplex model.Source: §1 · p. 2

2 Related Work

Prior work explains conversational timing and provides speech resources or synthesis methods, but does not establish which listener action is socially appropriate for a given scenario. DUPLEXGEN addresses this gap through human-calibrated, scenario-adaptive turn-taking synthesis for full-duplex dialogue data.

Turn-taking foundations

  • Turn-taking research models conversation through transition points, anticipatory response preparation, and acoustic, visual, and incremental language cues.These approaches often target coordination, latency reduction, or turn-boundary prediction.Source: §2.1 · pp. 2–3

Scenario-specific norms

  • Existing timing models do not determine whether silence, backchanneling, or floor-taking is socially appropriate across assistant roles and scenarios.The same partial utterance may call for different listener actions when the assistant is a tutor, planner, interviewer, or negotiation partner.Source: §2.1 · p. 3

Existing data resources

  • Generic human–human corpora provide natural timing phenomena but lack defined task roles, controlled scenarios, and slot-level listener-action supervision.Switchboard, Fisher, and Candor therefore offer conversational realism but weak supervision for scenario-specific behavior.Source: §2.2 · p. 3

Synthetic dialogue generation

  • Prompting-, heuristic-, and TTS-based synthesis pipelines improve controllability and scale without grounding inserted turn-taking behaviors in target-scenario human preferences.DUPLEXGEN instead combines scenario-controlled synthesis, slot-level human judgments, and context-aware calibration before generation.Source: §2.2 · p. 3

Full-duplex systems

  • Full-duplex benchmarks evaluate real-time behavior, but DUPLEXGEN targets the upstream need for training dialogues whose turn-taking is calibrated to human preferences in context.Its positioning is between generic speech corpora and prompted synthetic dialogue: human-calibrated, scenario-adaptive turn-taking synthesis.Source: §2.3 · p. 3

3 DUPLEXGEN Framework

DuplexGen synthesizes scenario-adaptive turn-taking by converting text dialogues into spoken-style transcripts, locating candidate slots, calibrating LLM decisions against human preferences, and applying the calibration during generation. Its framework models floor-taking and backchanneling as distinct actions, preserving soft distributions that reflect subjective human judgments.

Turn-taking behaviors

  • The framework models floor-taking as claiming the conversational floor and backchanneling as acknowledging it without taking control.Both behaviors are text-expressible and actionable for LLM assistants.Source: §3 · pp. 3–4

Framework overview

  • DuplexGen operates in four stages: spoken-style conversion, candidate-slot identification, human-calibrated decision learning, and calibrated dialogue synthesis.The framework focuses on turn-taking behaviors given user utterances.Source: §3.2 · p. 4

Candidate-slot identification

  • Candidate slots are identified from linguistic boundaries using both heuristic rules and an LLM, including clause endings, sentence completions, hesitations, and filler words.These slots mark natural locations where turn-taking actions could occur.Source: §3.2.2 · p. 5

Distribution calibration

  • Human calibration collects scenario-specific annotations and maps LLM prediction distributions toward human judgments rather than applying one context-agnostic transformation.This design targets the situational nature of turn-taking.Source: §3.2.3 · p. 5
  • Soft human annotation distributions are used instead of majority labels, preserving disagreement because multiple turn-taking actions may be appropriate at one slot.The calibration objective aligns predicted and human distributions using KL divergence.Source: §3.2.3 · p. 5

Dialogue synthesis

  • During synthesis, [TAKE_FLOOR] truncates the current turn and starts the next, whereas [BACKCHANNEL] adds a brief acknowledgment while the turn continues.The calibrated model scans candidate slots and decides whether to insert either marker.Source: §3.2.4 · pp. 5–6

4 Experiments

Experiments show that turn-taking preferences vary systematically by interaction scenario, and human-calibrated DUPLEXGEN models better reproduce these preferences than prompting or generic human-human dialogue training. DUPLEXGEN-synthesized data also yields full-duplex behavior with stronger scenario-dependent variation and higher human-evaluated adherence.

Scenario-dependent preferences

  • Across six tasks, cooperative and competitive interactions exhibit distinct turn-taking norms, while tasks within the same scenario type show similar patterns.Pairwise chi-square tests with Holm correction find significant differences across nearly all scenario pairs (p < 0.05), except within competitive scenarios.Source: §4.1 · p. 6
  • Cooperative settings show more backchanneling, whereas competitive scenarios show more floor-taking and cooperative scenarios less.Source: §4.1 · p. 6

Calibration results

  • SWBD+DUPLEXGEN consistently achieves the lowest KL divergence against human annotations, except for a marginal difference at 1.7B, indicating the closest alignment with human preferences.PROMPT-ONLY can perform decently at 4B and 14B but fails to capture scenario-specific variation as well as calibrated models.Source: §4.2.4 · p. 7
  • SWBD-ONLY is always less aligned than DUPLEXGEN-ONLY despite using a much larger human-human dialogue corpus.SWBD-ONLY overestimates silence and underestimates backchanneling relative to human annotations, while PROMPT-ONLY generates too many [TAKE_FLOOR] slots.Source: §4.2.4 · p. 8

Full-duplex evaluation

  • PP-DG shows more pronounced scenario-dependent turn-taking variation than Moshi and PP, including stronger floor-taking in competitive scenarios.This pattern matches the human annotation findings from §4.1.Source: §4.3.2 · p. 8, Figure 4 · p. 8
  • PP-DG significantly outperforms PP (p = 0.034) and Moshi (p = 0.0026) at the 95% confidence level in human evaluation.The results indicate that DUPLEXGEN provides scenario-specific turn-taking guidance while improving adherence to the interaction setting.Source: §4.3.3 · p. 9

5 Conclusion

DuplexGen addresses scenario-dependent turn-taking in full-duplex interaction by calibrating LLM predictions against a small set of human annotations. Its synthesized data produces scenario-specific behaviors preferred by humans, identifying human calibration as a practical path toward socially appropriate spoken AI.

  • DuplexGen synthesizes scenario-adaptive turn-taking dialogues by calibrating LLM turn-taking predictions against a small set of human annotations.The framework targets variable turn-taking norms across different scenarios.Source: §5 · p. 9
  • Human turn-taking preferences differ systematically across cooperative and competitive scenarios, shaping when agents should hold, yield, or take the turn.Source: §5 · p. 9
  • A full-duplex model trained on DuplexGen-synthesized data acquires scenario-specific behaviors preferred by humans.The findings identify human calibration as a practical and scalable path toward socially appropriate spoken AI.Source: §5 · p. 9

Limitations … C.4 Human Turn-Taking Annotation Interfaces

The paper’s limitations concern data diversity, training sensitivity, and text-grounded supervision, while its appendices detail the synthesis pipeline, prompting, annotation procedures, slot heuristics, and interface examples. Human annotation supports scenario-adaptive choices among backchanneling, floor-taking, and continuing to listen.

Limitations

  • The synthesized datasets cover limited scenarios and interaction types, while model behavior remains sensitive to turn-switching latency, pause duration, and learning rate.The paper identifies broader scenario coverage and robust training recipes as future directions.Source: Limitations · p. 9
  • Text-grounded calibration does not directly condition on prosody, pauses, intonation, speech rate, overlap, or visual cues.The paper calls for paired text/audio annotation and multimodal calibration because the listening study does not establish agreement between text-based and audio-conditioned preferences.Source: Limitations · p. 9

A Algorithm: DUPLEXGEN Pipeline

  • DUPLEXGEN calibrates turn-taking predictions before synthesizing utterances with TAKE_FLOOR or BACKCHANNEL markers.The pipeline has separate calibration and dialogue-synthesis phases and returns an utterance with turn-taking markers.Source: A Algorithm: DUPLEXGEN Pipeline · p. 14

B DUPLEXGEN Prompts

  • The prompts convert polished text into spoken interactions by giving users fillers and self-repairs while requiring assistants to remain concise, fluent, and free of disfluencies.Both prompts preserve the original facts, goals, and speaker personas while adapting delivery to conversational flow.Source: §B.1 · p. 14, §B.4 · p. 16

C.1 Heuristics for Turn-Taking Slot Identification

  • Candidate slots are detected after punctuation, hesitation markers, or LLM-identified clause boundaries, then thinned and supplemented at 4-token intervals when gaps exceed 8 tokens.Slots are inserted only in user turns, excluding the final user token.Source: §C.1 · p. 18

C.2 Human Turn-Taking Annotation Guidelines

  • Participants annotate timing and preferred action from three choices: Backchannel, Take Floor, or Continue, using partial dialogue context and incrementally revealed user utterances.The examples associate backchanneling with cooperative engagement, floor-taking with proactive or supportive responses, and continuing with silent listening.Source: §C.2 · p. 18

C.3 Human Annotation Density Plots

  • The annotation materials include ternary action-density visualizations and interface examples showing backchannel selection at a prior slot and floor-taking that truncates the remaining user turn.The figures illustrate how participants select actions for each slot in a streaming utterance.Source: Figure 5 · p. 19, Figure 6 · p. 20, Figure 7 · p. 20

C.4 Human Turn-Taking Annotation Interfaces

  • The interface examples show slot-level action selection during streaming utterances, including backchanneling at one slot and floor-taking at another.The appendix also reports a table on how calibration-data amount affects test KL divergence across six averaged scenarios, without providing values in the supplied passage.Source: Figure 6 · p. 20, Figure 7 · p. 20, Table 9 · p. 21

D Ablation on Calibration Data

A data-scaling ablation finds that calibration performance improves with more supervision but shows diminishing returns and gradual convergence, indicating that the current calibration set is sufficient.

Data-scaling ablation

  • The ablation trains a turn-taking predictor on 10%–100% of calibration dialogues, corresponding to 2–20 dialogues per scenario.Held-out test KL divergence is used for evaluation with the 4B model trained on SWBD+DUPLEXGEN as the turn-taking act.Source: D Ablation on Calibration Data · p. 21
  • Each training fraction below 100% is repeated with eight independent subsampling seeds, with mean and standard deviation reported.Source: D Ablation on Calibration Data · p. 21
  • Across all six scenarios, the macro-average shows diminishing returns as calibration data increases and performance gradually converges.The results indicate that the current calibration set is sufficient.Source: D Ablation on Calibration Data · p. 21

E Model Training Details · E.1 Turn-Taking Predictor Training

The turn-taking predictors share a common Qwen3-based architecture, soft-label objective, slot construction scheme, and optimization framework, while differing in their supervision corpora and curriculum. PROMPT-ONLY is evaluated separately without gradient updates.

E.1 Turn-Taking Predictor Training

  • Three trained variants—SWBD-ONLY, DUPLEXGEN-ONLY, and SWBD+DUPLEXGEN—share architecture, objective, and optimization, differing only in training corpora.PROMPT-ONLY performs no gradient updates and is described separately.Source: §E.1 · p. 21
  • A token-level classification head on Qwen3 backbones maps each final hidden state to [LISTEN], [BACKCHANNEL], or [TAKE_FLOOR].The language-modeling head is replaced with a linear layer W ∈ R^H×3.Source: §E.1 · p. 21
  • Models use 1.7B, 4B, and 14B parameter sizes, with candidate slots mapped to the final sub-word token of each boundary word and loss computed only there.Contexts use the Qwen3 chat template with thinking disabled.Source: §E.1 · p. 21
  • Training minimizes per-slot KL divergence between predicted action probabilities and categorical target distributions averaged across valid example-slot pairs.Human and SWBD supervision share this soft-label objective.Source: §E.1 · p. 21
  • DUPLEXGEN targets use empirical human-annotator vote distributions, whereas SWBD targets are one-hot vectors derived from inserted dialogue-act markers.The two corpora share the same boundary candidate scheme but differ in slot supervision.Source: §E.1 · p. 21
  • Each user turn becomes one prefix-based example per boundary candidate, containing one supervised slot at the truncated utterance end with context limited to the last four turns.This construction mirrors streaming inference.Source: §E.1 · p. 22
  • SWBD-ONLY trains for 1 epoch on SWBD, DUPLEXGEN-ONLY trains independently per scenario from off-the-shelf Qwen3 weights, and SWBD+DUPLEXGEN continues with per-scenario calibration fine-tuning.The combined variant first runs the SWBD stage, then fine-tunes its resulting LoRA adapter on DUPLEXGEN data.Source: §E.1 · p. 22
  • LoRA uses rank 8 and α = 16 with AdamW at learning rate 2×10^-4, no weight decay, a constant schedule, and batch size 16 for each 1-epoch stage.Qwen3-1.7B and 4B train on 2×48 GB GPUs, while 14B trains on 4×48 GB GPUs.Source: §E.1 · p. 22

E.2 Full-Duplex Model Training

PP-DG is trained separately for each scenario by fine-tuning PersonaPlex-7B on DuplexGen-generated dialogues. The pipeline filters role-swap artifacts, renders speech variants, conditions samples with hybrid prompts, and applies LoRA fine-tuning.

Model construction

  • PP-DG uses one PersonaPlex-7B checkpoint per scenario, fine-tuned on synthetic dialogues generated by DuplexGen.The source weights are public PersonaPlex-7B weights.Source: §E.2 · p. 22

Data filtering

  • A GPT-4.1 zero-shot classifier flags role-swapped dialogues and offending turns, which are removed before text-to-speech rendering.Role swaps occur when user-labeled turns speak in the assistant’s voice, or vice versa.Source: §E.2 · p. 22

Speech rendering

  • Each retained dialogue is rendered into ten two-channel speech variants using distinct LibriSpeech user speakers and Chatterbox-TTS assistant speech.User speakers require at least 5 s of reference audio; assistant voice prompting uses cumulative voice prompts.Source: §E.2 · pp. 22–23

Training conditioning

  • Every training sample uses a Hybrid System Prompt combining a reference voice prompt, a 0.5 s silence spacer, and a scenario-specific persona description.Loss on the prompt region is masked so the model is not penalized for reproducing its conditioning prefix.Source: §E.2 · p. 22

Fine-tuning setup

  • LoRA adapters with rank 32 and scaling 2.0 are trained on Streaming-Transformer blocks while embedding tables remain frozen.Optimization uses AdamW, a peak learning rate of 1×10−5, one epoch, bfloat16, gradient checkpointing, and FSDP sharding.Source: §E.2 · p. 22

F Source Dataset License … PROMPT-ONLY SWBD+DUPLEXGEN

The appendices document source-data licenses, restrict LLM use to manuscript language improvement, and compare PROMPT-ONLY with SWBD+DUPLEXGEN using matched dialogue-generation components. The examples illustrate turn-taking behavior in cooperative teaching and competitive negotiation scenarios.

F Source Dataset License

G Use of AI Assistants

  • LLMs were used only to improve manuscript grammar, clarity, and expression, not for brainstorming, original drafting, or substantive contributions.Source: G Use of AI Assistants · p. 23

H Example: Generated Turn-Taking Dialogues

PROMPT-ONLY SWBD+DUPLEXGEN

H.1 Cooperative Scenario: Teaching (TEA)

  • In cooperative teaching, the dialogue repeatedly marks user continuation attempts with TAKE_FLOOR and backchannels with BC while the assistant provides stepwise guidance.Source: PROMPT-ONLY SWBD+DUPLEXGEN · p. 24

PROMPT-ONLY SWBD+DUPLEXGEN

  • The negotiation example shows the assistant prompting the user through incomplete turns before discussing the bike’s backstory and accepting a $900 pickup offer.Source: PROMPT-ONLY SWBD+DUPLEXGEN · p. 25

I Full-Duplex Model Evaluation · I.1 LLM Prompt for Evaluating Instruction Following · I.2 Human Guideline for Evaluating Turn-Taking Naturalness

The evaluation framework separately measures instruction following with an LLM rubric and turn-taking naturalness with human judgments. Human raters assess whether the AI’s turn-taking is natural and appropriate for its assigned role, focusing on timing, backchannels, and interruptions.

I.1 LLM Prompt for Evaluating Instruction Following

  • The LLM evaluator scores instruction following on a 1–5 scale based on whether the assistant fulfills the user’s goals across the conversation.The listed user goals are treated as the ground-truth goal sequence, and the evaluator returns a JSON score with a brief rationale.Source: §I.1 · p. 26
  • Score 5 means all user goals are correctly addressed with accurate, coherent, and appropriately detailed responses.No goal is dropped or forgotten.Source: §I.1 · p. 26
  • Scores 4–1 represent progressively larger failures, from minor gaps to irrelevant or contradictory responses that miss the conversation’s core goals.Score 3 indicates partial task success, while scores 2 and 1 indicate several missed goals or failure to address the core task.Source: §I.1 · p. 26

I.2 Human Guideline for Evaluating Turn-Taking Naturalness

  • Human raters listen to synthesized human–AI dialogues and judge how natural the AI assistant’s turn-taking feels for its assigned role.They are instructed to focus on turn-taking rather than dialogue content or voice quality.Source: §I.2 · pp. 26–27
  • The evaluated behaviors include speaker transitions, brief backchannels, and floor-taking or interruptions while the human is speaking.Optional comments may address timing issues, backchannel frequency, interruptions, or other turn-taking problems.Source: §I.2 · pp. 26–27
  • Human judgments use a 5-point scale ranging from completely natural and role-appropriate to completely unnatural, disruptive, or inappropriate.Intermediate scores distinguish mostly natural behavior, acceptable but somewhat unnatural behavior, and often unnatural or poorly timed behavior.Source: §I.2 · p. 27
Loading 2607.26178v1…