PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaptDuplex: from static to adaptive full-duplex spoken dialogue

Zhiyang Zhou, Yingxin Shang, Zhou Wang, Hongwei Cai, Weixu Wang, Shuran Zhou, Shuofeng Zhao, Wenke Fan, Qingxiang Guo, Dawei Yang, Lin Yang, Yang Song

arXiv:2609.29217v1cs.SD

TL;DR

Full-duplex dialogue models must adapt to changing conversational timing and cognitive demands, but existing systems impose static operating points. AdaptDuplex upgrades Qwen3-Omni with a token-level protocol, adaptive interaction mechanisms, and progressive training. It reports stronger turn-taking, overlap, and timing results on Full-Duplex-Bench and the highest Final score among compared duplex models on HumDial-FDBench.

  • Problem

    Existing duplex models fix window duration or restrict available actions, while full-duplex dialogue requires responses to changing conversational timing and cognitive demands.

    Source: §1 · p. 1

  • Method

    AdaptDuplex upgrades Qwen3-Omni with an explicit-token protocol, adaptive window and action mechanisms, and a progressive training pipeline.

    Source: §1 · p. 1

  • Results

    AdaptDuplex more often handles turn-taking and overlap appropriately while maintaining sub-second response latency across reported Full-Duplex-Bench scenarios, and achieves the highest Final score among compared duplex models on HumDial-FDBench (72.9).

    Source: §3.2 · p. 3

  • Takeaways & Limitations

    The results support adaptive full-duplex interaction with gains in interaction decisions and response timing on the reported benchmarks, including human-recorded interactions.

    Source: §3.2 · p. 3

    Evaluation covers Chinese and English; the window controller selects among three discrete durations, and external reasoning is limited to four concurrent requests.

    Source: §4 · p. 4

Abstract

from arXiv · show

Full-duplex spoken dialogue requires simultaneous listening and speaking at sub-second latency, under conversational timing and cognitive demands that change moment to moment. Yet current models mostly impose static operating points, lacking a systematic mechanism for adaptive decisions. We present AdaptDuplex, which upgrades Qwen3-Omni with such a mechanism, co-designed across three layers. A compact token-level protocol represents every window as a canonical sequence, trains dual-stream alignment through a bounded text lead over speech, and exposes every behavioral decision as an explicit token for training-free runtime control via logits bias. Adaptive mechanisms dynamically predict among discrete window durations and augment direct response as needed with non-blocking cognitive consolidation and multi-flight external reasoning. A progressive pipeline introduces these behaviors through a three-stage Thinker curriculum, then Talker-only and joint SFT, with GRPO as a further increment. On Full-Duplex-Bench v1 and v1.5, AdaptDuplex outperforms DuplexOmni and MiniCPM-o 4.5 on the majority of comparable turn-taking, overlap-behavior, and timing metrics, with gains in both interaction decisions and response timing. On the human-recorded HumDial-FDBench, it attains the highest Final score (72.9) of the compared duplex models.

1. INTRODUCTION

Full-duplex dialogue must coordinate continuous listening, overlapping speech, and sub-second responses as conversational demands shift. AdaptDuplex addresses static interaction policies with a token-level protocol, adaptive mechanisms, and progressive training.

1. INTRODUCTION

  • Full-duplex interaction requires continuous listening, coordinated overlap, and sub-second responses while conversational timing and cognitive demands change moment to moment.Dedicated duplex models have demonstrated feasibility, and pretrained omni-models provide multimodal capabilities, but adapting to changing demands remains difficult.Source: §1 · p. 1
  • Static window durations and restricted action sets leave existing duplex models without a systematic way to adapt temporal granularity and compose actions at runtime.DuplexOmni fixes window duration and allows only single-flight reasoning without intermediate consolidation; MiniCPM-o 4.5 fixes duration and offers binary interaction control.Source: §1 · p. 1
  • Training protocol adherence, interaction control, and speech generation together entangles objectives, while staging by data scale, quality, or trainable modules does not order behaviors by difficulty.The paper identifies the need for a curriculum that stages duplex behaviors by their difficulty.Source: §1 · p. 1
  • AdaptDuplex builds on Qwen3-Omni and co-designs a canonical sequence protocol, adaptive interaction mechanisms, and a progressive training pipeline around explicit behavioral decision tokens.The protocol supports runtime logits-bias control; the mechanisms predict discrete window durations and add cognitive consolidation or multi-flight reasoning when needed.Source: §1 · p. 1

2. PROPOSED METHOD

AdaptDuplex changes Qwen3-Omni’s Thinker from producing a complete response per user turn to making per-window interaction decisions, with reply text optional; the Talker remains responsible for speech synthesis. Its method combines a canonical token sequence, adaptive windows and actions, and staged training.

2.1. Token-level Sequence Protocol

  • Each Unit uses a fixed-order sequence of control, optional action tokens and assistant text, a next-window token, and boundary tokens to coordinate the Thinker and Talker.The Talker bootstraps from environment history when speaking begins, then consumes queued text conditions through a turn-local FIFO; bounded Text-ahead trains streaming text–speech alignment, while explicit decision tokens permit inference-time logits-bias control.Source: §2.1 · pp. 1–2, Figure 1 · p. 2

2.2. Adaptive Interaction Mechanisms

  • The Thinker predicts the next window as short, mid, or long from available multimodal context, while independently selectable non-blocking actions add memory consolidation or asynchronous external reasoning.The three durations are 0.48, 0.64, and 0.96 seconds; up to four external requests can be in flight, and failures or non-returns do not block the current response.Source: §2.2 · p. 2

2.3. Progressive Training Pipeline

  • Training first stages Thinker SFT from protocol learning to basic duplex events and then full duplex with orchestration actions, before Talker-only and joint SFT; GRPO further refines the interaction policy.Talker-only SFT uses FIFO text conditions and bounded Text-ahead, and GRPO continues from the joint-SFT checkpoint without changing the architecture.Source: §2.3 · pp. 2–3

3. EXPERIMENTS

Experiments compare AdaptDuplex with duplex baselines on synthetic benchmarks and human-recorded dialogue, then examine its protocol, adaptive controls, and training strategy. Results show gains in turn-taking, overlap handling, and timing, alongside trade-offs and remaining control limits.

3.1. Experimental Setup

  • The evaluation uses FDB and FDB-v1.5, HumDial-FDBench in Chinese and English, and a 765,424-dialogue synthetic corpus for Curriculum SFT and Mixed SFT.Mixed SFT matches Curriculum in instance count and per-instance exposure; Joint SFT continues on a smaller high-quality DuplexDrama subset.Source: §3.1 · p. 3

3.2. Overall Full-Duplex Performance

  • 0.334 vs. 0.491 takeover rate and FDB-v1.5 behavior scores of 0.55 vs. 0.28 and 0.43 vs. 0.07 show stronger backchannel and overlap handling than DuplexOmni.Response latency is sub-second in all reported FDB scenarios and lowest in every FDB-v1.5 scenario; on FDB-v1 interruptions, TOR is 0.985 vs. 0.746 and latency is 0.131 vs. 0.158 s.Source: Table 1 · p. 3, §3.2 · p. 3
  • 72.9 is AdaptDuplex’s highest Final score among compared duplex models on human-recorded HumDial-FDBench interactions.Its delays are 0.835 s (ZH) and 1.345 s (EN), faster than MiniCPM-o 4.5 and DuplexOmni.Source: Table 2 · p. 3, §3.2 · p. 3

3.3. Protocol Efficiency, Alignment, and Control

  • 78% fewer tokens per window and lower decoding-time P50/P90/P99 by 47%/49%/42% accompany canonical serialization; protocol output share falls from 93.88% to 68.70%.The comparison holds semantic records, fixed 0.48-s windows, and the adapted Talker constant while replacing the canonical sequence with matched JSON encoding.Source: §3.3 · pp. 3–4, Table 3 · p. 4
  • UTMOSv2 rises from 2.90 to 3.55 with Fixed Ahead and 3.65 with Random Ahead; ValidEnd rises from 50.00% to 84.75% and 86.37%, respectively.Random Ahead is best on naturalness, bilingual error, and ValidEnd among the tested text-ahead schedules.Source: §3.3 · p. 4
  • Raising speak-logit bias α from 0.5 to 2 increases interruption behavior from 0.76 to 0.81 and shortens response latency from 0.42 to 0.33 s, while backchannel and talking-to-others scores fall.This is an inference-time listen/speak trade-off without retraining.Source: §3.3 · p. 4, Table 4 · p. 4

3.4. Adaptive Interaction

  • Dynamic windows combine near-short latency (0.37 s, 1.32 s) with near-long backchannel (0.55); gold-context tests also show improved fact and preference accuracy from injected memory state.Fact accuracy rises from 10.25% to 62.75% and preference accuracy from 16.75% to 59.25%; with at least two in-flight requests, action EM falls to 65.17% and invalid actions rise from 3.80% to 7.10%.Source: §3.4 · p. 4, Table 4 · p. 4

3.5. Training Strategy

  • Curriculum improves behavior in all four scenarios over matched-exposure Mixed SFT, with response latency changing from 1.65 to 0.37 s; Joint SFT and GRPO add mostly monotonic changes.Backchannel is unchanged after Joint SFT, while background-speech performance dips slightly after GRPO.Source: §3.5 · p. 4, §3.4 · p. 4

4. CONCLUSION

AdaptDuplex combines a compact token-level protocol, adaptive window and action mechanisms, and progressive training; its evaluation and controller remain bounded in several ways.

  • AdaptDuplex upgrades Qwen3-Omni by codesigning a compact token-level protocol, adaptive window and action mechanisms, and a progressive training pipeline.Source: §4 · p. 4
  • Evaluation currently covers Chinese and English only.Source: §4 · p. 4
  • The window controller selects among three discrete durations rather than a continuous horizon.Source: §4 · p. 4
  • External reasoning is bounded to four concurrent requests.Source: §4 · p. 4
Loading 2609.29217v1…