PaperlayerView exact revision on arXiv

Source-linked AI summary

D2: High-Capability Full-Duplex Spoken Language Models from Half-Duplex Backbones

Cheng-Kuang Chang, Alexander H. Liu, Brian Kingsbury, Hilde Kuehne, Rogerio Feris, Samuel Thomas, James Glass, Kai-Wei Chang

arXiv:2610.17124v1cs.CLcs.AIeess.AS

TL;DR

Full-duplex spoken language models support simultaneous listening and speaking but often trail half-duplex models in general capabilities. D2 converts pretrained half-duplex models through streaming encoder distillation and full-duplex supervised fine-tuning; across two architectures it reports strong interaction results and state-of-the-art VoiceBench performance among evaluated full-duplex models.

  • Problem

    Full-duplex spoken language models support simultaneous listening and speaking but often lag behind half-duplex models in audio understanding, knowledge, reasoning, and instruction following.

    Source: §1 · p. 1

  • Method

    D2 converts pretrained half-duplex models into full-duplex models through streaming encoder distillation followed by full-duplex supervised fine-tuning.

    Source: §1 · p. 1, §3.2 · p. 3

  • Results

    70.1 VoiceBench score for Qwen3-Omni-D2 at 12.5 Hz, with 99.2% turn-taking TOR and 99.5% interruption TOR; across two architectures, D2 achieves state-of-the-art VoiceBench performance among evaluated full-duplex models.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Across two different half-duplex architectures, D2 narrows the capability gap between full-duplex and half-duplex models while enabling low-latency streaming and strong interaction.

    Source: §7 · p. 8

    Speech-synthesis latency restricts D2's backbone choice to half-duplex models with sufficiently low-latency speech synthesis.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Full-duplex spoken language models (FD-SLMs) support simultaneous listening and speaking, but often lag behind half-duplex spoken language models (HD-SLMs) in general capabilities. We propose Dialogue-to-Duplex (D2), a framework that converts pretrained HD-SLMs into high-capability FD-SLMs at fine temporal granularity. D2 adapts the original audio encoder to streaming through decoder-free representation distillation and learns full-duplex interaction through supervised fine-tuning. We systematically study the effects of chunk size on full-duplex interaction, general capabilities, and inference computation. We apply D2 to two distinct HD-SLM architectures, Qwen3-Omni and LLaMA-Omni 2. At a 12.5Hz update rate, Qwen3-Omni-D2 achieves 99.2% turn-taking success with 521-ms latency, 99.5% interruption success, and a state-of-the-art VoiceBench score of 70.1.

1 Introduction

Full-duplex models support simultaneous listening and speaking, but often lag behind half-duplex models in general capabilities; D2 converts pretrained half-duplex models to fine-grained full-duplex operation. On Qwen3-Omni, D2 reports a 70.1 VoiceBench score alongside strong turn-taking and interruption results.

1 Introduction

  • Native full-duplex models can lag behind half-duplex models in audio understanding, knowledge, reasoning, and instruction following, motivating conversion of stronger half-duplex models.Full-duplex models support simultaneous listening and speaking, interruptions, overlapping speech, and low-latency turn taking.Source: §1 · p. 1
  • D2 converts pretrained half-duplex models into fine-grained full-duplex models, maintaining strong general capabilities even at 12.5 Hz.It addresses the encoder streaming bottleneck through representation distillation and learns full-duplex interaction through supervised fine-tuning.Source: §1 · p. 1
  • VoiceBench score of 70.1, 9.1 points above the strongest evaluated full-duplex baseline, accompanies 99.2% turn-taking TOR and 99.5% interruption TOR for Qwen3-Omni-D2.The VoiceBench score is reported at 12.5 Hz; the TOR results are on Full-Duplex-Bench.Source: §1 · p. 2
  • Decoder-free representation distillation reduces encoder latency without requiring the original training decoder or transcription labels.Source: §1 · p. 2
  • The paper systematically studies how chunk size affects full-duplex interaction, general capabilities, and inference computation.Source: §1 · p. 2

2 Related Work

Related work covers pretrained half-duplex backbones, models that already support simultaneous listening and speaking, and strategies for converting models to full-duplex operation.

2.1 Half-Duplex Spoken Language Models

  • Pretrained half-duplex spoken language models include recent backbones such as GLM-4-Voice, LLaMA-Omni 2, Kimi-Audio, and Qwen3-Omni (Zeng et al., 2024; Fang et al., 2025b; Ding et al., 2025; Xu et al., 2025).Earlier systems include SpeechGPT and AudioPaLM (Zhang et al., 2023; Rubenstein et al., 2023).Source: §2.1 · p. 2

2.2 Full-Duplex Spoken Language Models

2.3 Conversion Strategy

  • OmniFlatten converts a pretrained text LLM through speech-modality alignment and full-duplex dialogue training (Zhang et al., 2025), whereas D2 starts from an HD-SLM.BayLing-Duplex converts GLM-4-Voice but uses up to 4-s input blocks and 800-ms chunks; D2 uses streaming encoder distillation and supervised fine-tuning for 80–100-ms chunks (Fang et al., 2026; Zeng et al., 2024).Source: §2.3 · p. 2

3 Method

D2 turns pretrained half-duplex models into full-duplex models by adapting their encoders for streaming and fine-tuning them to make chunk-level conversational decisions. Its chunked inference uses current user audio and preceding model audio to generate speech while interaction granularity depends on chunk size.

3.1 Half-Duplex vs. Full-Duplex

  • The formulation represents each interaction unit with user audio Xi and model output Yi, conditioned on the preceding dialogue context Hi.Half-duplex units span complete turns; full-duplex units contain k encoder frames.Source: §3.1 · p. 2

3.2 Dialogue-to-Duplex (D2)

  • D2 treats full-duplex dialogue as ordinary dialogue segmented into chunks of duration k∆, so smaller k means more frequent observation and response.Half-duplex models instead use complete user turns as interaction units, while full-duplex models use shorter chunks.Source: §3.1 · p. 2, §3.2 · p. 3

3.3 Stage I: Encoder Distillation

  • D2 distills the unchanged offline encoder into a streaming student, preserving bidirectional attention within chunks while making attention causal across chunks.The student recomputes the convolutional frontend using only audio available at the current chunk.Source: §3.3 · pp. 3–4

3.4 Full-Duplex Inference

  • During inference, D2 processes newly accumulated user audio alongside the preceding chunk’s self-generated audio, then autoregressively generates the next k output frames.User audio is captured continuously, including while the model speaks; increasing k makes interaction coarser and less responsive.Source: §3.4 · p. 4

3.5 Stage II: D2 SFT

  • D2 supervised fine-tuning trains the pretrained model on temporally aligned chunks, weighting response-initiation and interruption outputs more heavily than padding.Each chunk pairs newly observed user audio and preceding assistant audio with target output frames, using explicit control tokens for response initiation and interruption.Source: §3.5 · p. 4

4 Experimental Setup

The experimental setup applies D2 to Qwen3-Omni and LLaMA-Omni 2, trains it on speech, general-audio, and constructed dialogue data, and evaluates interaction and general spoken-language capabilities.

4 Experimental Setup

  • The setup specifies D2 backbones, datasets, training procedures, and evaluation benchmarks.The following subsections describe each component.Source: §4 · p. 4

4.1 Half-Duplex Models

  • D2 is applied to Qwen3-Omni and LLaMA-Omni 2, whose audio representations have temporal resolutions of 12.5 Hz and 10 Hz, respectively.D2 uses Qwen3-Omni’s audio input pathway and adapts the Whisper-large-v3 encoder in LLaMA-Omni 2.Source: §4.1 · pp. 4–5

4.2 Datasets

  • Training uses LibriSpeech and AVQA for encoder distillation, then InstructS2S-200K, SODA, and AVQA for D2 training.Dialogue turns are concatenated into sequences spanning at least 720 seconds, with transitions sampled as interruptions, direct concatenations, or silent gaps.Source: §4.2 · p. 5, Figure 3 · p. 5

4.3 Implementation Details

  • Encoder distillation runs for 30,000 updates and D2 training for 10,000 updates, each with batch size four.D2 training uses 90-second segments and carries a detached 90-second KV cache; response and interruption control tokens receive higher weights than ordinary lexical tokens.Source: §4.3 · p. 5

4.4 Benchmarks

  • Full-Duplex-Bench measures turn taking, interruption handling, and pause handling, while VoiceBench assesses general spoken-language capabilities across nine sub-benchmarks.VoiceBench evaluation uses the first 200 samples from each sub-benchmark.Source: §4.4 · pp. 5–6

5 Results

D2 models show strong full-duplex interaction across chunk sizes, while Qwen3-Omni-D2 leads evaluated full-duplex models on VoiceBench. In the tested inference setting, computation remains below chunk duration.

5.1 Full-Duplex-Bench

  • At the smallest chunk size, Qwen3-Omni-D2 reaches 521-ms turn-taking and 403-ms interruption latencies, while LLaMA-Omni 2-D2 reaches 746 ms and 621 ms.Turn-taking and interruption take-over rates remain close to 100% across chunk sizes; larger chunks increase latency but generally improve post-interruption response quality.Source: §5.1 · p. 6

5.2 VoiceBench

  • 70.1 at 80 ms puts Qwen3-Omni-D2 about 9.1 points above MiniCPM-o 4.5, and it has the highest overall VoiceBench score among evaluated full-duplex models at every tested duration.At 1040 ms, its score is 73.0, comparable to half-duplex Kimi-Audio’s 72.3.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3 Computation Analysis

  • Mean and p95 inference computation times remain below chunk duration at every tested chunk size on a single RTX PRO 6000 GPU using BF16 and batch size one.Larger chunks yield lower real-time factors by amortizing audio-encoder computation over larger input chunks.Source: §5.3 · p. 7

6 Ablation

Stage-I encoder adaptation improves LLaMA-Omni 2-D2 results, while control-signal and PAD-token weighting support turn control. The ablations assess these choices at a 100-ms chunk duration.

6.1 Stage-I Encoder Distillation

  • Distillation needs only the pretrained encoder, enabling conversion when the original training decoder is unavailable; when available, Stage I can instead use the original speech-recognition objective.The evaluation compares both approaches with a baseline that omits Stage I.Source: §6.1 · p. 7
  • Omitting Stage I lowers LLaMA-Omni 2-D2 VoiceBench from 34.7 to 30.9 and post-interruption quality from 3.33 to 2.75 at 100 ms.Supervised encoder fine-tuning has a slightly higher VoiceBench score but lower post-interruption quality than distillation.Source: §6.1 · p. 7

6.2 Control Signal

  • Removing either control-signal or PAD-token weighting lowers turn-taking TOR from 100% to 95.8%; removing both lowers turn-taking and interruption TOR to 62.2% and 90.5%.The original weighting also gives the highest post-interruption score.Source: §6.2 · p. 8

7 Conclusion

D2 converts pretrained half-duplex spoken language models into full-duplex models using streaming encoder distillation and supervised fine-tuning. Across two architectures, it achieves strong interaction results and state-of-the-art VoiceBench performance among evaluated full-duplex models.

7 Conclusion

  • D2 converts pretrained HD-SLMs into FD-SLMs through streaming encoder distillation followed by full-duplex supervised fine-tuning.The framework uses several hundred hours of training data per stage.Source: §7 · p. 8
  • Across two different HD-SLM architectures, D2 achieves strong results on full-duplex interaction benchmarks and state-of-the-art VoiceBench performance among evaluated full-duplex models.Source: §7 · p. 8
  • D2 retains substantial general capabilities of the original HD-SLM, narrowing the capability gap between full- and half-duplex models.Source: §7 · p. 8

Limitations

The paper identifies latency, training-data availability, and pause handling as limitations of D2.

Limitations

  • Speech synthesis contributes a large portion of end-to-end latency, constraining backbone choice to models with sufficiently low-latency synthesis.Reducing synthesis latency often degrades speech quality or causes unstable generation.Source: Limitations · p. 8
  • Full-duplex training data is constructed from turn-based conversations because high-quality public full-duplex dialogue data is currently limited.Source: Limitations · p. 8
  • Pause handling remains limited, particularly for Qwen3-Omni-D2 at small chunk durations.Source: Limitations · p. 8

Ethical Considerations

The paper identifies privacy, consent, impersonation, and inherited-bias risks in full-duplex spoken language models. It also details real-time measurement, streaming encoders, and chunk-based training and inference.

Ethical Considerations

  • Continuous audio processing raises privacy and consent concerns, while low-latency interaction may facilitate impersonation, social engineering, and deceptive conversations.D2 may also inherit biases and harmful behaviors; the paper recommends consent, transparency, privacy-preserving processing, and safeguards against misuse.Source: Ethical Considerations · p. 9

A Real-Time Computation

  • Timing runs from CPU input PCM to CPU output PCM, including both streaming encoders and the Thinker, Talker, and incremental Code2Wav.The timed region excludes initialization, graph capture, cache reset, audio acquisition, transport, and playback.Source: A Real-Time Computation · p. 12
  • RTF is computation time divided by chunk duration; measurements use chunk sizes k ∈ {1, 2, 4, 8, 13}, starting from zero conversational history.The protocol reports the mean and 95th percentile; the setup uses one GPU, one CPU core, batch size one, and synthetic PCM inputs.Source: A Real-Time Computation · p. 12

B.1 Stage-I Streaming Encoder

  • Qwen3-Omni streaming encoding attends bidirectionally within each current chunk and retains at most 104 − k earlier positions, with future chunks absent.Stage-I training uses the same attention pattern and distills the student output against the frozen original encoder.Source: §B.1 · p. 12
  • LLaMA-Omni 2 streaming encoding concatenates five Transformer rows per embedding and recomputes block-causal attention within a window, rather than using Qwen’s KV cache.It processes 10k new Mel frames into 5k encoder rows per chunk.Source: §B.1 · p. 12

B.2 Qwen3-Omni-D2 Training and Inference

  • Qwen training serializes user audio, delayed self-audio, and text inputs in causal order, while keeping the two audio-stream encoder states independent.The Thinker input has shape [B, 3T, 2048], and learned stream embeddings distinguish user and self audio.Source: §B.2 · pp. 12–13
  • Text and codec targets share an 80-ms clock, with RESPONSE and INTERRUPT offsets fixed at eight native frames independently of chunk size.The first lexical token and model audio start one frame after RESPONSE; training self-audio includes a measured playback shift followed by a one-chunk feedback delay.Source: §B.2 · p. 12
  • At inference, each chunk combines current user audio with the preceding chunk’s model audio, then emits speech frame by frame and feeds rendered audio into the next chunk.RESPONSE opens the Talker, INTERRUPT closes it with EOS, and PAD ends lexical generation without itself closing the Talker.Source: §B.2 · pp. 13–14

C Training and Reproducibility Details

Training uses staged optimization and scheduled mixtures of speech and AVQA data. Dataset construction synthesizes conversations, creates full-duplex turn patterns, and applies acoustic augmentation to user audio.

C Training and Reproducibility Details

  • Both training stages use AdamW with gradient clipping at 1.0; Stage II starts from the same-chunk-size Stage-I encoder and adds fresh attention adapters over frozen dense attention.Qwen learning rates reach their floors at update 2,000, while LLaMA-Omni 2 uses cosine decay to 10% of peak at update 10,000.Source: C Training and Reproducibility Details · pp. 14–15
  • Stage I alternates four LibriSpeech batches with one AVQA batch, while Stage II cycles two InstructS2S cohorts, one SODA cohort, and sixteen AVQA cohorts.Each cohort contains four examples, and each training segment produces one optimizer update without gradient accumulation.Source: C Training and Reproducibility Details · p. 14

D Dataset Construction

  • Stage I uses audio-only LibriSpeech and AVQA data, while Stage II synthesizes both conversation roles using 904 user voice recipes and the fixed af_heart model voice.AVQA inputs concatenate an audio clip, spoken question, and answer choices; the model speaks the correct letter and complete choice text.Source: D Dataset Construction · pp. 15–16
  • Full-duplex sequences preserve turn order and sample interruption, direct concatenation, or silence at model-to-user transitions with probabilities 0.5, 0.3, and 0.2.User tracks receive noise and other acoustic augmentation, while self audio and speech targets remain clean.Source: D Dataset Construction · p. 16

E Speech-Quality Evaluation

Speech-quality evaluation compares D2 checkpoints with their original backbones on generated responses. The section also defines FD-Score as a latency-adjusted measure of turn-taking and interruption performance.

E Speech-Quality Evaluation

  • Evaluation uses the first user turn from 100 InstructS2S rows excluded from local D2 training, with all models responding to the same generated waveforms without teacher forcing.Responses are scored after removing outer silence only; internal pauses and streaming boundaries remain, and silent or sub-0.5-second crops receive no score.Source: E Speech-Quality Evaluation · pp. 16–17
  • 4.54 versus 4.46 UTMOS: Qwen3-Omni-D2 scored higher than its backbone on 99 mutually scorable prompts, while LLaMA-Omni 2-D2 scored 4.16 versus 4.43 on all 100 prompts.The table reports UTMOS mean ± sample standard deviation on shared prompts at chunk size k = 1.Source: E Speech-Quality Evaluation · p. 17, Table 10 · p. 17

F FD-Score Definition

  • FD-Score equally averages latency-adjusted turn-taking and interruption take-over rates, with higher scores indicating better performance under this definition.The two terms are computed separately and averaged before final rounding; the example uses Qwen3-Omni-D2 with 80-ms chunks.Source: F FD-Score Definition · p. 17
Loading 2610.17124v1…