PaperlayerView exact revision on arXiv

Source-linked AI summary

Full-duplex Spoken Language Modeling with Discrete Diffusion

Chen Chen, Pin-Jui Ku, Ankita Pasad, Ante Jukić, Zhehuai Chen, Boris Ginsburg

arXiv:2610.17780v1cs.CL

TL;DR

Full-duplex spoken models can provide fluid interaction while losing linguistic capability by binding response generation to real-time audio processing. Diffuplex separates conversational timing from response generation, using an autoregressive policy and parallel diffusion. With thinking enabled, it reports strong VoiceBench performance and fast generation; it also reports a trade-off in jointly adapting speech and language capabilities.

  • Problem

    Full-duplex models can be responsive but exhibit linguistic and cognitive degradation relative to frontier autoregressive models when conversational timing and response generation share one autoregressive timeline.

    Source: §1 · p. 2

  • Method

    Diffuplex uses an autoregressive policy to track incoming speech and determine when to respond, while a speech-conditioned diffusion model independently generates response content.

    Source: §1 · p. 2

  • Results

    With thinking enabled, Diffuplex scores 86.9 across nine VoiceBench subsets and finalizes an average of 13.0 tokens per forward pass, while turn-taking and barge-in latency are comparable to strong full-duplex models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results show that separating conversational control from response generation can support spoken intelligence alongside responsive full-duplex interaction.

    Source: §6 · p. 10

    Joint adaptation presents a trade-off: lower learning rates weakly ground responses in speech, while higher rates cause pretrained language-distribution drift, repetition, and incorrect termination.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

We introduce Diffuplex, a full-duplex spoken language model based on discrete diffusion that achieves an unprecedented balance between linguistic intelligence and response efficiency. With thinking mode, Diffuplex delivers 13.0 tokens per forward pass, reaching an average of 1872 tokens per second and substantially surpassing the frontier autoregressive (AR) models in decoding speed. Without thinking, Diffuplex matches the first-audio latency of existing full-duplex models while achieving markedly stronger linguistic intelligence. This capability is enabled by separating when to say from what to say at the training level, rather than compressing both into a single autoregressive timeline. During inference, Diffuplex continuously tracks the user speech through an AR interaction policy, while independently generating and refining response content through parallel diffusion. Consequently, semantic computation is no longer bounded by the perception audio frame rate, allowing intelligence and responsiveness to improve simultaneously. Taken together, these results establish a new paradigm for spoken agents that can sustain real-time dialogue while devoting parallel computation to intelligent response generation. Training code and checkpoints will be avaliable soon.

1 INTRODUCTION

Full-duplex models can be responsive but often sacrifice linguistic capability by coupling conversational timing and response generation to one autoregressive timeline. Diffuplex separates these functions, using autoregressive interaction control and parallel diffusion to generate responses.

1 INTRODUCTION

  • Conversational timing and response formulation should be coordinated without being forced into a single sequential process.The introduction frames fluid conversation as requiring both tracking the interlocutor and progressively formulating internal thoughts.Source: §1 · p. 2
  • Full-duplex models often show linguistic and cognitive degradation relative to frontier autoregressive models because one timeline couples timing and semantic generation.The authors identify conflicting training objectives and binding semantic computation to the physical audio frame rate as reasons for this coupling.Source: §1 · p. 2
  • Diffusion enables response tokens to be generated and refined in parallel, independently of physical time, after an autoregressive policy determines when speaking begins.An anchored first n tokens can be synthesized early while the rest of the response canvas continues denoising.Source: §1 · p. 2
  • Diffuplex combines a pretrained speech encoder and text diffusion model, with autoregressive interaction control and diffusion response generation sharing a backbone.The two modes respectively determine when to speak and what to say.Source: §1 · p. 2
  • With thinking enabled, Diffuplex scores 86.9 across nine VoiceBench subsets and finalizes 13.0 tokens per forward pass, while maintaining full-duplex turn-taking and barge-in latency comparable to strong models.It exceeds existing open-source systems by more than 20 points in spoken intelligence.Source: §1 · p. 2
  • Diffuplex averages 1,872 tokens per second on one GB300 GPU and far outpaces its autoregressive counterparts.Its autoregressive policy tracks streaming speech while diffusion generates response content.Source: §1 · p. 2

2 RELATED WORK

Prior spoken models support turn-based or full-duplex interaction, while diffusion language models have been extended to spoken input primarily for understanding and transcription. Diffuplex combines streaming conversational control with parallel diffusion response generation.

2.1 SPOKEN LANGUAGE MODELS

  • Turn-based spoken models extend text language models with speech encoders or discrete acoustic tokens, but largely retain turn-oriented interaction and autoregressive response generation.Examples include AudioPaLM, SpeechGPT, and Qwen2.5-Omni.Source: §2.1 · p. 3
  • Full-duplex systems support concurrent user and agent speech, but typically bind conversational control and semantic generation to the same frame-synchronous autoregressive process.Diffuplex instead uses an autoregressive timing policy and a diffusion model for response content.Source: §2.1 · p. 3

2.2 DIFFUSION LANGUAGE MODELS WITH SPOKEN INPUT

  • Diffusion language models generate text through iterative refinement, and DiffusionGemma refines 256-token canvases in parallel while retaining autoregressive capabilities.DiffusionGemma converts a pretrained autoregressive mixture-of-experts model into a discrete diffusion language model.Source: §2.2 · p. 3
  • Diffusion models for spoken input have addressed speech recognition and audio understanding, but not full-duplex interaction or an explicit separation of timing from response generation.Diffuplex combines a streaming autoregressive interaction policy with parallel diffusion response generation.Source: §2.2 · p. 3

3 METHOD

Diffuplex separates streaming conversational control from response generation through a three-stage training curriculum and two decoupled inference modes. Anchored-prefix decoding and a modular synthesis interface let speech begin before response denoising is complete.

3 METHOD

  • Diffuplex trains in three stages: speech–text alignment, spoken-instruction response learning, and an autoregressive interaction policy for conversational timing.At inference, an autoregressive branch processes incoming speech while a diffusion branch generates responses from accumulated context.Source: §3 · pp. 3–4

3.1 STAGE 1: MODALITY ALIGNMENT

  • The Stage 1 model reaches 2.3 WER on LibriSpeech test-clean and 4.2 WER on test-other.The stage aligns speech representations to the pretrained text diffusion model using speech–text data and an auxiliary autoregressive next-token loss.Source: §3.1 · p. 4

3.2 STAGE 2: SPOKEN INSTRUCTION-FOLLOWING

  • Joint adaptation creates a trade-off: lower learning rates preserve coherent long-form generation but weakly ground responses in speech, while higher rates improve adaptation but cause language-distribution drift.The authors report that separate learning rates for the three modules or text-only replay do not resolve this tension.Source: §3.2 · p. 4
  • Stage 2 uses a frozen textual anchor to transfer the pretrained model’s textual capabilities to the speech-conditioned student.The student receives spoken requests and the teacher their written counterparts; forward KL divergence is minimized for diffusion-canvas and autoregressive-readout distributions.Source: §3.2 · pp. 4–5

3.3 STAGE 3: FULL-DUPLEX INTERACTION MODELING

  • The Stage 3 interaction policy predicts four states—<listen>, <speak>, <bos>, and <eos>—from continuous speech divided into 80-ms chunks.The states represent listening, retaining the floor, taking the floor, and yielding it; transition events receive larger loss weights than persistent states.Source: §3.3 · p. 5

3.4 INFERENCE AND DENOISE

  • At inference, an autoregressive branch tracks speech while a diffusion branch refines 256-token response canvases, committing the first eight tokens after at most K passes.The anchored prefix is streamed to synthesis while the remaining canvas continues denoising; the number of passes trades opening latency against its quality.Source: §3.4 · pp. 5–6

3.5 SPEECH SYNTHESIZER

  • Diffuplex can pass its early finalized prefix to any high-quality TTS model, without requiring native streaming inference from the synthesizer.The synthesis choice can trade off quality, speaker flexibility, and latency without retraining Diffuplex.Source: §3.5 · p. 6
  • If the user interrupts within one second, Diffuplex discards the unspoken response canvas rather than appending it to the next interaction’s context.This retains only content that has effectively entered the conversation.Source: §3.5 · p. 6

4 EXPERIMENTAL SETUP

The experiments assess spoken intelligence and real-time conversational behavior against turn-based and full-duplex model classes. They use VoiceBench and Full-Duplex-Bench variants, alongside specified training and evaluation settings.

4.1 TRAINING DETAILS

  • The training details section directs readers to Appendix C for the data, configurations, computing resources, and optimization settings for all three stages.The supplied passage does not specify those settings.Source: §4.1 · p. 6

4.2 BENCHMARK AND EVALUATION

  • VoiceBench evaluates spoken intelligence across nine subsets covering open-ended dialogue, knowledge and reasoning, instruction following, and safety.Its average metric combines LLM judge scores and accuracy, with both real and synthetic speech and varied speakers, acoustic environments, and linguistic content.Source: §4.2 · p. 6
  • Full-Duplex-Bench v1 tests pause handling, backchanneling, smooth turn-taking, and user interruption, with a turn-taking-dynamics track added from v3.The v3 track uses recordings from human speakers with conversational disfluencies.Source: §4.2 · p. 6

4.3 BASELINE

  • The baselines include turn-based autoregressive models, turn-based diffusion models, and full-duplex autoregressive models.Thinking is enabled for turn-based autoregressive models where supported; full-duplex autoregressive systems have no separate thinking phase because their steps are tied to audio chunks.Source: §4.3 · pp. 6–7
  • Table 1 reports VoiceBench results across turn-based AR and full-duplex models; open-ended QA uses a 5-point judge scale, while remaining columns and Overall use 0–100 scales.Source: Table 1 · p. 7

5 EVALUATION AND RESULTS

Diffuplex is evaluated for spoken-response quality, conversational behavior, decoding efficiency, and capability transfer through ablations. Results show strong spoken intelligence and speed, with conversation variants trading response timing against pause handling.

5.1 RESULTS ON SPOKEN INTELLIGENCE

  • Stage 3 scores 82.3 without thinking and exceeds the strongest full-duplex AR baseline by 24.2 points; Stage 2 scores 86.9 with thinking.Stage 2 scores 84.1 without thinking; the Stage 3 score is 1.8 points lower, a comparison that does not isolate the effects of streaming decisions and early-prefix decoding.Source: §5.1 · p. 7
  • The thinking-enabled Nemotron-Labs-Diffusion-8B speech-conditioned model reaches an approximately 78.7 VoiceBench overall score, showing transfer of the strategy to a second diffusion backbone.The authors attribute its lower score primarily to weaker text-model capability than DiffusionGemma.Source: §5.1 · p. 7

5.2 RESULTS ON CONVERSATION BEHAVIOR

  • Diffuplex achieves the strongest turn-taking accuracy on both benchmark versions and responds correctly to interruptions, while Fisher training favors faster transitions at the cost of pause handling.The two variants offer operating points between interaction latency and robustness to natural hesitation and disfluency.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 TPF AND TPS ANALYSIS

  • 13.0 tokens per forward pass and up to 1,872 tokens/s compare with 1.00 token per pass for evaluated AR baselines and throughputs of 394 for Nemotron-3 Omni and 300 for Qwen3-Omni-Thinking.The throughput comparison also shows 4B active parameters for Diffuplex versus 3B for each of those two models; each diffusion pass processes a 256-token canvas.Source: §5.3 · p. 8, Figure 3 · p. 9

5.4 ABLATION AND MORE RESULTS

  • The text-input model scores 90.1 versus Diffuplex’s 84.1 with spoken input; removing Stage 1, the text anchor, or DLM adaptation yields 29.9, 66.1, or 64.7, respectively.The ablations indicate that acoustic grounding, the text anchor, and controlled language-backbone adaptation each matter for spoken instruction following.Source: §5.4 · p. 9, Table 3 · p. 9
  • Diffuplex also responds to spoken requests in AR mode, and its AR performance is better than diffusion decoding in the reported ablation.Appendix E separately examines how the pass cap K for committing the first n tokens affects first-audio latency and VoiceBench scores.Source: §5.4 · p. 9
  • Table 4 groups Diffuplex’s broader benchmark evaluation by capability, separating spoken audio-input benchmarks from text benchmarks.The reported capability areas include reasoning, knowledge, and instruction following.Source: §5.4 · p. 9, Table 4 · p. 10

6 CONCLUSION

Diffuplex separates conversational control from semantic generation: an AR policy tracks speech and determines when to speak, while a speech-conditioned discrete diffusion model generates what to say. The paper concludes that this two-axis design narrows the capability gap between responsive full-duplex systems and intelligent turn-based models.

6 CONCLUSION

  • Diffuplex decouples conversational control from semantic generation in a full-duplex spoken language model.An AR policy determines when to speak while a speech-conditioned DLM independently generates response content through parallel refinement.Source: §6 · p. 10
  • Separating semantic computation from the physical audio frame rate preserves strong text-model capabilities while supporting low-latency interaction.The conclusion also identifies anchored-prefix decoding and modular speech synthesis as part of this design.Source: §6 · p. 10
  • Experiments across spoken-understanding and full-duplex dialogue benchmarks substantially narrow the capability gap between responsive full-duplex systems and intelligent turn-based models.Source: §6 · p. 10

A FIGURE 1 DETAILS

The appendix reports decoding-speed measurements beyond the GB300 and discusses implementation choices, turn-policy training, and the scope of multi-token-prediction comparisons.

A FIGURE 1 DETAILS

  • Diffuplex’s TPF ranges from 13.0 to 13.7 on AlpacaEval across the reported RTX 6000 and H100 measurements.Source: A FIGURE 1 DETAILS · p. 16
  • The GB300 inference runtime uses NVFP4 for mixture-of-experts layers, FP8 for dense projections and the head, and FlashAttention-4 for attention.The model occupies 16.2 GB, reported as 3.2× less than bf16.Source: A FIGURE 1 DETAILS · p. 16

B MORE DISCUSSION

  • Without listening-mode LoRA, the interaction head mainly learns voice activity and fails to reliably distinguish within-turn hesitation from a true turn ending.The listening-mode LoRA gives the causal speech pathway flexibility to use conversational context while the diffusion response pathway remains fixed.Source: B MORE DISCUSSION · p. 16
  • Diffuplex’s 10.15 finalized answer-text tokens per backbone forward pass exceeds representative four-token text MTP configurations, but the comparison is not a universal upper bound on MTP.The cited speech-input MTP studies use different output units or tasks, so their token rates are not directly comparable to open-ended spoken question answering.Source: B MORE DISCUSSION · p. 17

C TRAINING DETAILS

The speech-conditioned response model is trained in three phases—Stage 1a, Stage 1b, and Stage 2—using a DiffusionGemma backbone with a Parakeet encoder and projection layer.

C TRAINING DETAILS

  • The three training phases are Stage 1a, Stage 1b, and Stage 2, using DiffusionGemma-26B-A4B-it with a Parakeet speech encoder and projection layer.Source: C TRAINING DETAILS · p. 17
  • The projection layer maps speech features into the backbone embedding space, and training uses a 256-token diffusion canvas with an auxiliary AR next-token objective.Source: C TRAINING DETAILS · p. 17
  • Audio is converted to 16-kHz mono waveforms before feature extraction, and the training seed is 42.Source: C TRAINING DETAILS · p. 17

C.1 STAGE 1A: PROJECTOR WARM-UP

Training proceeds from speech-to-text alignment through speech-encoder adaptation to spoken-instruction tuning, preparing the model for spoken input while retaining textual capabilities.

C.1 STAGE 1A: PROJECTOR WARM-UP

  • Stage 1a trains a projection layer on 281,099 LibriSpeech utterances while freezing the speech encoder and language model.Targets are lowercase transcripts without punctuation; training runs for two epochs.Source: §C.1 · p. 17

C.2 STAGE 1B: SPEECH ENCODER ADAPTATION

  • Stage 1b unfreezes the speech encoder and trains it with the projector on a 32-source English ASR mixture.Targets retain punctuation and capitalization, unlike Stage 1a.Source: §C.2 · p. 17

C.3 STAGE 2: SPOKEN INSTRUCTION TUNING

  • Stage 2 tunes the speech encoder, projector, and diffusion language model on spoken-instruction and ASR replay examples.A frozen language-model copy supplies forward KL losses to preserve pretrained textual capabilities; the MoE router remains frozen.Source: §C.3 · p. 17
  • Stages 1a, 1b, and 2 use BF16 computation across 64 GPUs, with FP32 optimizer parameters and gradient-norm clipping at 1.0.The selected Stage 2 checkpoint follows 2,266 optimizer updates.Source: §C.3 · p. 17

C.4 STAGE 3: FULL-DUPLEX INTERACTION TRAINING

Stage 3 trains interaction control using conversational speech, while the comparison set spans turn-based, diffusion-based, and full-duplex systems.

C.4 STAGE 3: FULL-DUPLEX INTERACTION TRAINING

  • Stage 3 combines 120 hours of clean synthetic conversations with 70 hours of internal conversation data, including hesitations, backchannels, and interruptions.A Fisher variant adds spontaneous exchanges; noise-augmented data include ambience, quiet noise, and short nonspeech events.Source: §C.4 · p. 18
  • Stage 3 trains a four-state interaction head and listening-mode LoRA adapter while keeping the response generator fixed.Cross-entropy uses 80-ms state labels, with extra weight on start- and stop-speaking transitions.Source: §C.4 · p. 18

D COMPARED MODELS AND THEIR SOURCES

  • Comparisons include turn-based speech models, DIFFA-2 as a speech-conditioned diffusion baseline, and full-duplex systems such as Moshi and PersonaPlex.The authors caution that systems address different operating regimes, so scores and latencies should be read with their evaluation settings.Source: D COMPARED MODELS AND THEIR SOURCES · p. 18

E SENSITIVITY OF EARLY PREFIX COMMITMENT

The sensitivity analysis varies denoising passes before committing an eight-token opening prefix and examines the resulting latency–quality trade-off.

E SENSITIVITY OF EARLY PREFIX COMMITMENT

Related papers

Loading 2610.17780v1…