PaperlayerView exact revision on arXiv

Source-linked AI summary

TiDAR: Think in Diffusion, Talk in Autoregression

Jingyu Liu, Xin Dong, Zhifan Ye, Rishabh Mehta, Yonggan Fu, Vartika Singh, Jan Kautz, Ce Zhang, Pavlo Molchanov

arXiv:2511.08923v1cs.CLcs.AI

TL;DR

The paper asks whether diffusion’s parallel generation can be combined with autoregressive quality without sacrificing hardware efficiency. TiDAR answers with a hybrid model that drafts tokens diffusively and samples them autoregressively in one forward pass, achieving substantially higher throughput while maintaining competitive or near-AR quality. The authors report 4.71× and 5.91× more tokens per second at 1.5B and 8B scales, respectively.

  • Problem

    Diffusion language models offer parallel decoding but face quality degradation with increased parallelism, while autoregressive models provide stronger quality but underuse compute during memory-bound decoding.

  • Method

    TiDAR uses a sequence-level hybrid architecture with structured causal–bidirectional attention to draft tokens in diffusion and sample final outputs autoregressively within one model forward.

  • Results

    TiDAR produces 7.45 and 8.25 tokens per NFE and achieves 4.71× and 5.91× more tokens per second than AR at 1.5B and 8B scales while maintaining competitive quality.

  • Takeaways & Limitations

    TiDAR demonstrates that parallel diffusion drafting and autoregressive sampling can jointly improve generation efficiency without requiring a separate draft model.

  • Takeaways & Limitations

    The current implementation doubles training sequence length with appended mask tokens, so efficient long-context extensions are deferred to future work.

Abstract

from arXiv · show

Diffusion language models hold the promise of fast parallel generation, while autoregressive (AR) models typically excel in quality due to their causal structure aligning naturally with language modeling. This raises a fundamental question: can we achieve a synergy with high throughput, higher GPU utilization, and AR level quality? Existing methods fail to effectively balance these two aspects, either prioritizing AR using a weaker model for sequential drafting (speculative decoding), leading to lower drafting efficiency, or using some form of left-to-right (AR-like) decoding logic for diffusion, which still suffers from quality degradation and forfeits its potential parallelizability. We introduce TiDAR, a sequence-level hybrid architecture that drafts tokens (Thinking) in Diffusion and samples final outputs (Talking) AutoRegressively - all within a single forward pass using specially designed structured attention masks. This design exploits the free GPU compute density, achieving a strong balance between drafting and verification capacity. Moreover, TiDAR is designed to be serving-friendly (low overhead) as a standalone model. We extensively evaluate TiDAR against AR models, speculative decoding, and diffusion variants across generative and likelihood tasks at 1.5B and 8B scales. Thanks to the parallel drafting and sampling as well as exact KV cache support, TiDAR outperforms speculative decoding in measured throughput and surpasses diffusion models like Dream and Llada in both efficiency and quality. Most notably, TiDAR is the first architecture to close the quality gap with AR models while delivering 4.71x to 5.91x more tokens per second.

1. Introduction

TiDAR addresses the tension between diffusion’s parallelism and autoregressive quality by combining diffusion drafting with autoregressive sampling in one forward pass. It uses free token slots and hybrid attention to improve throughput while preserving quality.

  • Motivation: Autoregressive decoding is memory-bound, while diffusion can predict multiple masked tokens in parallel using additional free token slots before computation becomes latency-limiting.The extra slots share loaded weights and KV cache, so they can add little latency in the memory-bound regime.
  • Motivation: Diffusion models face a quality–parallelizability trade-off: decoding multiple tokens independently improves parallelism but can degrade sequence quality.Strict one-token decoding can preserve quality, but it removes much of diffusion’s parallel-generation advantage.
  • TiDAR: TiDAR partitions each generation step into prefix tokens, previously proposed tokens, and newly pre-drafted tokens, then combines diffusion drafting with autoregressive rejection-sampled outputs.The design reuses prefix-token KV caches within a single forward pass.
  • TiDAR: A structured causal–bidirectional attention mask lets one model learn both autoregressive and diffusion behavior, with both losses computed from the same training sample.During training, tokens in the diffusion section are masked.
  • Contribution: TiDAR is presented as a sequence-level hybrid architecture that combines parallel token drafting with autoregressive sampling to unite diffusion speed and autoregressive quality.The paper evaluates the approach across likelihood and generative tasks and reports ablations of its design choices.
  • Results: 4.71× and 5.91× relative throughput speedups are reported for TiDAR 1.5B and 8B, with lossless 1.5B quality and minimal 8B loss versus AR counterparts.These results are measured in tokens per second.

2. Background and Related Work

Diffusion language models offer parallel generation but often lose quality as more tokens are decoded simultaneously, while prior caching and speculative approaches leave efficiency or quality gaps. TiDAR combines a high-capacity base model with parallel drafting and simultaneous verification in one forward pass.

  • Diffusion Language Models: Diffusion models can accelerate generation through parallel token prediction, but increasing tokens per step generally reduces quality because independently sampled tokens weaken sequence-level coherence.Dream-7B accuracy on GSM8K drops by 10% when increasing from one to two tokens per step under entropy-based sampling.
  • Diffusion Language Models: Exact KV caching remains difficult for bidirectional diffusion models; approaches such as Fast-dLLM, d-KV cache, and Block Diffusion mitigate this issue but retain quality–parallelism trade-offs.Block Diffusion supports exact caching yet still suffers from intra-block quality degradation and limited parallelizability.
  • Speculative Decoding: Speculative decoding normally relies on a smaller draft model followed by target-model verification, so severe drafting-quality degradation can reduce overall speed.The comparison framework distinguishes separate high-capacity drafters from parallel or sequential drafting and verification.
  • Speculative Decoding: TiDAR uses one model to draft and sample simultaneously, allowing the base model itself to provide high-capacity drafting while avoiding sequential drafting–verification overhead.Its diffusion drafting uses all input mask tokens in parallel rather than only the final token.

3. Method

TiDAR combines diffusion drafting with autoregressive sampling in one forward pass, using structured attention and exact KV caching to improve efficiency while preserving quality.

  • 3.2. Fully Parallelizable Self-Speculative Generation: TiDAR performs parallel diffusion drafting and autoregressive sampling within a single forward pass using three token sections.Each step contains prefix tokens, previously proposed tokens, and tokens pre-drafted for the next step.
  • 3.1. Diffusion-AR Dual-mode Backbone Training: A hybrid causal–bidirectional attention mask lets one model represent both autoregressive and diffusion distributions and compute both losses on the same sample.The diffusion section uses fully masked tokens, producing a denser diffusion loss and simplifying objective balancing.
  • 3.2. Fully Parallelizable Self-Speculative Generation: One-step diffusion drafting supplies proposals in parallel, while rejection sampling autoregressively verifies them to preserve generative quality.The method uses one diffusion step because it produces drafts sufficient for a high acceptance rate.
  • 3.3. Training and Inference Optimization: TiDAR reuses exact KV caches and slices a pre-initialized attention mask during inference without recomputing cached token representations.Causally computed KV entries are stored and evicted only when corresponding sampled tokens are rejected.

4. Experiments

The experiments evaluate TiDAR through generative and likelihood tasks, measuring both quality and single-example decoding efficiency across multiple model sizes and baselines.

  • 4.1. Setup: Evaluations cover coding, mathematics, factual knowledge, and commonsense reasoning across generative and likelihood tasks.The task suite includes HumanEval, MBPP, GSM8K, Minerva Math, MMLU, ARC, Hellaswag, PIQA, and Winogrande.
  • 4.1. Setup: Efficiency is measured by tokens produced per model forward and wall-clock tokens per second at batch size one.The study reports average tokens per network function evaluation alongside throughput speedup.
  • 4.1. Setup: Experiments compare TiDAR with similarly sized autoregressive models, Dream, Llada, and Block Diffusion trained under the same recipe.Qwen2.5 and Qwen3 results use base models unless explicitly specified.

4.2. Main Results

TiDAR maintains competitive generative quality across coding and math tasks while supporting efficient likelihood evaluation through its autoregressive mode. It also balances quality and efficiency across diffusion and autoregressive comparisons.

  • TiDAR is highly competitive at 1.5B–1.7B with 7.45 tokens per model forward and incurs minimal quality loss at 8B with 8.25 tokens per forward.The evaluation covers coding and math tasks across two model sizes.
  • TiDAR consistently outperforms Dream, Llada, and Block Diffusion when diffusion models decode one token per forward for best quality.The comparison uses the same training recipe for Block Diffusion.
  • TiDAR balances generative quality and efficiency, making it suitable for application scenarios with stringent latency requirements.
  • TiDAR’s autoregressive mode supports likelihood evaluation aligned with autoregressive models, remains competitive, and requires a single NFE.Traditional diffusion likelihood evaluation uses Monte Carlo sampling and remains difficult to compare directly.

4.3. Efficiency Benchmarking

TiDAR delivers substantial measured throughput gains against autoregressive, speculative-decoding, and diffusion baselines while maintaining comparable performance. The benchmark uses exact caching on a single H100 GPU with batch size one.

  • The wall-clock benchmark compares TiDAR, AR, EAGLE-32 speculative decoding, and Block Diffusion with exact caching on one H100 GPU at batch size one.
  • TiDAR provides better efficiency–quality trade-offs than Block Diffusion across all tested tasks and surpasses the measured efficiency gains of EAGLE-3 speculative decoding.TiDAR’s raw acceptance rate and conversion from tokens per NFE to tokens per second are higher than those of the tested EAGLE-3 weights.
  • All evaluated methods could benefit from custom kernels, improved KV-cache management, and request scheduling.

4.4. Ablation Studies

TiDAR’s ablations attribute its quality–efficiency profile to parallel diffusion drafting, autoregressive sampling, balanced verification, and full-mask training. Across controlled comparisons, it improves the efficiency–quality frontier without decoding hyperparameter tuning.

  • 4.4. Ablation Studies: The proposed parallel draft-and-sampling process outperforms confidence-, negative-entropy-, and autoregressive decoding strategies without requiring decoding hyperparameter tuning.
  • 4.4.3. Sampling with AR v.s. Diffusion Prediction: TiDAR achieves the best 1.5B Pareto frontier against AR and Block Diffusion and approaches fine-tuned AR quality with 7x more tokens per NFE.The comparison uses the same training recipe and varies TiDAR drafting lengths and Block Diffusion thresholds.
  • 4.4.3. Sampling with AR v.s. Diffusion Prediction: Varying the AR–diffusion logit balance preserves performance, indicating that autoregressive rejection sampling maintains the quality–speed trade-off.The analysis varies β across different loss-balancing α values.
  • 4.4.4. How Useful is the Full Mask Strategy?: Full-mask training improves quality and efficiency by reducing train–test discrepancy and providing richer diffusion loss signals.The improvement is especially consistent in coding tasks and also enables flexible selection of predictions for rejection sampling.

5. Limitations

The paper identifies batch-size benchmarking, long-context training overhead, and further system optimization as practical scope boundaries.

  • Batch-size evaluation focuses on batch size = 1, although TiDAR can adjust draft length for different compute profiles.The authors also report competitive FLOPs per token for larger-batch settings.
  • Long-context extension remains future work because training currently doubles sequence length with appended mask tokens.The authors do not consider TiDAR intrinsically limited in long-context capability compared with standard AR models.
  • Custom attention kernels and scheduling algorithms may further improve throughput beyond the substantial gains already achieved with native PyTorch and Flex Attention.

6. Conclusions

TiDAR combines diffusion drafting with autoregressive sampling in one forward pass to balance generation efficiency and quality. Across model scales, it delivers substantially higher throughput while maintaining competitive quality.

  • TiDAR drafts tokens in diffusion and samples them autoregressively within a single model forward using a specially designed attention mask.
  • 4.71× and 5.91× more tokens per second than AR are achieved by TiDAR 1.5B and 8B, respectively, with competitive quality.The models average 7.45 and 8.25 tokens per NFE, respectively.
  • The architecture exploits parallel one-step diffusion for efficiency and autoregressive sampling for quality, and is reported to beat speculative decoding for latency-critical applications.

A. Evaluation Task Configuration

The evaluation configuration is summarized in Table 6, with task-specific prompt counts, generation lengths, and performance metrics. HumanEval tasks additionally use standard post-processing.

  • Table 6 specifies the number of few-shot prompts, generation length, and performance metric for each evaluated task.
  • The benchmark configuration covers the tasks adopted in the work through a common tabulated setup.
  • HumanEval and HumanEval Plus use standard post-processing for all models, with lm_eval_harness version 0.4.8.

B. Inference Prefill Mask

The inference prefill mask is initialized once at a maximum size and reused across prompt lengths by reordering inputs and slicing the relevant region.

  • The implementation reuses one initialized attention mask across samples by reordering input components and slicing the larger mask.
  • This prefill-mask procedure avoids recomputing the full mask for each prompt length.
  • Figure 7 shows the prefill mask initialized with dimensions (max_seq_len + block_size, max_seq_len + block_size) and sliced for the current sample length.
Loading 2511.08923v1…