DiffusionGemma Technical Report

Introduction

• AR token-by-token decoding is memory-bound at low concurrency; blockwise text diffusion shifts execution toward compute-bound generation, while existing options compromise speed, intelligence, or access.Source: §1 · p. 2
• Around 20 TPF versus ∼3-6 TPF for state-of-the-art speculative decoding, DiffusionGemma generates around 1,500 TPS on one H100 while refining 256-token blocks in about 12 passes.Third-party benchmarking by Unsloth reports speeds up to 2,000 TPS on an NVIDIA RTX 6000.Source: §1 · pp. 2–3
• Warm-started from Gemma 4 26B A4B MoE, DiffusionGemma uses under 10% of its training-token budget for bidirectional 256-token denoising SFT, then sampler distillation plus RL.Source: §1 · p. 3
• Despite a performance penalty versus the AR baseline, it retains multimodal understanding, long contexts, thinking mode, and AR generation, enabling request routing or hybrid decoding.Source: §1 · p. 3
• The Apache 2.0 open-weights release provides unrestricted parameters as a white-box research baseline and allows integration into experimental and commercial applications.Source: §1 · p. 4
• Its efficient architecture supports rapid domain-specific finetuning, with early applications spanning multilingual ASR and interactive radiology-report drafting (Van Puyvelde et al., 2026).Source: §1 · p. 4

2. Generative Text Modeling with Discrete Diffusion

• Discrete diffusion iteratively refines whole sequences in parallel, and operating directly on categorical states avoids the embedding-projection mismatches of continuous-text approaches.Unlike strictly left-to-right autoregressive generation, this approach can revise tokens using information from across the canvas.Source: §2 · pp. 4–5

2.1. Probability Paths and Denoising

• A continuous-time Markov chain interpolates from clean data at t = 0 to a canvas of independently uniform tokens at t = 1, enabling parallel refinement.The canvas is a length-C token sequence, and its corruption path connects the data distribution to a uniform prior over the joint state space.Source: §2.1 · p. 5

2.1.1. The Forward Process

• The forward process corrupts each token independently under a decreasing noise schedule, making uniform random replacement increasingly likely as time approaches 1.The closed-form process supplies corrupted training examples from clean text.Source: §2.1.1 · pp. 5–6

2.1.2. The Backward Denoising Process

• The reverse process uses a neural estimate of each clean token's posterior to sample successive states, but factorized updates can produce local inconsistencies.The pipeline encodes the prompt into a KV cache, denoises a canvas with bidirectional and cross-attention, then encodes and appends the clean canvas to the cache.Source: §2.1.2 · pp. 6–7, Figure 4 · p. 7

3. The DiffusionGemma Architecture

• Initialized from the public Gemma 4 26B A4B MoE checkpoint, the shared-weight encoder-decoder inherits extended context and multimodal understanding while reducing training compute relative to training diffusion from scratch.The architecture is an encoder-decoder transformer with shared weights (Vaswani et al., 2017); the checkpoint is from Gemma Team et al. (2026).Source: §3 · p. 7

3.1. Block-Autoregressive Generation

• The model denoises 256-token canvases in sequence, conditioning each on a KV cache of context and prior canvases that grows without re-encoding the history.A causal encoder and bidirectional canvas decoder enable cache appending while previously generated canvases remain fixed.Source: §3.1 · p. 8

3.2. The Denoising Framework at Inference

• Each denoising step predicts clean-token probabilities from the noisy canvas, KV cache, and self-conditioning signal, then samples an updated canvas using a temperature-scaled transition.Multinomial diffusion allows tokens in the current canvas to be revised, while tokens in prior canvases are frozen.Source: §3.2 · pp. 8–9

3.3. The DiffusionGemma Sampler

• The default entropy-bounded sampler accepts tokens from lowest to highest entropy under a 0.1 error budget, renoising remaining positions to preserve local exploration.Temperature is annealed linearly from 0.8 to 0.4, supporting exploration in highly noised states and stronger commitment as denoising proceeds.Source: §3.3 · pp. 9–10
• Approximately 12 effective denoising steps on average—rather than the 48-step maximum—yield a 4× latency reduction without sacrificing generation quality.Adaptive stopping requires both mean canvas entropy below 0.005 and identical deterministic predictions across consecutive steps; harder tasks require more steps.Source: §3.3 · p. 11, Figure 5 · p. 11

3.4. Inference Efficiency Metrics

• Effective Denoising Steps is the token-weighted average of steps across canvases, normalizing inference effort across generated sequences.Total Denoising Steps drives latency; without adaptive stopping, Effective Denoising Steps equals the denoising budget.Source: §3.4 · p. 12

3.5. Retained Autoregressive Capability

• In autoregressive mode, DiffusionGemma’s performance falls between its primary diffusion mode and the baseline Gemma 4 checkpoint.The final weights load into the original transformer architecture and use causal attention for standard AR generation.Source: §3.5 · p. 12

4. Supervised Finetuning

• The model is trained to predict 256-token blocks using discrete multinomial diffusion, uniformly sampled vocabulary-token corruption, and a uniformly sampled noise level t.A block-diagonal attention mask permits bidirectional attention within each block while preventing conditioning on other denoising blocks; the model also conditions on the prompt and previous uncorrupted tokens via an encoder KV cache.Source: §4 · p. 12
• SFT makes denoising possible; moderate training yields good non-thinking performance, while extended training is crucial for thinking behavior, whose log-linear gains start lower but rise more steeply.Before SFT, the model cannot denoise text; thinking traces initially stutter or cycle, and extended SFT improves them.Source: §4 · p. 12, Figure 6 · p. 13, Figure 7 · p. 13

Training setup

• A single online learning stage jointly maximizes reward and distills high-quality outputs into the few-step regime, using SFT-initialized teacher trajectories across thinking and non-thinking modes.The setup targets helpfulness, mathematical reasoning, coding, and instruction-following, and adapts the Gemma 4 RL recipe (Gemma Team et al., 2026).Source: §5 · pp. 13–14

Training dynamics & implicit curriculum effect

• As average reward rises, adaptive stopping requires fewer effective denoising steps; falling predictive entropy triggers earlier stopping and shifts training toward shorter trajectories.This implicit curriculum makes continued SD·RL beneficial for inference speed even after the reward metric plateaus.Source: §5 · p. 14, Figure 8 · p. 14

Improved speed-to-intelligence Pareto frontier

• SD·RL expands the SFT speed–quality frontier with a 10-point gain on the combined GPQA-Diamond and LiveCodeBench-v6 score and TPF rising from 5 to nearly 20.The comparison averages both benchmarks in thinking mode over 3 seeds; the SD·RL configuration uses a maximum of N=48 denoising steps.Source: §5 · p. 15, Figure 9 · p. 15

SD·RL specialization in the few-step regime

• After SD·RL, performance improves through N=48 and shows diminishing returns beyond it, while SFT performance scales up to N=192; SFT at N=48 can collapse into repetitive loops.Those loops prematurely trigger adaptive stopping, whereas post-SD·RL samples sustain coherent reasoning at low latency.Source: §5 · p. 15, Figure 10 · p. 16

Emergent conciseness

• The final checkpoint generates outputs nearly 2× shorter than SFT and uses less than 5% of the total forward passes of the Gemma 4 AR baseline across the evaluation suite.Fewer generated tokens combine with fewer effective denoising steps to reduce end-to-end latency, though shorter outputs forgo some gains associated with extended reasoning.Source: §5 · p. 15, Figure 9 · p. 15

• Fewer memory transfers can outweigh higher FLOPs per generated token, making text diffusion effective for low-batch, single-request latency on memory-bound serving hardware.The analysis compares batch-size-1 DiffusionGemma with Gemma 4 26B A4B; each denoising step processes a 256-token canvas, and throughput relative to AR depends on TPF divided by per-step slowdown.Source: §6 · p. 16
• 3.2× slower per step lets DiffusionGemma process 256× more tokens than single-token AR generation, with GPU time concentrated in MoE, sampling, and attention.MoE is 4.3× slower and attention 4× slower; sampling takes 3.06 ms versus AR sampling’s 0.56 ms. Dense architecture would reduce per-step feed-forward slowdown to less than 2×.Source: §6 · pp. 16–17, Figure 11 · p. 17
• Around 32 concurrent requests is where AR models begin to gain a throughput advantage; below that, DiffusionGemma has higher per-user TPS and total throughput.These results are not optimized for batch sizes above 1; kernel selection is suboptimal and sampling has not been tuned to scale with batch size.Source: Figure 12 · p. 18, §6 · pp. 18–19
• Asynchronous scheduling and a per-sequence causal attention flag handle adaptive stopping and mixed denoising/KV-cache-update passes without additional CPU-GPU synchronization.Source: §6 · p. 18
• 1,456 TPS is DiffusionGemma’s average single-request throughput, 7.1× Gemma 4 AR (204 TPS) and 4.8× Gemma 4 AR with MTP (303 TPS).This assumes an average 19.74 tokens per forward pass across seven benchmarks; on an H100 with FP8 and 4096 input tokens, a denoising step takes 13.56 ms end-to-end.Source: §6 · p. 18
• DiffusionGemma trades KV-cache data movement for MoE compute, potentially helping long-context, agentic workloads, but realistic-traffic performance remains unmeasured.It performs TPF× fewer KV-cache transfers and proportionally more MoE FLOPs as effective denoising steps increase.Source: §6 · p. 19

Evaluation setup

• The evaluation covers four operating modes—TD and AR, each with and without thinking—and benchmarks reasoning, coding, knowledge, multimodal understanding, instruction following, and agentic tasks.Comparisons include Gemma 4 26B A4B, LLaDA 2.1 Flash 100B, Nemotron Diffusion 14B, and Mercury 2; TD with thinking is enabled unless specified otherwise.Source: §7 · p. 19
• Figure 13 summarizes unweighted mean scores across reasoning and knowledge, coding, and instruction-following and agentic capability areas alongside output throughput.Models appear only when they completed every benchmark in an area, so missing bars indicate incomplete coverage, not a zero score; hatched bars mark no-think variants.Source: §7 · p. 19, Figure 13 · p. 20

Performance and speed

• Roughly 1,500 output tokens per second on one H100 gives DiffusionGemma a roughly 2.5× speedup over Mercury 2 while setting a new text-diffusion performance frontier.It substantially outperforms existing open-weight diffusion baselines, increases Tokens Per Forward by approximately an order of magnitude, and is highly competitive with Mercury 2 in quality.Source: §7 · pp. 19–20, Figure 13 · p. 20
• 1,479 tokens per second versus 303 for Gemma 4 under optimized MTP serving, while TD mode reduces capability scores; standard AR mode recovers part of the gap at lower throughput.The retained AR mode supports routing requests according to latency requirements and task complexity.Source: §7 · p. 20, Table 3 · p. 21

Case study: Sudoku puzzle solving

• With LoRA rank 8, fine-tuning exceeded 80% Sudoku accuracy, while the base model failed to produce a correct grid.The table notes that lower predictive entropy made denoising steps significantly less effective.Source: Table 5 · p. 23, §8 · p. 24

Open-source toolkit

• The released open-source toolkit provides LoRA recipes for adapting DiffusionGemma to domain-specific datasets on consumer hardware.It builds on Hackable Diffusion.Source: §8 · p. 23

Finetuning procedure

• The SFT objective sums causal-encoder cross-entropy over the sequence and diffusion-decoder denoising loss on a uniformly sampled canvas.The decoder uses the prompt and prior-canvas KV cache; half of batch examples use a self-conditioning state, while the rest use zero.Source: §8 · p. 23

LoRA for parameter-efficient finetuning

• LoRA is applied to attention projections, MLP gates, MoE routers, and the self-conditioning feedforward block, training only a small fraction of parameters on 2× A100 80GB GPUs.The toolkit’s LoRA setup is intended to deliver downstream performance with parameter-efficient training.Source: §8 · p. 23

Case study: Sudoku puzzle solving

• >85% puzzle-level accuracy was achieved with full fine-tuning on a held-out set of 4,096 Sudoku puzzles.The result uses the sampler in Algorithm 1 and is reported in Figure 14.Source: §8 · p. 24

9.1. Bidirectional Reasoning and Self-Correction

• DiffusionGemma’s bidirectional refinement lets emerging reasoning revise premature answers, converging on −25 where causal Gemma commits to −1 before its calculations.Figure 15 shows an intermediate −15 estimate and convergence after five denoising steps.Source: §9.1 · p. 24, Figure 15 · p. 25

9.2. Dynamic and Adaptive Computation

• Adaptive stopping uses fewer effective denoising steps on easier tasks and more on harder ones, while a configurable step cap trades latency for output precision.A lower step count makes the reverse process coarser and faster; a higher count enables finer generation at greater compute cost.Source: §9.2 · pp. 24, 26

9.3. Structured and Constrained Outputs

• Structured outputs such as JSON extraction and code debugging can converge in just two to three diffusion steps, unlike AR’s sequential O(N) generation.Parallel refinement exploits predictable syntax and lexical structure across the sequence.Source: §9.3 · p. 26

Limitations & Known Issues

• DiffusionGemma has lower absolute performance than its AR initialization, Gemma 4 26B A4B, due to warm-starting, short SFT, latency-focused SD·RL, and inherited design choices.The inherited architectural, optimization, and data-mixture decisions may be suboptimal for discrete diffusion.Source: §10 · p. 26
• The final checkpoint produces highly concise outputs, which increases inference speed but limits quality gains from longer reasoning traces.This brevity is an emergent behavior of the final checkpoint.Source: §10 · p. 26
• Rare repetitive loops or token stuttering persist despite SD·RL mitigation, as aggressively reduced denoising steps can compromise generation robustness.An example is endlessly repeating a common token such as “the.”Source: §10 · pp. 26–27
• On MMMU-Pro, unreliable closing thought tags in multimodal prompts can lower thinking-mode performance; the thinking score is 54.3 vs. 66.0 non-thinking.The issue was identified too late to fix in this release, even when the reasoning is correct.Source: §10 · p. 27
• Beyond ∼32 concurrent users, higher per-token compute cost gives AR models a throughput advantage; these results lack targeted batch-size optimization.At up to ∼32 concurrent users, DiffusionGemma outperforms Gemma 4 AR (with MTP) in per-user and total throughput; realistic-traffic analysis remains future work.Source: §10 · p. 27

11. Conclusion

• Around 1,500 tokens per second on a single H100 accompanies highly competitive reasoning and multimodal capabilities.The report says this establishes a new Pareto frontier for the speed-to-intelligence tradeoff.Source: §11 · p. 27
• Fine-tuning the Gemma 4 26B A4B autoregressive model with SFT and SD·RL provides a practical, compute-efficient path to text diffusion.Source: §11 · p. 27
• The experimental open-weight release includes reference implementations in HuggingFace Transformers and vLLM for community exploration and further optimization.The report invites specialized-task fine-tuning, novel sampling algorithms, and improved inference efficiency.Source: §11 · p. 27

Core contributors (workstream leads marked with ‘*’)

• Core contributors include Adrien Ali Taïga, James Assiene, Daniele Calandriello*, Rahma Chaabouni, João Gante*, Tamara von Glehn*, Nate Keating, Chris Knutsen, Martin Kukla*, Tianlin Liu, Ivan Lobov*, Ofir Nabati, João Gabriel Oliveira, and Nicolas Perez-Nieves*.Source: Core contributors (workstream leads… · p. 38

Finetuning framework

• Finetuning framework contributors include Omar Sanseviero, Piotr Stanczyk, Arnaud Doucet, Romuald Elie, Alexandre Galashov, Klaus Greff, Quentin Berthet, Andrew Campbell, Clément Crepy, and Valentin De Bortoli.Source: Finetuning framework · p. 38

Leads

• Leads include Alexis Jacq, David Ruhe, Yu-Han Wu, George Scrivener, Shantanu Thakoor, Sebastian Flennerhag, and Brendan O’Donoghue.Source: Leads · p. 38

Sponsors

• Named sponsors include Danilla Sinopalnikov and Gemma Team, alongside Demis Hassabis, Prateek Jain, Armand Joulin, and Koray Kavukcuoglu.Source: Sponsors · p. 38
• Other named sponsors are Olivier Bachem, Jeff Dean, Zoubin Ghahramani, Raia Hadsell, Marc’Aurelio Ranzato, and Oriol Vinyals.Source: Sponsors · p. 38

A.1. Continuous Diffusion for Text

• Continuous and hybrid text diffusion have resurged; recent advances suggest early failures reflected suboptimal spatial geometries and restrictive objectives, not inherent modality limits.Source: §A.1 · p. 39
• Embedded Language Flows outperform discrete baselines by remaining in continuous embeddings until discretization, while flow maps compress trajectories into one- or few-step generation (Hu et al., 2026; Lee et al., 2026; Potaptchik et al., 2026).Source: §A.1 · p. 39
• Cosmos and TEncDM use compressed, smooth latent spaces for Gaussian diffusion, while hyperspherical flows rotate token embeddings to avoid Gaussian corruption (He et al., 2024; Meshchaninov et al., 2025; Shabalin et al., 2025; Deschenaux and Gulcehre, 2026b).Source: §A.1 · p. 39
• CANDI decouples discrete and continuous corruption, while Score Entropy Discrete Diffusion uses continuous-time Markov chains to learn discrete-data probability ratios (Pynadath et al., 2026; Lou et al., 2024).Source: §A.1 · p. 39

A.2. Speculative Decoding

• DiffusionGemma operates as a standalone diffusion model, eliminating verification bottlenecks and scaling to longer generated canvases than draft-then-verify approaches.Unlike TiDAR, which combines diffusion drafting with AR verification, and DFlash, which uses a separate drafter, DiffusionGemma does not require verification.Source: §A.2 · p. 39
• Speculative decoding verifies draft sequences in parallel, but AR drafters remain sequential and independent parallel drafters can have suboptimal acceptance; DFlash also exhibits suffix decay (Chen et al., 2023a; Leviathan et al., 2023; Xia et al., 2024; Li et al., 2026; Cai et al., 2024; Chen et al., 2026).Source: §A.2 · p. 39

B. Samples Before and After SD·RL Training

• SD·RL training resolves the SFT model’s repetitive loops, enabling it to complete complex reasoning traces and produce correct answers in two GPQA-Diamond examples.The examples involve a biology question and a 2D harmonic oscillator derivation; the SFT generations loop and return empty final responses.Source: B. Samples Before and After SD·RL… · p. 39, Figure 16 · p. 40, Figure 17 · p. 41

C. Additional Open-Source Downstream Finetuning Results

• Downstream finetuning adapts DiffusionGemma to PubMedQA long-form explanations evaluated with BLEU, while the Sudoku LoRA recipe updates all linear layers at rank 8 using 8M parameters.The recipes use adapters on all linear layers and train for 2,000 steps; after Sudoku finetuning, the model outputs a correctly solved 9x9 grid in the response channel.Source: C. Additional Open-Source Downstream… · p. 42, Table 7 · p. 42, Figure 18 · p. 43

D. Prompt Formatting

• The chat template uses BOS and EOS tokens, role-tagged turns, multimodal image routing, thinking channels, and delimiters for JSON-like tool schemas.Thinking mode is enabled with <|think|>; image tokens are replaced by image features in the forward pass.Source: §D.1 · p. 44, §D.2 · p. 44, §D.3 · p. 44

E. Mercury 2 Speed Estimation

• Across seven benchmarks, Mercury 2 is estimated at 600 TPS with high reasoning effort and 547 TPS with medium effort; corrupted 50,000-token loops slightly bias estimates upward.For GPQA-Diamond and LiveCodeBench-v6, the estimated average is 489 TPS; task-level estimates are 452.6 TPS and 525.5 TPS, respectively.Source: E. Mercury 2 Speed Estimation · p. 46, Figure 19 · p. 46, Figure 20 · p. 47

F. Denoising trajectory sampled from DiffusionGemma

• A sampled short sentence describing text diffusion is generated in 7 denoising steps.The final response describes text diffusion as iterative refinement of random noise into a coherent message.Source: Figure 22 · p. 49

G.1. Bidirectional Reasoning and Self-Correction

• In non-thinking mode, DiffusionGemma revises an initially likely “Yes” to the correct “No” in six denoising steps, while Gemma 4’s “Yes” contradicts its own reasoning.The frog must land on the 20-foot mark, which sends it back to 5 feet and creates an infinite loop.Source: Figure 23 · p. 51, §G.1 · p. 51

G.2. More on Dynamic and Adaptive Computation

• DiffusionGemma resolves the entire 40-token sequence in 7 denoising steps, versus 40 discrete sequential steps for an AR model; on a static-input convolutional variant, it resolves the output in 4 steps.Source: §G.2 · p. 52

G.3. Structured and Constrained Outputs

• Schema-constrained JSON output converges in 2 denoising steps, and a localized code correction produces a bug-free implementation in 3 steps.The JSON example begins above 83% average confidence per token and reaches 100% by the second step; most code tokens are copied verbatim.Source: Figure 27 · p. 54, Figure 29 · p. 55