Source-linked AI summary
Discrete Diffusion Language Models for Interactive Radiology Report Drafting
Max Van Puyvelde, Halil Ibrahim Gulluk, Wim Van Criekinge, Olivier Gevaert
TL;DR
Medical foundation models largely rely on autoregression, leaving diffusion’s accuracy and interactive drafting value in medicine uncertain. The paper compares matched diffusion and autoregressive backbones under identical LoRA adaptation on medical VQA and evaluates diffusion-based report infill. Diffusion matches or exceeds autoregression across datasets, is competitive with frontier VLMs, decodes 3.5–4.4× faster, and uses right-side context substantially better for gap filling.
Problem
Medical foundation models and radiology report generation are almost exclusively autoregressive, leaving diffusion’s medical viability and interactive drafting utility largely untested.
Method
The paper finetunes matched diffusion and autoregressive mixture-of-experts backbones with identical LoRA recipes, benchmarks medical VQA with an LLM judge, and evaluates bidirectional report infill.
Results
Diffusion matches or exceeds autoregression on all three medical VQA datasets, is competitive with frontier VLMs, and decodes 3.5–4.4× faster.
Takeaways & Limitations
Any-order infill lets radiologists fix report fragments and have diffusion fill gaps using context from both sides, a capability autoregression lacks.
Abstract
from arXiv · showhide
Diffusion language models, which generate text by denoising a token canvas bidirectionally instead of emitting tokens left to right, have become competitive with autoregressive (AR) generation. Medical foundation models, however, remain almost entirely autoregressive. We adapt a mixture-of-experts diffusion language model, DiffusionGemma-26B, and benchmark it against its same-size AR sibling Gemma-4-26B under an identical LoRA recipe on medical visual question answering datasets, scored by a verbosity-robust LLM judge. Diffusion matches or exceeds AR on all of them, and the finetuned model (3.8B active) is competitive with frontier vision-language models; its decoding is also 3.5-4.4x faster. Beyond this parity, the diffusion model offers a drafting capability AR lacks: any-order infill. Because the canvas is denoised bidirectionally, a radiologist can fix report fragments and have the model fill the text between them, an operation inherent to diffusion but not to autoregression, which is subpar at it. This suits real reports, which are often terse or inconsistent across clinicians and institutions.
1 Introduction
Diffusion language models offer a bidirectional alternative to autoregressive medical generation, whose clinical viability and interactive drafting utility remain largely untested. The paper evaluates matched diffusion and autoregressive backbones and introduces any-order infill for editing report fragments.
- Diffusion models iteratively denoise a fixed token canvas, allowing every position to attend to the entire canvas rather than only preceding tokens.
- Medical foundation models and radiology report generation remain dominated by autoregressive models, leaving diffusion models’ medical viability largely untested.
- The study finetunes diffusion and autoregressive siblings with identical LoRA recipes and benchmarks them against frontier vision-language models using a verbosity-robust LLM judge.
- Any-order infill lets radiologists fix report fragments at arbitrary positions and have diffusion fill gaps using context on both sides.This addresses variation in whether findings are stated and how report sections are organized across institutions.
- The paper reports diffusion parity with autoregression, competitive frontier-model performance, faster decoding, and stronger bidirectional gap filling, while releasing code and checkpoints.The contribution claims DiffusionGemma-26B equals or exceeds its autoregressive sibling and decodes 3.5–4.4× faster.
2 Related Work
Prior medical report-generation diffusion systems use bidirectionality for complete reports but do not establish matched diffusion-versus-autoregressive comparisons or interactive infill. The paper adopts LLM judging for open-ended medical VQA because exact match penalizes valid paraphrases.
- Existing radiology report-generation diffusion systems include AnchorDiff, MeDiM, and ECHO, but none isolate the paradigm against a matched autoregressive backbone or expose interactive infill.
- Medical VQA datasets pair radiology images with short open- or closed-ended questions, and LLM judges evaluate semantic equivalence when exact match penalizes valid paraphrases.
3 Method
The method compares same-size diffusion and autoregressive mixture-of-experts backbones under identical adaptation and data conditions. It implements infill by clamping fixed report tokens throughout bidirectional diffusion denoising.
- Matched Backbones: DiffusionGemma-26B and Gemma-4-26B are matched 25.2B/3.8B-active mixture-of-experts backbones with the same vision encoder and identical LoRA adaptation recipe.The recipe updates shared attention and MLP projections while freezing experts, router, and vision tower.
- Training Setup: The models condition on images while generating either short VQA answers or Findings-and-Impression report text within a single 256-token canvas.The image is never generated, and both paradigms are supervised only on target tokens.
- Training Objectives: Diffusion training replaces random target tokens with uniform vocabulary draws and learns to recover them, whereas autoregression uses next-token cross-entropy on the same targets.
- Any-Order Infill: Infill fixes tokens at positions F and samples the remaining positions from the report distribution conditioned on those fixed tokens.
- Any-Order Infill: During each denoising step, the diffusion sampler re-imposes fixed tokens before and after updating free positions, preserving them while enabling conditioning from both sides.
- Any-Order Infill: Autoregressive factorization cannot sample the same infill conditional because tokens cannot attend to positions after them.
4 Experiments
Experiments assess medical VQA accuracy, inference speed, and any-order infill. Diffusion matches or exceeds autoregression across medical VQA datasets, decodes faster, and benefits substantially more from right-side context when filling report gaps.
- The evaluation covers medical VQA accuracy, inference speed, and any-order infill using adapted diffusion and autoregressive backbones.
- Medical VQA: Medical VQA uses VQA-RAD, SLAKE, and VQA-Med-2019, with base and per-dataset finetuned models compared against three zero-shot frontier VLMs.Each model answers the same 350 held-out questions per dataset.
- Medical VQA: Semantic LLM judging replaces exact-match scoring because full-sentence answers can be correct despite differing from short references.Claude Sonnet 4.6 returns binary semantic-equivalence verdicts.
- Diffusion versus AR: Finetuned diffusion equals or exceeds finetuned AR on all three datasets, while base diffusion exceeds base AR on all three.The diffusion-versus-AR difference is significant on finetuned SLAKE and base VQA-RAD, but not on the other four comparisons.
- Frontier VLMs: The finetuned 26B diffusion model has 3.8B active parameters and is competitive with frontier VLMs, achieving the highest judge accuracy on SLAKE at 0.863.
- Inference Speed: DiffusionGemma-26B drafts 3.5–4.4× faster than its AR sibling and provides 5.7–7.1× higher throughput.The comparison favors AR because AR produces shorter natural outputs while diffusion fills the full canvas.
- Any-Order Infill: In MIMIC-CXR infill, adding right-side context raises diffusion token-F1 by +0.109 and judge accuracy by +0.129, whereas AR gains are not significant.The model×context interaction is significant on both metrics, indicating diffusion benefits about 3.5× more from bidirectional context.
5 Conclusion
The diffusion model matches or exceeds its autoregressive sibling on matched medical-VQA evaluation and decodes faster. It also provides any-order infill for interactive report drafting, exploiting context on both sides of a gap.
- Diffusion matches or exceeds its AR sibling on matched medical-VQA evaluation and decodes 3.5–4.4× faster.
- Any-order infill lets radiologists fix report fragments at arbitrary positions while the diffusion model fills the gaps between them.
- +0.109 token-F1 and +0.129 judge accuracy show diffusion exploiting context on both sides of a gap on MIMIC-CXR.AR does not achieve this even when right-side context is included in its prompt.
A Backbones and Adaptation Recipe
The two backbones use the same family, size, vision tower, LoRA targets, and data, isolating the generative paradigm and optimizer as differences.
- The adaptation recipe is identical in family, size, vision tower, LoRA targets, and data across both backbones.
- Only the generative paradigm and its established optimizer differ between the diffusion and autoregressive backbones.
B Datasets
The evaluation uses three medical-VQA datasets, with dataset sizes reported as train/test QA pairs and a fixed test subset for evaluation.
- The two backbones share the same family, size, vision tower, LoRA targets, and data; vision is frozen for both.
- Table A2 reports dataset sizes as train/test QA pairs and evaluates a fixed random subset of n=350 test items per dataset.
C Infill Sampler
The infill sampler clamps user-fixed tokens at their positions throughout denoising. The wrapper applies this constraint to the incoming canvas and both outgoing canvases at each step.
- Fixed tokens are clamped at fixed positions on the incoming canvas and both outgoing canvases during every denoising step.
- The implementation patches the uncompiled outer denoising step rather than the compiled inner sampler.
- Figure A1 represents fixed_tokens and fixed_mask as [B, L] inputs to the any-order infill wrapper.
D.1 Medical VQA
The medical VQA examples show that finetuned diffusion answers can match or improve on autoregressive answers across varied radiology tasks, including lesion, abnormality, organ-count, and mass-identification questions.
- Ventricular assessment: For ventricular enlargement, finetuned diffusion answers no while AR and frontier models answer yes; the base diffusion response describes right lateral-ventricle enlargement.The diffusion answer also distinguishes generalized enlargement from asymmetric compression and effacement.
- Mass detection: Finetuned diffusion and AR both answer the chest-X-ray mass question affirmatively, unlike several frontier-model answers.The diffusion and AR outputs both say yes; GPT-4.1-mini and Sonnet-4.6 answer no.
- Mass identification: Finetuned diffusion correctly identifies a cecal mass origin, while finetuned AR attributes the mass to the ascending colon.The ground truth is the base of the cecum.
- Abnormality counting: Finetuned diffusion correctly reports three abnormality kinds in one brain MRI, matching AR and differing from the base diffusion answer of one.The corresponding frontier-model outputs are also shown for comparison.
- Organ counting: Finetuned diffusion answers the organ-count example with two, matching AR and Gemini-3.5-Flash, whereas base diffusion declines to count.A separate abdominal CT example shows finetuned diffusion and AR both answering six.
- Tumor identification: On the tumor-identification example, finetuned diffusion names papillary astrocytoma, matching the provided juvenile pilocytic astrocytoma ground truth more closely than several alternatives.The displayed diffusion output is truncated, but its visible completion contains “pillary astrocytoma.”
D.2 Any-Order Infill
The any-order infill evaluation compares diffusion and autoregressive fills under bidirectional versus left-only conditioning. Only bidirectional diffusion reconstructs the masked report sentence when fixed text appears on both sides of the gap.
- Any-Order Infill: Only bidirectional diffusion reconstructs the masked sentence across the held-out MIMIC-CXR infill examples.The evaluation forms a four-way grid over diffusion or AR generation and bidirectional or left-only conditioning.
- Any-Order Infill: Bidirectional conditioning supplies fixed report text on both sides of the gap, whereas left-only conditioning supplies only the preceding text.For autoregression, both-sided context is included in the prompt, but it still cannot condition on the right-side text.
- Any-Order Infill: The diffusion fill reconstructs report content between surrounding fragments, while the other three grid conditions fail to reconstruct the masked sentence.The comparison includes autoregressive generation with both-sided context and both models under left-only conditioning.