Source-linked AI summary
MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding
Hejun Dong, Junbo Niu, Bin Wang, Weijun Zeng, Wentao Zhang, Conghui He
TL;DR
Long-document OCR is bottlenecked by sequential autoregressive decoding and its reliance on linguistic priors. MinerU-Diffusion reframes OCR as visually conditioned inverse rendering with block-wise diffusion and uncertainty-driven curriculum learning, achieving competitive performance and stronger robustness while improving decoding efficiency.
Problem
Autoregressive OCR remains sequential and can amplify reliability problems on long, structured documents while encouraging reliance on linguistic priors.
Method
MinerU-Diffusion uses visually conditioned block-level parallel diffusion decoding with confidence-guided scheduling and two-stage curriculum learning.
Results
MinerU-Diffusion achieves competitive performance across document, table, formula, and Semantic Shuffle benchmarks, with stronger resilience to disrupted semantics.
Takeaways & Limitations
The framework positions diffusion-based parallel decoding as a promising alternative for efficient and reliable document OCR.
Takeaways & Limitations
The training data primarily cover Chinese and English document parsing, with no dedicated evaluation of low-resource languages.
Abstract
from arXiv · showhide
Optical character recognition (OCR) has evolved from line-level transcription to structured document parsing, requiring models to recover long-form sequences containing layout, tables, and formulas. Despite recent advances in vision-language models, most existing systems rely on autoregressive decoding, which introduces sequential latency and amplifies error propagation in long documents. In this work, we revisit document OCR from an inverse rendering perspective, arguing that left-to-right causal generation is an artifact of serialization rather than an intrinsic property of the task. Motivated by this insight, we propose MinerU-Diffusion, a unified diffusion-based framework that replaces autoregressive sequential decoding with parallel diffusion denoising under visual conditioning. MinerU-Diffusion employs a block-wise diffusion decoder and an uncertainty-driven curriculum learning strategy to enable stable training and efficient long-sequence inference. Extensive experiments demonstrate that MinerU-Diffusion consistently improves robustness while achieving up to 3.2x faster decoding compared to autoregressive baselines. Evaluations on the proposed Semantic Shuffle benchmark further confirm its reduced dependence on linguistic priors and stronger visual OCR capability.
1 Introduction
Document OCR systems achieve strong performance with autoregressive decoding, but sequential generation creates efficiency and reliability bottlenecks for long, structured documents. MinerU-Diffusion reframes OCR as visually conditioned inverse rendering and uses diffusion decoding to improve parallelism and robustness.
- Autoregressive VLMs generate structured OCR text left to right, making decoding strictly sequential.
- Sequential decoding creates efficiency and reliability bottlenecks when parsing long documents, tables, and formulas.
- Autoregressive OCR can over-rely on linguistic priors, causing semantic hallucinations and cumulative errors when visual or semantic signals are disrupted.
- Masked diffusion is better aligned with OCR because visual conditioning and partial observations support approximately conditionally independent token prediction.
- MinerU-Diffusion combines block-wise diffusion decoding with uncertainty-driven curriculum learning for parallel reconstruction, stable training, and efficient long-sequence inference.
2 Related Works
Related work frames document OCR as structured sequence generation and introduces masked diffusion as a non-autoregressive alternative. Block diffusion further combines parallel sampling with structured attention and caching for scalable decoding.
- End-to-end document OCR has shifted from modular pipelines toward VLMs that generate structured text directly from pixels.
- Representative OCR systems formulate recognition as token-by-token autoregressive sequence generation, inheriting limitations from fixed decoding order.
- Masked diffusion progressively replaces clean tokens with [MASK] under a continuous corruption schedule, yielding a noised sequence x_t.
- Block diffusion uses block-wise attention to combine autoregressive training stability with parallel diffusion sampling and KV-cache reuse.
- Document OCR is suited to masked diffusion because image-to-text mappings are near-deterministic and have limited semantic ambiguity.
- Diffusion-based decoding is presented as a modeling paradigm that matches OCR structure while offering practical efficiency advantages.
3 Method
MinerU-Diffusion models OCR as inference over a spatially structured token sequence and replaces globally coupled or causal decoding with block-wise diffusion. Its structured attention and uncertainty-driven curriculum target scalability, stability, and boundary precision.
- 3.1 Problem Formulation: Inverse Rendering via Diffusion: Document OCR is represented as a unified structured token sequence whose vocabulary covers text, layout, tables, and mathematical operators.
- 3.1 Problem Formulation: Inverse Rendering via Diffusion: The serialized one-dimensional output reflects spatial and formatting dependencies rather than an intrinsic causal generation order.
- 3.2 MinerU-Diffusion: Unified Diffusion Architecture for OCR: Full-attention diffusion has O(L^2) complexity and can cause positional instability, long-range drift, and unnecessary coupling across locally structured document regions.
- 3.2 MinerU-Diffusion: Unified Diffusion Architecture for OCR: MinerU-Diffusion partitions outputs into blocks, using coarse autoregressive structure across blocks and parallel diffusion refinement within each block.
- 3.2 MinerU-Diffusion: Unified Diffusion Architecture for OCR: Its attention mask allows bidirectional attention within a block, causal attention to preceding blocks, and no attention to future blocks.
- 3.2 MinerU-Diffusion: Unified Diffusion Architecture for OCR: Block attention reduces complexity from O(L^2) to O(BL′^2), supports KV-caching across blocks, and preserves parallel decoding within blocks.
- 3.3 Training Strategy: The two-stage curriculum combines broad foundational learning with uncertainty-driven refinement of hard, boundary-focused samples.
4 Experiments
Experiments evaluate MinerU-Diffusion across document parsing, table and formula recognition, decoding efficiency, threshold trade-offs, and curriculum learning. The results show competitive recognition and parsing performance, substantial throughput gains, and sensitivity to layout quality, decoding thresholds, and staged training.
- Full-Document Parsing Task Results: MinerU-Diffusion achieves an Overall score of 88.94 without GT Layout and 93.37 with GT Layout on OmniDocBench.The gap between settings indicates that layout understanding remains a key bottleneck, while accurate layout signals are effectively leveraged.
- Table Recognition: 81.18/88.66 TEDS/TEDS-S on OCRBench v2 and 73.77/82.06 on CC-OCR show competitive table recognition.The strong TEDS-S scores indicate reliable preservation of table structure during diffusive decoding.
- Formula Recognition: 91.6/91.6/92.0/96.8 on CPE/HWE/SCE/SPE demonstrate strong formula recognition across complex, handwritten, and printed settings.The remaining gap to the best specialized pipeline is most visible on harder categories such as CPE.
- Confidence Threshold vs. Decoding Parallelism: 3.2× peak acceleration is achieved at 164.8 TPS while maintaining over 90% accuracy, compared with approximately 52 TPS for MinerU2.5 at matched accuracy.At thr=0.95, the method reaches 108.9 TPS at 93%+ accuracy, corresponding to a 2.1× speedup.
- Decoding Parallelism vs Accuracy: 93.37 Overall at thr=0.95 balances accuracy and efficiency, while increasing the threshold from 0.5 to 0.99 lowers TPF and TPS monotonically.Accuracy rises from 87.61 at thr=0.5 to 93.37 at thr=0.95, then slightly declines at 0.99; Table TEDS improves from 81.10 to 90.99.
- Decoding Strategy Comparisons: 93.34 Overall and 98.32 TPS for dynamic scheduling outperform static step=6 at 88.31 Overall and 91.56 TPS.The two-stage curriculum also stabilizes training: the full strategy exceeds 89% accuracy and reaches 88.937 versus 35.712 for Stage 2 alone without GT Layout.
5 Conclusion
MinerU-Diffusion is a 2.5B-parameter diffusion framework that replaces autoregressive OCR with block-level parallel decoding and confidence-guided scheduling. Its experiments show competitive performance against autoregressive baselines and stronger resilience to disrupted semantics.
- MinerU-Diffusion replaces autoregressive decoding with block-level parallel diffusion decoding and confidence-guided scheduling to improve efficiency and scalability.The framework uses a two-stage curriculum learning strategy to stabilize training and enhance boundary precision and robustness.
A Training Details
Training proceeds through three stages that initialize multimodal alignment, adapt the model to large-scale OCR, and fine-tune it on harder document cases. Table 6 summarizes the setup and hyperparameters for these stages.
- MinerU-Diffusion is trained in three phases: generic multimodal alignment, large-scale OCR adaptation on Dbase, and fine-tuning on the harder subset Dhard.Stage-0 uses generic image-text data, Stage-1 performs OCR adaptation, and Stage-2 targets difficult samples.
- Table 6 summarizes the training setup and hyperparameters across the three training stages.
- Stage-0: Stage-0a trains only the MLP adaptor, while Stage-0b unfreezes all parameters to improve multimodal instruction following and longer visual-context adaptation.The two sub-stages provide initialization before document-specific supervision.
- Stage-0: Stage-0 uses shorter sequences and no document augmentation, prioritizing stable initialization for later large-scale OCR optimization.
A.2 Stage-1: Large-Scale OCR Adaptation
Stage-1 is the main OCR adaptation phase, using the Layout&OCR mixture to establish reliable performance across page layout and element recognition. It combines task diversity, scale, and augmentation to improve stability under masked-diffusion training.
- Stage-1 trains on Layout&OCR data covering page-level layout detection and recognition of text, formulas, and tables.Layout samples use full-page images with relative coordinates, while recognition samples use cropped regions to preserve local detail.
- Masked diffusion trains from partially observed targets, making optimization more sensitive to label noise and heterogeneous data quality than fixed left-to-right-prefix training.
- Stage-1 uses scale, task diversity, and document augmentation to improve training stability and establish reliable OCR on standard samples.
- Stage-1 uses sequence length 12288, batch size 256, learning rate 4 × 10−5, and 9 training epochs.
A.3 Stage-2: Hard-Case Specialization
Stage-2 specializes the model on difficult document examples while replaying broader Stage-1 data to preserve baseline OCR coverage. It increases sequence length and lowers the learning rate during four-epoch fine-tuning.
- Stage-2 shifts sampling toward unstable cases, including crowded layouts, ambiguous crop boundaries, complex tables, and noisy supervision.A smaller replay portion from the broader Stage-1 distribution preserves baseline OCR coverage while targeting residual failures.
- Stage-2 increases sequence length to 16384 and reduces the learning rate to 2 × 10−5 while retaining batch size 256.The phase uses a 0.1 warmup ratio, keeps all parameters trainable, and lasts 4 epochs.
A.4 Training Dynamics
The four-stage training recipe maintains stable optimization throughout, including during multimodal initialization, broad OCR training, and hard-case refinement.
- Loss and gradient norm remain stable across Stage 0a, Stage 0b, Stage 1, and Stage 2.Stage 0a and Stage 0b converge rapidly, Stage 1 supports smooth long-horizon optimization, and Stage 2 remains stable during hard-case refinement.
A.5 Data Augmentation
Training uses document-specific perturbations and task-aware processing to improve robustness while preserving valid OCR targets and separating global layout learning from precise recognition.
- Four augmentation types perturb OCR inputs: geometric changes, background disturbances, color shifts, and image degradation.Examples include scaling, grid distortion, rotation, texture overlays, watermarks, scanlines, shadows, brightness changes, and contrast adjustments.
- Augmentations simulate uneven lighting, compression artifacts, scan traces, and cluttered backgrounds without altering layout-detection geometry.Strength and sampling probability are tuned by task type to limit drift from valid OCR targets.
- Page-level layout detection and crop-level recognition are trained separately within one unified model.The visual token budget is limited to 2048 tokens per image, with resizing only when exceeded.
C.1 Why Full-Attn Is Inferior
Full-Attn is inferior because global attention is slower and fixed decoding lengths mismatch documents, wasting computation and harming generation quality.
- Full-Attn processes roughly L^2/B token interactions per step versus about 2L for Block-Attn, making Block-Attn faster to train.Full-Attn applies global pairwise attention, while Block-Attn reduces the effective cost to a near-linear form in practice.
- Fixed-length Full-Attn decoding wastes computation when the preset length is larger or smaller than the document target.Over-provisioning allocates unused positions, whereas under-provisioning exhausts the decoding budget before completion.
- Generation quality is best when preset length L is close to target length N; mismatch can cause truncation, instability, or repeated rows and empty cells.When L < N, generation may stop early; when L ≫ N, it often fails to stop cleanly.
C.2 Generation Quality under Length Mismatch
Qualitative comparisons show that Full-Attn behavior changes sharply with table length and preset decoding length, while the examples also illustrate complete parsing and diffusion refinement across OCR tasks.
- Generation Quality under Length Mismatch: Full-Attn repeats empty rows on short tables, generates normally only at matched lengths on medium tables, and truncates or repeats on long tables.The comparison tests Full-Attn at L = 256, 512, and 1024 against Block-Attn at L = 1024.
- Qualitative Examples: Figures 13 and 14 show complete recognition from original page images through predicted layouts to rendered parsed outputs.Examples span academic literature, slides, reports, newspapers, exam solutions, and instructional pages.
- Qualitative Examples: Figures 15–18 visualize progressive diffusion refinement for layout, text, table, and formula generation.Each figure compares two samples from the same task category using cropped inputs and decoding visualizations.