Source-linked AI summary
olmOCR 2: Unit Test Rewards for Document OCR
Jake Poznanski, Luca Soldaini, Kyle Lo
TL;DR
OCR systems need evaluations that reflect document correctness across layouts, tables, formulas, and reading order more faithfully than edit distance alone. olmOCR 2 combines synthetic HTML-based unit-test generation with RLVR training, achieving state-of-the-art olmOCR-Bench performance and a 14.2-point gain over its initial release. The authors release the model, data, and code openly while identifying broader document types and reward comparisons as future work.
Problem
Edit-distance-based OCR evaluation can misalign with practical correctness, motivating a unified framework for diverse OCR errors.
Method
The paper generates synthetic HTML representations of real documents, extracts binary unit tests, and uses them as rewards for GRPO training of an OCR-specialized VLM.
Results
olmOCR 2 achieves state-of-the-art performance on olmOCR-Bench, with a +14.2 point overall improvement over the initial release.
Takeaways & Limitations
Binary unit-test rewards are effective for improving OCR extraction of equations, tables, and multi-column layouts.
Takeaways & Limitations
Future work must extend the synthetic pipeline to more complicated document types and unit tests and further compare binary tests with continuous scores and RL rewards.
Abstract
from arXiv · showhide
We present olmOCR 2, the latest in our family of powerful OCR systems for converting digitized print documents, like PDFs, into clean, naturally ordered plain text. olmOCR 2 is powered by olmOCR-2-7B-1025, a specialized, 7B vision language model (VLM) trained using reinforcement learning with verifiable rewards (RLVR), where our rewards are a diverse set of binary unit tests. To scale unit test creation, we develop a pipeline for generating synthetic documents with diverse and challenging layouts, known ground-truth HTML source code, and extracted test cases. We show that RL training on these test cases results in state-of-the-art performance on olmOCR-Bench, our English-language OCR benchmark, with the largest improvements in math formula conversion, table parsing, and multi-column layouts compared to previous versions. We release our model, data and code under permissive open licenses.
1 Introduction
olmOCR 2 is an OCR-specialized VLM trained with RLVR, using synthetic documents and binary unit tests to improve PDF content extraction. It achieves state-of-the-art olmOCR-Bench performance while releasing its model, data, and code under permissive licenses.
- olmOCR 2 uses an OCR-specialized VLM trained with reinforcement learning and verifiable rewards for extracting and linearizing digitized print documents.
- A synthetic document pipeline renders standard documents into clean HTML and generates verifiable unit tests for checking OCR output.
- +14.2 points overall on olmOCR-Bench compared with the initial release, with olmOCR 2 achieving state-of-the-art performance.
- The project releases its model, data, and code under permissive open-source licenses.
2 Why Unit Tests?
The paper motivates binary unit tests as an OCR evaluation framework because edit distance can misrepresent correctness for ties, reading order, and rendered math formulas. Unit tests instead target diverse document properties directly.
- olmOCR-Bench tests text presence, text absence, natural reading order, table accuracy, math formula rendering, and baseline robustness.
- Figure 1 shows that a unit test accepts equivalent caption placements while edit distance penalizes one placement and partially rewards severe ordering failures.
- Figure 2 shows that rendered equation-element positions can make model A pass and model B fail despite model A being more dissimilar to reference LaTeX.
- Binary unit tests treat multiple equivalent representations similarly when floating elements lack a definitive ground-truth order.
- Edit distance can reward or penalize OCR outputs inconsistently with practical correctness, including reading order and rendered formula fidelity.
- Binary unit tests provide one framework for evaluating diverse OCR errors, whereas calibrated continuous scores remain less developed beyond math formulas.
3 Scaling Unit Test Generation for RLVR
olmOCR 2 scales RLVR unit-test creation by converting real document pages into synthetic HTML with extracted test cases, then training on binary pass-rate rewards. The resulting data and training pipeline support large-scale OCR specialization.
- 3.1 Data: The pipeline converts real PDF pages into similar HTML renderings whose raw HTML enables programmatic unit-test generation.
- 3.1 Data: Layout analysis identifies columns, images, tables, headers, and footers to guide HTML generation and broaden unit-test coverage.
- 3.1 Data: HTML semantics support tests for headers and footers, while rendered math and sampled table cells provide additional test cases.
- 3.1 Data: The final synthetic mix contains 2,186 PDF pages and 30,381 test cases, with unit tests generated from HTML output alone despite OCR hallucinations.
- 3.2 Training: Additional rewards enforce EOS termination and document metadata output, while six models are weight-souped after training.
4 Results
olmOCR 2 combines supervised and reinforcement-learning updates with inference-system changes, achieving a significant improvement on olmOCR-Bench. The reported system integrates synthetic-data RL, checkpoint averaging, prompt and sampling changes, and a newer base VLM.
- Dynamic temperature scaling starts sampling at 0.1 and raises it toward 0.8 after failures to generate an EOS token.This approach targets better low-temperature performance while mitigating repetition loops.
- Matching text-first prompt order between training and inference substantially improved benchmark performance and enables prompt caching.The authors found no meaningful OCR difference for the reverse order, but fixed text first supports inference-engine caching.
- The benchmark comparison includes in-house reproductions alongside results reported by model authors for entries marked with an asterisk.
- Switching from Qwen 2 VL to Qwen 2.5 VL produced a slight benchmark-score improvement.
- olmOCR 2 achieves a significant improvement on olmOCR-Bench through combined training and inference-system updates.The reported best model incorporates SFT, RL on synthetic data, checkpoint averaging, and additional system changes.
- Six-seed checkpoint averaging combines token-level and sequence-level importance-sampling runs in the final RL-trained model.Three runs use token-level importance sampling and three use sequence-level importance sampling.
5 Related Work
Related OCR work has shifted from hand-written pipelines toward end-to-end vision-language models and reinforcement learning. olmOCR 2 is closest to Infinity Parser but differs in how it constructs synthetic HTML and defines verifiable rewards.
- Modern OCR increasingly uses vision-language models to generate plain text from document images end to end.Earlier systems relied largely on hand-written pipelines tied to PDF representations, while VLM-based OCR became more prominent recently.
- Recent OCR research applies reinforcement-learning rewards to improve OCR or visual document-answering systems.Examples include reasoning-focused OCR and visual document answering approaches.
- Infinity Parser is the closest related system, using synthetic HTML data and GRPO with verifiable rewards for an OCR-specialized VLM.olmOCR 2 samples real content to seed full HTML-page generation, whereas Infinity Parser injects sampled content into pre-made layouts; their reward definitions also differ.
6 Conclusion
The paper presents olmOCR 2 as an OCR-specialized VLM trained with binary unit-test rewards generated through a synthetic HTML pipeline. It releases the model, code, and training data under permissive licenses while identifying broader document coverage and reward comparisons as future work.
- olmOCR 2 uses binary unit tests as verifiable rewards and scales their generation through synthetic HTML renderings of sampled real documents.
- The authors release model checkpoints, training and inference code, and two training-data mixes under permissive open licenses.
- Future work will extend the synthetic pipeline to more complicated document types and unit tests and compare binary rewards with continuous scores.The planned comparisons include edit distance and other reinforcement-learning rewards.