Source-linked AI summary
olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models
Jake Poznanski, Aman Rangapur, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Aman Rangapur, Christopher Wilhelm, Kyle Lo, Luca Soldaini
TL;DR
PDFs are valuable but difficult to convert into faithful, logically ordered text, while proprietary VLM extraction can be costly. The paper presents olmOCR, combining document anchoring with a fine-tuned 7B VLM and a dedicated benchmark. olmOCR achieves competitive results at substantially lower reported cost, while benchmark construction still involves source-validation limitations.
Problem
PDF formats encode content for fixed-page rendering rather than logical text structure, making faithful extraction difficult, while high-performing proprietary VLM extraction can be costly.
Method
olmOCR combines document anchoring with a fine-tuned 7B vision language model and evaluates extraction using binary unit tests in olmOCR-Bench.
Results
olmOCR achieves state-of-the-art benchmark performance, outperforming GPT-4o, Mistral, and Qwen 2.5 VL while costing less than $176 per million PDF pages.
Takeaways & Limitations
The released toolkit, model, training data, code, inference pipeline, and benchmark support open development and large-scale PDF linearization.
Takeaways & Limitations
Benchmark coverage depends on finding and validating enough cases from each source, and some pages with legitimate unusual charsets are excluded from relevant tests.
Abstract
from arXiv · showhide
PDF documents have the potential to provide trillions of novel, high-quality tokens for training language models. However, these documents come in a diversity of types with differing formats and visual layouts that pose a challenge when attempting to extract and faithfully represent the underlying content for language model use. Traditional open source tools often produce lower quality extractions compared to vision language models (VLMs), but reliance on the best VLMs can be prohibitively costly (e.g., over 6,240 USD per million PDF pages for GPT-4o) or infeasible if the PDFs cannot be sent to proprietary APIs. We present olmOCR, an open-source toolkit for processing PDFs into clean, linearized plain text in natural reading order while preserving structured content like sections, tables, lists, equations, and more. Our toolkit runs a fine-tuned 7B vision language model (VLM) trained on olmOCR-mix-0225, a sample of 260,000 pages from over 100,000 crawled PDFs with diverse properties, including graphics, handwritten text and poor quality scans. olmOCR is optimized for large-scale batch processing, able to scale flexibly to different hardware setups and can convert a million PDF pages for only 176 USD. To aid comparison with existing systems, we also introduce olmOCR-Bench, a curated set of 1,400 PDFs capturing many content types that remain challenging even for the best tools and VLMs, including formulas, tables, tiny fonts, old scans, and more. We find olmOCR outperforms even top VLMs including GPT-4o, Gemini Flash 2 and Qwen-2.5-VL. We openly release all components of olmOCR: our fine-tuned VLM model, training code and data, an efficient inference pipeline that supports vLLM and SGLang backends, and benchmark olmOCR-Bench.
1 Introduction
PDFs encode content for fixed-page rendering rather than logical text structure, making faithful extraction difficult and costly at scale. olmOCR addresses this with an open-source toolkit, benchmark, and trained model for high-quality plain-text conversion.
- The extraction problem: PDFs store rendered-page information instead of logical units such as headings, making reading-order and layout-rich extraction challenging.The difficulty is especially pronounced for documents with multiple columns, floating diagrams, headers, and footnotes.
- The extraction problem: $6,200 per million pages makes GPT-4o-based PDF conversion costly despite recent improvements in end-to-end extraction.The passage reports this cost for converting a million pages.
- olmOCR: olmOCR introduces an open-source toolkit for converting PDFs or document images into clean plain text suitable for language-model development.The toolkit targets general-purpose context extraction and linearization.
- olmOCR: 260,000 crawled PDF pages paired with GPT-4o OCR outputs form olmOCR-mix-0225, covering diverse publicly available document types.The collection includes academic papers, public-domain books, legal documents, brochures, and more.
- Evaluation and impact: 1,400 PDF pages and over 7,000 unit-test cases comprise olmOCR-Bench, enabling direct binary comparisons across extraction systems.The benchmark covers diverse document types and avoids fuzzy reference matching and LLM-as-judge evaluation.
- Evaluation and impact: $176 per million PDF pages is the reported olmOCR processing cost, while the toolkit is packaged for scaling from one to hundreds of GPUs.The system uses SGLang and vLLM inference engines and is reported to outperform several alternatives.
2 Creating and Training on olmOCR mix
The authors build training data from diverse crawled and scanned PDFs, use GPT-4o to generate linearized supervision with document anchoring, and fine-tune a specialized 7B VLM. They also retain structured outputs and scalable training and inference procedures.
- Data acquisition: Two data-acquisition challenges are obtaining a large, diverse PDF collection and obtaining linearized plain-text supervision targets.These challenges motivate the crawling, filtering, and data-generation pipeline.
- Data acquisition: Up to three pages per PDF are uniformly sampled after filtering non-English, unparsable, spam, fillable-form, and overly short documents.The source pool includes web-crawled PDFs and public-domain book scans from the Internet Archive.
- Supervision generation: GPT-4o generates linearized plain-text targets because human annotation is expensive and existing PDF-internal extraction tools fail on images and provide unreliable ground truth.The authors selected gpt-4o-2024-08-06 for its reported balance of accuracy, reliability, and batch cost-efficiency.
- Supervision generation: Document anchoring augments page images with extracted text blocks and position information to guide GPT-4o on dense or complex layouts.The authors report that GPT-4o otherwise omits content, rewrites it unfaithfully, or captions images inappropriately.
- Supervision generation: Structured JSON output first extracts metadata such as language, orientation, and table presence before generating page text in natural reading order.The format also supports more efficient output processing and reduces inappropriate image captions on text-free pages.
- Fine-tuning: A Qwen2-VL-7B-Instruct checkpoint is fine-tuned on olmOCR-mix-0225 to produce olmOCR-7B-0225-preview.Training uses 8 NVIDIA H100 GPUs, 10,000 steps, and approximately 1.2 epochs.
- Fine-tuning: The fine-tuned model retains the structured JSON output used in the training data, while document anchoring is simplified and capped for training inputs.Training prompts use rendered pages up to 1,024 pixels on the longest edge and are truncated to 8,192 tokens when necessary.
3 Building olmOCR-Bench
olmOCR-Bench evaluates PDF linearization with deterministic unit tests spanning general extraction properties and challenging document types. Its scoring averages pass rates across test sources and document types.
- Evaluation design: Each test asks whether plain-text output satisfies a specific property or contains a specific element using simple, deterministic pass-or-fail rules.This design avoids dependence on LLM judges and fuzzy reference-text metrics that can miss fine-grained errors such as incorrect formulas.
- Benchmark construction: 1,402 documents and 7,010 unit-test cases form olmOCR-Bench, covering diverse PDF sources and challenging extraction targets.The benchmark combines manually designed and GPT-4o-assisted tests with manual review, while excluding certain private or duplicate documents.
- Unit-test categories: The benchmark tests text presence, text absence, natural reading order, table accuracy, and equation-related structure.Examples include preserving table-cell relationships, excluding recurring headers and footers, and checking relative symbol positions in equations.
- Test conditions: Text comparisons use normalization, while test cases may enforce conditions such as document position, case sensitivity, neighboring table cells, or preserved rowspan and colspan structure.Normalization includes whitespace, quote, hyphen, Markdown emphasis, and Unicode NFC handling.
- Sourcing documents and creating tests: Custom document collections target arXiv mathematics, old mathematical scans, tables, historical scans, headers and footers, and long tiny-text pages.These sources were selected because olmOCR or earlier iterations often struggled with them, and each received tailored acquisition and test-generation procedures.
- Scoring: The final tool score is the average percentage of passed tests across each test source after evaluating every PDF page with each system.Scores are macro-averaged by document type, with a default baseline test checking that usable alphanumeric output was produced.
4 Evaluating olmOCR
The evaluation measures olmOCR on its benchmark, downstream language-model tasks, and inference cost. Across these settings, the toolkit outperforms strong alternatives while reducing processing expense.
- 4.1 olmOCR-Bench Results: olmOCR significantly outperforms Mistral, GPT-4o, and Qwen 2.5 VL on olmOCR-Bench.The benchmark comparison was conducted after olmOCR-Bench development to avoid iterating on the benchmark before evaluation.
- 4.1 olmOCR-Bench Results: olmOCR produces significantly cleaner plain text than specialized open-source tools.
- 4.2 Downstream Evaluation: +1.3 percentage point average improvement on widely reported language-model benchmark tasks follows replacing Grobid-plus-rules peS2o tokens with olmOCR-processed tokens.The comparison continued pretraining the same 7B checkpoint for 50B tokens on each version of the source data.
- 4.3 Cost Evaluation: Processing all peS2o PDFs at 1,000 tokens per page would cost $10.3M in H100 usage, while Mistral OCR is over five times more expensive than olmOCR.
5 Related Work
Related work includes pipeline-based PDF extraction systems, end-to-end models, and document-understanding benchmarks. The paper positions olmOCR against narrow-domain benchmarks and limited study of linearization effects on language-model training.
- Tools and Models for Linearizing PDFs to Plain Text: PDF linearization tools include parser-based and OCR-based systems, pipeline architectures, and end-to-end models such as Nougat and GOT Theory 2.0.
- Benchmarking VLMs on Linearization: Existing benchmarks such as FUNSD, SROIE, and RVL-CDIP primarily target narrow document categories and constrained formatting variations.
- Benchmarking VLMs on Linearization: olmOCR-Bench instead uses a diverse corpus spanning multiple domains and document types.
- Linearization for Language Modeling: Prior language-model data work emphasizes filtering and source mixing, while relatively little attention addresses linearization effects on downstream training.
6 Conclusion
The paper releases olmOCR as an open-source PDF-to-text toolkit combining document-anchoring with a fine-tuned 7B VLM. It also releases a broad benchmark and pipeline intended for large-scale document processing.
- olmOCR is an open-source toolkit for converting PDF documents into clean plain text.
- Document-anchoring combines PDF metadata with a fine-tuned 7B vision language model to approach closed commercial solutions at lower cost.
- olmOCR-Bench contains 7,010 test instances across 1,403 PDFs, covering text presence, reading order, tables, formulas, and baseline functionality.
- The released inference pipeline supports conversion from single documents to million-page PDF archives.The authors frame this scale as a way to process high-quality PDF sources that are underrepresented in web-heavy datasets.
A Methodology
olmOCR uses document-anchoring to provide a VLM with extracted PDF text, element coordinates, and a page raster. The method is applied during data generation, fine-tuning, and inference, reducing hallucinations while retaining image-only processing for PDFs without digital metadata.
- Document-anchoring extracts salient text-block and image coordinates and injects them with raw PDF text alongside a rasterized page image.This supplements the raster-only input used by many end-to-end OCR models.
- The anchoring prompt construction is used for GPT-4o training samples, olmOCR fine-tuning, and toolkit inference.
- The implementation extracts page text blocks and images with position information, samples relevant elements, and adds them to the VLM prompt up to a character limit.
- Document-anchored prompts result in significantly fewer hallucinations than page-image-only prompts.Image-only prompting was prone to completing unfinished sentences or inventing larger texts when image data was ambiguous.
- For PDFs without digital metadata, the pipeline relies on the rasterized page image while maintaining high performance.
B Cost Estimates of PDF Extraction Systems
The cost comparison estimates operational expenses for commercial APIs and local PDF extraction systems, with olmOCR tested on both L40S and H100 GPUs.
- $16.07 was the GPT-4o cost for 1,288 pages before batch pricing, which reduced the total to $8.03.The test used 3,093,315 input tokens and 833,599 output tokens.
- $0.767 was the MinerU cost for 1,288 pages, while Marker cost $14.84 for 10,000 pages.MinerU ran for 58 minutes 22 seconds on one L40S; Marker ran for 5 hours 31 minutes on one H100.
- $0.226 per 1,288 pages is the estimated olmOCR cost on an L40S GPU, compared with $0.229 on an H100.The L40S run took 17 minutes 10 seconds, while the H100 run took 5 minutes 7 seconds.
- $0.643 was the Gemini Flash 2.0 cost for the same 1,288-page evaluation set.The cited February 2025 rates were $0.10 per million input tokens and $0.40 per million output tokens.
C Evaluation of Trained Models
The trained-model evaluation examines validation loss, qualitative side-by-side comparisons, and alignment with GPT-4o outputs across web PDFs and Internet Archive books.
- Validation loss: Full fine-tuning produced lower validation loss than LoRA on both web PDFs and Internet Archive books, so it was used for the final model.The loss curves are shown in Figures 4 and 5.
- Development evaluation: Side-by-side evaluation compared outputs from two methods alongside the rendered page during development.The comparison software was released as open-source software as part of olmOCR.
- Alignment: 0.875 was olmOCR-7B-0225-preview’s average page-weighted alignment with its GPT-4o teacher model.GPT-4o self-alignment was 0.954, while the fine-tuned model more closely matched GPT-4o than GPT-4o mini.
- Alignment: Most documents parsed with olmOCR had medium-to-high alignment with GPT-4o, while temperature τ = 0.8 increased low-match cases.Low, medium, and high alignment correspond to below 70%, 70–95%, and above 95% alignment, respectively.
C.2 Intrinsic Human Evaluation
Intrinsic human evaluation compares PDF linearization outputs through pairwise judgments, ELO ratings, and win/loss statistics, while also describing the scalable processing pipeline and costs.
- Human evaluation: 452 valid preferences from 2,000 comparison pairs produced ELO ratings averaged over 100 simulations with bootstrapped 95% confidence intervals.The comparisons involved Marker, MinerU, GOT-OCR 2.0, and olmOCR, with approximately 75 judgments per tool pair.
- Human evaluation: Over 1800 is olmOCR’s ELO score, exceeding the other compared PDF extraction tools.Figure 7 visualizes the ELO rankings.
- Human evaluation: 71.4% is olmOCR’s highest reported win rate, achieved against MinerU in pairwise comparisons.The comparison criteria included reading order, content comprehensiveness, and structured-information representation.
- Operational scaling: 32 times cheaper than GPT-4o in batch mode is olmOCR’s reported cost advantage; it is also over 6 times cheaper than MinerU and one-third the cost of Marker.The pipeline batches work items of around 500 pages and can scale from single nodes to hundreds of nodes using shared cloud storage.
D.2 Increasing Robustness
The robustness design combines heuristics, prompt-length controls, retries, rotation handling, and structured processing to improve reliability while preserving throughput.
- Prompt control: Document-anchoring tokens are regenerated with exponentially lower character limits when the prompt would exceed 8,192 tokens.The procedure keeps the overall prompt within the model’s acceptable length during inference.
- Page handling: Rotation fields trigger page rotation and reprocessing when the model marks the page orientation as valid and supplies a correction angle.The structured schema also records language, table presence, diagram presence, and natural text.
- Decoding and retries: Repeated generations are detected through context-length or JSON-schema failures, and failed pages can be reprocessed up to N times before plain-text fallback.Higher temperature from τ = 0.1 to τ = 0.8 reduces the likelihood of repetitive outputs.
- Limitations: Frequent retries can significantly reduce total generation throughput and consume substantial SGLang memory when generations reach maximum sequence length.The planned mitigation is to detect repeated generations earlier and abort promptly.
- Training data and prompting: The training dataset uses structured GPT-4o outputs generated from page images plus layout-aware text produced by document-anchoring.The prompt requests natural reading order, restrained hallucination, and appropriate equation and table formatting.
- PII filtering: PII filtering marks identifiers directly, requires co-occurrence with identifiers for some categories, and always marks government IDs, financial information, and biometric data.Filled form fields are considered potential PII, while references to documents that typically contain PII are not marked by themselves.
F.3 Sample Test Classes
olmOCR-Bench includes targeted test classes for document structure, reading order, headers and footers, tables, and mathematical notation.
- The old_scans_math class evaluates recognition of equations in degraded historical scans.
- The arXiv_math class evaluates transcription of mathematical equations with variables, powers, and trigonometric expressions.
- The benchmark tests whether OCR omits document headers and page numbers when they should be excluded.
- The table_tests class evaluates horizontal cell ordering, while reading_order tests traversal from the left column to the right column.
G Example output
The examples compare outputs from olmOCR, MinerU, GOT-OCR 2.0, and Marker on challenging mathematical, historical, and document-layout content.
- The example comparison includes olmOCR, MinerU, GOT-OCR 2.0, and Marker under their default settings.
- The historical-document example contains OCR outputs for a passage about Christians, the viceroy, Dominican friars, and Solor.
- The exercise examples preserve prompts about velocity, acceleration, and determining when an object is slowing down or speeding up.
- The mathematical examples include derivative exercises, polynomial position functions, rocket motion, falling objects, car motion, and hummingbird motion.