PaperlayerView exact revision on arXiv

Source-linked AI summary

GutenOCR: A Grounded Vision-Language Front-End for Documents

Hunter Heidenreich, Ben Elliott, Olivia Dinica, Yosheb Getachew

arXiv:2601.14490v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Document OCR needs explicit grounding and controllable reading for downstream systems, but existing approaches often provide limited token–box alignment and rigid interfaces. GutenOCR fine-tunes Qwen2.5-VL into a unified grounded OCR front-end with prompt-controlled reading, detection, and localization. It improves structured reading and detection across benchmarks while exposing trade-offs in formula-heavy and visually diverse settings.

  • Problem

    Existing OCR-specialized VLMs often provide limited token–box alignment and coarse control over where and how documents are read.

  • Method

    GutenOCR fine-tunes Qwen2.5-VL into single-checkpoint models exposing full-page reading, detection, localized reading, and conditional detection through prompts.

  • Results

    Across in-domain business and scientific documents, Fox, and OmniDocBench v1.5, GutenOCR substantially improves structured reading and detection over Qwen2.5-VL backbones and OCR-specialized baselines.

  • Takeaways & Limitations

    Explicit grounding and controllable OCR interfaces provide stable primitives that downstream retrieval, extraction, and document QA systems can compose.

  • Takeaways & Limitations

    The recipe can reduce robustness on visually diverse and color-guided layouts and degrade formula recognition on formula-heavy pages.

Abstract

from arXiv · show

GutenOCR is a family of grounded OCR front-ends obtained by fine-tuning Qwen2.5-VL-3B and Qwen2.5-VL-7B. The resulting single-checkpoint vision-language models expose reading, detection, and grounding through a unified, prompt-based interface. Trained on business documents, scientific articles, and synthetic grounding data, the models support full-page and localized reading with line- and paragraph-level bounding boxes and conditional ``where is x?'' queries. We introduce a grounded OCR evaluation protocol and show that GutenOCR-7B more than doubles the composite grounded OCR score of its Qwen2.5-VL-7B backbone on 10.5K held-out business and scientific pages (0.40 to 0.82). On Fox and OmniDocBench v1.5, our approach substantially improves region- and line-level OCR as well as text-detection recall, but reveals trade-offs in page-level linearization, color-guided OCR, and formula-heavy layouts.

1 Introduction

GutenOCR addresses the gap between flexible but weakly grounded vision–language OCR and modular classical pipelines by providing a unified grounded front-end. It combines prompt-controlled OCR interfaces, grounded evaluation, empirical gains, and an open release.

  • Motivation: Grounded OCR links page transcripts and spans to 2D boxes while supporting prompted reading of arbitrary regions.This targets the limited grounding and indirect reading control of OCR-free vision–language models.
  • Motivation: Production systems need precise local reading, token–pixel links, and layout-aware representations for extraction, filtering, RAG, and human oversight.Non-grounded page-to-Markdown outputs make end-to-end review slow and error-prone.
  • Approach: GutenOCR uses one VLM checkpoint for full-page reading and detection, localized reading, and conditional detection with plain or structured grounded outputs.Structured outputs include line- and paragraph-level boxes.
  • Approach: The training recipe specializes Qwen2.5-VL-3B/7B without tokenizer changes, adapters, or frozen modules while supporting 8k–16k-token page transcripts.
  • Evaluation: The grounded OCR protocol jointly measures recognition, localization, and page fidelity using CER, WER, F1@0.5, mCER@0.5, and CERe2e.The metrics separate recognition errors from localization and reading-order failures.
  • Release: The project releases training and evaluation code, data provenance, and open model weights.

2 GutenOCR Overview

GutenOCR exposes traditional OCR primitives through a single prompt-driven vision–language checkpoint. Its interfaces accept page images with optional queries or boxes and return plain or structured text-and-box outputs, including layout-sensitive text2d.

  • 2.1 Model Overview: GutenOCR is a single-checkpoint VLM family exposing reading, detection, and grounding as stable, composable OCR primitives.It is designed as an API-like layer between page images and downstream systems.
  • 2.1 Model Overview: A page image may be augmented with a text query or bounding box, producing plain transcripts or line- and paragraph-level outputs with boxes.
  • 2.2 Task Families and Interfaces: The unified interface covers full-page reading, full-page detection, conditional detection, and localized reading.
  • 2.2 Task Families and Interfaces: Grounded outputs use JSON objects containing text and pixel-coordinate bbox values [x1, y1, x2, y2], while detection-only outputs contain bbox arrays.Rotated or skewed text is represented by an axis-aligned enclosing rectangle.
  • 2.2 Task Families and Interfaces: The text2d format preserves 2D layout in a plain string by converting horizontal gaps to spaces and vertical gaps to blank lines.
  • 2.2 Task Families and Interfaces: Conditional detection returns boxes for lines matching a normalized query, whereas localized reading transcribes a user-specified region.

3 Data & Training

GutenOCR is trained on real business and scientific documents plus synthetic grounding data, using staged length and mixture changes to specialize general-purpose VLMs for grounded OCR.

  • 3.1 Data curation: The training corpus combines real document collections with synthetic layouts emphasizing line geometry, paragraph geometry, math, and noisy business forms.
  • 3.1 Data curation: Real sources cover noisy business documents, varied layouts, and long multi-column scholarly articles containing equations, symbols, captions, and references.
  • 3.1 Data curation: Synthetic sources explicitly target line and math grounding through Grounded LATEX and SynthDoG grounding.Grounded LATEX associates rendered equations with tight bounding boxes; SynthDoG supplies line-level boxes.
  • 3.1 Data curation: 10.5K pages are reserved for held-out evaluation, with selection stratified by source and sequence length.
  • 3.2 Training recipe: Training fine-tunes public Qwen2.5-VL-3B and Qwen2.5-VL-7B checkpoints on single rasterized page images and prompts.Optimization uses AdamW on 8×H100 GPUs.
  • 3.2 Training recipe: Prompt templates and image-reference variation expose every task to diverse phrasings while preserving task semantics.The authors observed qualitatively more robust behavior under minor inference-time phrasing changes.
  • 3.2 Training recipe: A length-based curriculum progresses from short sequences to long-context page reading while shifting the dataset mixture.

4 Evaluation Setup

The evaluation separates text accuracy, box localization, and end-to-end page fidelity across grounded OCR tasks. It covers held-out in-domain pages and public Fox and OmniDocBench v1.5 benchmarks.

  • 4.1 Metrics: Grounded OCR evaluation measures text errors, detection quality, and end-to-end box–text and page-level fidelity.
  • 4.1 Metrics: CER and WER quantify character- and word-level errors for plain-text outputs.
  • 4.1 Metrics: F1@0.5 and Recall@0.5 evaluate detection by one-to-one box matching at IoU ≥0.5.
  • 4.1 Metrics: mCER@0.5 measures recognition among successfully localized boxes, while CERe2e measures page-level fidelity after deterministic linearization.CERe2e is sensitive to missing lines, spurious lines, reading-order errors, and layout whitespace.
  • 4.1 Metrics: The metric suite jointly tests region finding, region reading, and preservation of reading order and layout-sensitive whitespace.
  • 4.1 Metrics: Composite grounded OCR scores average converted reading scores with detection and conditional-detection F1 values in [0, 1].
  • 4.2 Benchmarks: The experiments cover held-out business and scientific pages plus Fox and OmniDocBench v1.5.In-domain evaluation includes full-page, localized, detection, and conditional-detection tasks.

5 Results

GutenOCR substantially improves grounded OCR over Qwen2.5-VL backbones across in-domain and external evaluations, especially for localized reading, detection, and region- or line-level OCR. These gains involve trade-offs in page linearization, color-guided OCR, formula recognition, and model-size or curriculum priorities.

  • In-Domain Results: 0.40→0.82 composite grounded OCR score more than doubles for GutenOCR-7B over Qwen2.5-VL-7B on 10.5K held-out pages.The in-domain composite gains are driven primarily by localized reading and detection.
  • Fox: GutenOCR substantially improves Fox region- and line-level OCR while sacrificing some page-level linearization and color-guided OCR.GutenOCR-3B reaches 0.053 region CER versus 0.260 for its backbone, while line CER falls from 0.817 to 0.240.
  • OmniDocBench v1.5: 0.620 and 0.549 text-span recall at R@0.5 for GutenOCR-3B and -7B, respectively, exceed the Qwen2.5-VL backbones’ ≈0.02 recall.This is a recall-only stress test, not a full detector ranking, because precision and unannotated predicted boxes are not measured.
  • OmniDocBench v1.5: Formula recognition degrades after fine-tuning, with the largest negative transfer for GutenOCR-3B.For 3B, CDM drops from 0.936 to 0.866 and CER rises from 0.189 to 0.294; the 7B degradation is milder but consistent.
  • Model Size: 0.819 versus 0.811 composite scores show only a modest advantage for GutenOCR-7B over GutenOCR-3B after fine-tuning.The 7B model favors global reading and conditional detection, whereas 3B favors localized reading and unconditional detection.
  • Training Stage Ablation: Stage 1 captures most full-page reading and detection gains, while Stage 3a mainly sharpens localized reading and conditional detection.Stage 3a yields the best composite scores; Stage 3b slightly degrades the composite, consistent with over-specialization on long contexts.

6 Discussion

GutenOCR is positioned as a grounded, controllable OCR front-end for downstream systems and human verification, rather than a page-only transcription tool. Its current limitations include reduced robustness on diverse, color-guided, formula-heavy, and structurally complex documents.

  • Grounded front-ends versus page-only OCR: Grounded OCR prioritizes text-region recall, stable line and paragraph outputs, and predictable localized or conditional reading over page-level fidelity alone.Downstream systems can tolerate some transcription noise but are brittle to missing content, scrambled layout, and unreproducible pointing behavior.
  • Human-in-the-loop verification and control: Line- and paragraph-level boxes and multiple transcript formats let reviewers check coverage, reread regions, and attribute extracted values to evidence.The design supports review tools, exception workflows, and active-learning loops by making text quickly verifiable and provenance actionable.
  • Human-in-the-loop verification and control: False positives are typically easier to dismiss visually than false negatives, which force page rescanning and complicate exception triage.This asymmetric cost profile motivates the system’s bias toward recall and fine-grained grounding.
  • Interfaces, semantic surface, and comparison to prior work: GutenOCR supports full-page, localized, and conditional reading through stable grounded interfaces backed by a single checkpoint.Outputs include multiple transcript formats and structured text-plus-box representations that applications can transform into their own schemas.
  • Limitations and open challenges: Business and scientific specialization can reduce robustness on visually diverse layouts and color-guided cues while degrading formula recognition on formula-heavy pages.The models also remain text- and line-centric, without explicit table structure, math layout, cross-page links, or higher discourse units.
  • Toward document holograms: GutenOCR is a front-end module for document holograms, not a complete solution for structure, semantics, evidential QA, or fine-grained provenance.Higher hologram layers and evaluation protocols for evidential QA remain future work.

7 Related Work

Related work spans modular OCR, document conversion, OCR-based and OCR-free VLMs, text spotting, grounding, and direct grounding benchmarks. GutenOCR combines pipeline-like OCR primitives with a unified VLM interface rather than committing to one export format or task schema.

  • Classical OCR pipelines: Classical OCR decomposes processing into normalization, detection, recognition, and post-processing, providing modularity but struggling with complex layouts and reading order.Multi-column pages, nested tables, stamps, marginalia, and rotated inserts can scramble text streams and propagate errors.
  • GutenOCR’s pipeline formulation: GutenOCR replaces handcrafted pipeline components with one VLM interface supporting full-page reading, detection, conditional detection, and localized reading.It preserves explicit grounding and structured outputs while enabling controllable reading order and task-specific representations.
  • Document conversion: Document-conversion systems often produce Markdown, HTML, JSON, or other canonical representations that work well when downstream applications match the target schema.GutenOCR instead exposes composable OCR primitives as a stable contract.
  • Document VLMs: OCR-based document models consume OCR tokens and boxes, whereas OCR-free VLMs emit text or structured markup directly from pixels.General-purpose VLMs may perform OCR-like behaviors but typically lack a specified OCR API contract and use ad-hoc coordinate formats.
  • GutenOCR’s position: GutenOCR retrains a general VLM backbone to output structured text-and-box results through prompt-selectable schemas rather than QA answers or one canonical linearization.Its interface explicitly mirrors classical OCR operations.
  • Grounding and text spotting: Text spotting and phrase-grounding systems provide explicit region-text alignment, but often target scene text or language-region matching rather than document abstractions and language-driven OCR queries.GutenOCR generalizes these ideas to document reading, detection, localized rereading, and string-conditioned search.
  • Evaluation: Document benchmarks increasingly evaluate region- and line-level OCR, color-guided OCR, layout regions, formulas, and tables rather than only isolated recognition.Fox and OmniDocBench v1.5 exemplify this broader evaluation scope.

8 Conclusion

GutenOCR reframes OCR as a grounded front-end implemented by one prompt-driven vision-language checkpoint. Across document benchmarks it improves structured reading and detection while exposing limitations that motivate richer document hologram representations.

  • Conclusion: GutenOCR supports full-page reading, detection, localized reading, and conditional detection through a unified prompt interface.The same checkpoint provides the grounded OCR primitives used across these task families.
  • Conclusion: GutenOCR substantially improves structured reading and detection over Qwen2.5-VL backbones and several OCR-specialized baselines across in-domain, Fox, and OmniDocBench v1.5 evaluations.The results also reveal failure modes in formula-heavy and color-guided settings.
  • Conclusion: Explicit grounding and controllable reading are presented as important complements to page-level transcripts for downstream document systems.The conclusion points toward richer document hologram representations as future work.

A.1 Task families & I/O schemas

GutenOCR defines prompt-selectable OCR tasks and I/O schemas that produce text, structured text, or grounded boxes. Its coordinate and text2d procedures convert line predictions into normalized, layout-sensitive outputs for downstream use and evaluation.

  • Task families & I/O schemas: The interface is parameterized by task type, input type, and output type across reading, detection, conditional detection, and localized reading.Outputs include text, text2d, lines, paragraphs, and BOX schemas.
  • Task families & I/O schemas: Prompt templates are sampled across task–input–output combinations, with image nouns and determiners randomized to improve robustness to phrasing.The task semantics remain fixed while inference prompts vary slightly.
  • Task families & I/O schemas: Detection prompts return bounding-box arrays without text, while conditional detection searches for a query string and localized reading transcribes content inside a specified region.Full-page reading prompts can request plain or structured formats.
  • Coordinate system: Grounded outputs use axis-aligned boxes in image coordinates, with integer pixels, top-left origin, rightward x, and downward y.Rotated or skewed text is represented by its minimal enclosing axis-aligned rectangle.
  • Normalization and evaluation: Post-processing clips boxes to image bounds and discards non-positive-area boxes before evaluation.A tolerant JSON-repair step is applied first, with irreparable outputs treated as invalid.
  • text2d construction: The text2d representation places ordered line predictions on a character grid, preserving horizontal alignment, vertical gaps, and plain-text compatibility.Whitespace is retained during evaluation so layout-sensitive errors such as scrambled columns remain visible.

A.4 Task-Specific Semantics

GutenOCR exposes multiple OCR behaviors through distinct prompt-selected input–output schemas. These include full-page reading, detection, localized reading, and conditional detection, with either plain text or structured grounded outputs.

  • Conditional detection: Conditional detection takes a page image and query string, returning line boxes whose normalized transcripts match the query.Normalization includes Unicode NFKC conversion, whitespace trimming, and internal-whitespace collapse.
  • Localized reading: Localized reading takes a page image and pixel-coordinate box, then transcribes only text inside that region as a plain string.Line- and paragraph-level localized-reading variants use overlapping reference boxes to construct the target transcript.
  • Full-page reading and detection: Full-page reading accepts only a page image and returns linear, whitespace-preserving, or structured line-/paragraph-level outputs.Structured outputs are JSON arrays containing text and bounding-box fields.
  • Output handling: Structured predictions pass through JSON repair and normalization, while irreparable outputs are treated as maximal-error failures.The pipeline standardizes structurally similar outputs before evaluation.
  • Full-page reading and detection: Full-page detection accepts only a page image and returns bounding boxes for lines or paragraphs without transcripts.Detection uses IoU-based matching at threshold 0.5.

B Data

The data pipeline standardizes document images, text annotations, and box geometry across OCR-IDL, TabMe++, PubMed-OCR, and synthetic grounding corpora. Approximately 10,500 pages are reserved for held-out evaluation.

  • Document preprocessing: OCR-IDL and PubMed-OCR begin from document PDFs and source OCR outputs, which are converted into page images and normalized annotations.TabMe++ was rasterized during the original TabMe construction and is reused as provided.
  • Annotation normalization: Text normalization applies Unicode NFKC, canonical punctuation conversion, whitespace trimming, and internal-whitespace collapse.The same operator is used for supervision and evaluation.
  • Geometry unification: All dataset boxes are mapped to clipped axis-aligned [x1, y1, x2, y2] pixel coordinates, with degenerate boxes discarded.This unifies differing interval and indexing conventions.
  • Synthetic grounding data: Grounded LATEX supplies localized mathematical-expression supervision by mining fragments and rendering them with randomized rotation and sub-pixel position.Each equation is rendered on a blank page.
  • Synthetic grounding data: SynthDoG grounding uses narrative and transactional snippets with word-aware line breaking and no hyphenation for line-level supervision.The layout modifications avoid mid-word splits.
  • Splits and validation: ∼10,500 pages from IDL, TabMe++, and PubMed-OCR are held out for evaluation, while validation samples are randomly carved out and fixed within each training stage.Early stopping uses validation loss, checkpoint rollback, and fixed logged random seeds.

C Training

Training fine-tunes Qwen2.5-VL models on single-page document images and routes all task families through prompts. The system standardizes outputs, normalization, and metric inputs across reading and grounding modes.

  • Training setup: GutenOCR feeds each single-page PDF rasterization directly to the Qwen2.5-VL vision encoder at 72 DPI without cropping, tiling, or high-resolution modes.The original aspect ratio is preserved.
  • Training setup: All modules of Qwen2.5-VL-3B-Instruct and 7B-Instruct are fine-tuned without tokenizer changes, frozen components, adapters, or LoRA layers.The public instruction-tuned checkpoints serve as the starting models.
  • Context handling: Sequence lengths range roughly from 2k to 16k tokens during curriculum training, with one page per example and truncation when targets exceed stage limits.Inference caps generated tokens at 4,096 externally and 16,384 internally.
  • Task routing: Prompt-based routing selects reading, detection, localized_reading, or conditional_detection tasks and their output schemas using one checkpoint.Reported results use deterministic greedy decoding with no sampling.
  • Output schemas: Grounded outputs use JSON arrays of text–bounding-box objects, while detection outputs contain boxes only and localized reading returns recognized text within a specified box.The interface also specifies coordinate conventions, clipping, and natural reading order.
  • Output processing: A tolerant repair-and-normalization pipeline converts imperfect structured outputs into canonical text and bbox fields before evaluation.Irreparable outputs receive maximal error.
  • Metric inputs: Text metrics operate on normalized strings, while text2d preserves line breaks and intra-line spacing so layout changes affect scores.Normalization uses Unicode NFKC, trimming, and internal-whitespace collapse.

D.2.2 Detection metrics

Detection metrics match predicted and ground-truth boxes one-to-one using IoU thresholds, then quantify precision, recall, and F1. End-to-end metrics separate recognition quality after localization from page-level fidelity.

  • Detection matching: IoU measures the area of box intersection divided by the area of box union for predicted and ground-truth boxes.Boxes use pixel coordinates in [x1, y1, x2, y2] form.
  • Detection matching: At τ = 0.5, Hungarian one-to-one matching counts true positives, false positives, and false negatives from pairs meeting the IoU threshold.The main text reports F1@0.5 and Recall@0.5.
  • End-to-end metrics: mCER@0.5 averages per-span CER only over boxes matched at IoU ≥0.5, isolating recognition quality conditional on successful localization.Detection recall accounts for coverage separately.
  • End-to-end metrics: CERe2e computes page-level CER after sorting predicted and ground-truth boxes in deterministic reading order and concatenating their texts.It jointly reflects localization, reading order, insertions, and deletions.
  • Edge cases: Invalid structured predictions receive maximal errors, including mCER@0.5 = 1, CERe2e = 1, F1@0.5 = 0, and Recall@0.5 = 0.The convention applies when JSON cannot be repaired or no usable boxes/text remain.
  • Composite score: The composite grounded OCR score averages reading scores and detection or conditional-detection F1@0.5 values equally across task families.The resulting score lies in [0, 1].

E.1 Fox Benchmark Details

The Fox evaluation tests page, region, line, and color-guided OCR under unified prompting and deterministic decoding. GutenOCR excels at grounded region and line reading but trades off canonical page ordering and color-guided selection.

  • Benchmark setup: The evaluation covers English page, region, line, and color-guided OCR subtasks using official Fox splits and scripts.Pages are filtered to those whose language metadata is purely English.
  • Benchmark setup: All models use official benchmark prompts, with only pointer-box coordinate conversion adapted for region- and line-level tasks.GutenOCR retains its training system prompt, while baselines receive a minimal OCR-focused prompt.
  • Benchmark setup: Deterministic greedy decoding uses temperature = 0, no sampling, and a fixed maximum new-token budget.
  • Page metrics: Page F1 is order-agnostic token overlap, whereas Page CER compares predictions with Fox’s canonical linearization as one string.Consequently, block reordering, extra whitespace, or missing spans affect Page CER more strictly than Page F1.
  • Results: GutenOCR achieves strong region- and line-level OCR, while its layout-sensitive outputs produce high Page F1 but substantially worse Page CER than page-to-Markdown baselines.The discrepancy reflects different target behaviors: grounded, layout-preserving outputs versus adherence to Fox’s canonical markup order.
  • Results: GutenOCR performs poorly on color-guided OCR because its training mixture lacks color-guided supervision and fine-tuning appears to overwrite color-selection heuristics.Observed failures include reading the words “red box” literally or returning box coordinates instead of requested text.
  • Evaluation caveat: OmniDocBench detection is evaluated with recall only because unmatched predictions may be valid under GutenOCR’s broader text-line ontology or hallucinations.The benchmark cannot distinguish these cases, making precision and F1 ill-defined for this setting.

G.1.1 Reading

GutenOCR fine-tuning converts Qwen2.5-VL backbones into structured readers and strong line detectors across business and scientific datasets. Gains extend to plain, layout-preserving, localized, and conditional reading, while later specialization can trade generality for PubMed-OCR performance.

  • Structured, line-based reading: GutenOCR converts both Qwen2.5-VL backbones into stable structured readers, reducing page and line recognition errors relative to the raw models.On 7B, fine-tuning eliminates the gap between strong per-line accuracy and poor page assembly.
  • Plain-text reading: All GutenOCR variants reduce CER and WER versus their backbones for plain-text OCR, with 7B Stage 3a the strongest linear-text reader.Stage 1 has the lowest average CER, while Stage 3a improves WER, especially on PubMed-OCR.
  • Layout-sensitive reading: Both GutenOCR backbones roughly halve CER and WER relative to Qwen baselines on layout-preserving text2d outputs.Stage 3a improves WER by better handling spaces and line breaks, especially on PubMed-OCR.
  • Localized reading: Localized reading lowers average CER from ≈0.7/0.53 to 0.11–0.20 and WER from ≈0.8/0.67 to 0.20–0.32 across the backbones.GutenOCR-3B Stage 3a achieves the best overall localized-reading CER and WER.
  • Cross-domain trade-offs: Stage 3b improves PubMed-OCR while modestly degrading business-dataset performance, indicating a domain-specialization trade-off.
  • Full-page line detection: GutenOCR raises full-page line-detection F1 and recall from approximately 0.1 to 0.75–0.82 across datasets.Stage 1 is the best general-purpose detector, while Stage 3a and Stage 3b favor TabMe++ and PubMed-OCR respectively.
  • Phrase-conditioned detection: Conditional detection reaches F1 and recall of 0.85–0.90 across datasets, with Stage 3a best on average and 7B retaining a small edge.Stage 3b again specializes toward PubMed-OCR.
Loading 2601.14490v2…