PaperlayerView exact revision on arXiv

Source-linked AI summary

TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX Reconstruction

Chengye Wang, Lin Fu, Zexi Kuang, Yilun Zhao

arXiv:2604.22880v1cs.CL

TL;DR

Reliable page-level reconstruction of scientific PDFs into compilable, structure-faithful LaTeX remains under-served by OCR systems focused on plain text, Markdown, or localized regions. The paper introduces benchmarks, training data, and verifiable-reward optimization, finding consistent RLVR improvements over SFT and strong open-source performance while reducing functional failures.

  • Problem

    Existing OCR rarely addresses end-to-end page-level reconstruction of scientific PDFs into executable LaTeX while preserving global structure and reference integrity.

  • Method

    The paper introduces TEXOCR-Bench and TEXOCR-Train, then trains a page-level reconstruction model with SFT followed by reinforcement learning using verifiable LaTeX unit-test rewards.

  • Results

    RLVR yields consistent improvements over SFT across component- and structure-level metrics, while TEXOCR achieves the strongest performance among open-source baselines on TEXOCR-Bench.

  • Takeaways & Limitations

    The benchmark, training corpus, and results establish a stronger baseline for developing OCR systems that recover scientific documents into usable, compilable LaTeX.

  • Takeaways & Limitations

    The page-wise setting does not fully capture document-level cross-page structure, global consistency, and long-range dependencies.

Abstract

from arXiv · show

Existing document OCR largely targets plain text or Markdown, discarding the structural and executable properties that make LaTeX essential for scientific publishing. We study page-level reconstruction of scientific PDFs into compilable LaTeX and introduce TexOCR-Bench, a benchmark, and TexOCR-Train, a large-scale training corpus, for this task. TexOCR-Bench features a multi-dimensional evaluation suite that jointly assesses transcription fidelity, structural faithfulness, and end-to-end compilability. Leveraging TexOCR-Train, we train a 2B-parameter model, TexOCR, using supervised fine-tuning (SFT) and reinforcement learning (RL) with verifiable rewards derived from LaTeX unit tests that directly enforce compilability and referential integrity. Experiments across 21 frontier models on TexOCR-Bench show that existing systems frequently violate key document invariants, including consistent section structure, correct float placement, and valid label-reference links, which undermines compilation reliability and downstream usability. Our analysis further reveals that RL with verifiable rewards yields consistent improvements over SFT alone, particularly on structural and compilation metrics.

1 Introduction

The paper addresses the under-served problem of reconstructing scientific PDFs into executable, structure-preserving LaTeX by introducing a benchmark, training corpus, and combined SFT–RLVR approach. It evaluates frontier models and finds that compilability and structural consistency remain challenging, while verifiable-reward training improves performance beyond SFT.

  • Motivation: Scientific PDF reconstruction remains under-served because plain-text or Markdown linearization and localized LaTeX conversion fail to enforce global structural and reference constraints.These constraints include cross-page structure, float placement, numbering, and label-reference integrity; small syntax errors can make sources uncompilable.
  • Benchmark: 2,135 expert-annotated examples make up TEXOCR-Bench, which jointly evaluates transcription fidelity, structural faithfulness, and end-to-end compilability.The benchmark targets page-level reconstruction across diverse document types and technical domains and includes zero-touch compilation success.
  • Data and training: 57K papers and 404K page image–LaTeX pairs comprise TEXOCR-Train, supporting a recipe that combines SFT and RLVR to train the 2B-parameter TEXOCR model.The corpus is built from arXiv LaTeX source archives paired with PDFs and provides page-aligned supervision.
  • Evaluation: 78.5 accuracy is achieved by the best-performing evaluated model, GPT-5.3, while existing OCR systems degrade on TEXOCR-Bench despite strong performance on PDF-to-Markdown benchmarks.The results indicate that compilable, structure-consistent LaTeX reconstruction requires capabilities beyond high-quality transcription.
  • Results: 21 frontier models are evaluated, with TEXOCR achieving state-of-the-art performance among open-source models and RLVR consistently improving over SFT on component- and structure-level metrics.The improvements are driven by verifiable unit-test rewards that directly enforce compilability and referential integrity.

2 Related Works

Related work evaluates document OCR through end-to-end PDF conversion and structured LaTeX transcription, using similarity, consistency, and unit-test metrics. Methods have shifted from modular pipelines and supervised fine-tuning toward direct MLLM generation and reward-guided systems.

  • OCR Evaluation Benchmarks: OCR benchmarks cover end-to-end PDF-to-Markdown or plain-text conversion and structured LaTeX transcription for tables and formulas.READoc, OmniDocBench, and olmOCR-Bench address document extraction, while Table2LaTeX-RL and CMER-Bench target structured elements.
  • OCR Evaluation Benchmarks: Evaluation typically combines string or token similarity, consistency checks, and unit-test pass rates.olmOCR-Bench checks text presence, reading order, tables, formulas, and baseline functionality; Table2LaTeX-RL adds layout-consistent and render-based checks.
  • Document OCR Methods: Document OCR has shifted from modular pipelines toward MLLMs that transcribe directly from document images.Recent systems generate linearized text or markup end to end, while hybrid systems use MLLMs as recognition backbones within conventional document-processing frameworks.
  • Document OCR Methods: Earlier systems mainly used large-scale SFT on paired document images and text, whereas newer work increasingly adopts RL with task-specific or verifiable rewards.Nanonets-OCR2 and olmOCR exemplify SFT-based systems, while later approaches use reward signals to improve consistency on structured document elements.

3 TEXOCR-Bench Benchmark

TEXOCR-Bench evaluates page-to-LaTeX reconstruction with nine metrics spanning transcription fidelity, structural faithfulness, and end-to-end usability. It is constructed from heterogeneous PDF sources, 2,135 selected documents, and expert annotation followed by rule-based test-set assembly.

  • Evaluation Suite: Nine metrics assess OCR and LaTeX reconstruction across transcription fidelity, structural faithfulness, and end-to-end usability.The three dimensions organize complementary measures of content recovery, document structure and references, and output executability.
  • Evaluation Suite: Transcription Fidelity measures fine-grained content recovery using Complex Text Preservation and Formula Accuracy.Complex Text Preservation strictly matches extracted plain-text sentences for case, punctuation, and character accuracy; Formula Accuracy evaluates normalized mathematical expressions.
  • Evaluation Suite: Structural Faithfulness checks section-title recovery, citation coverage, and figure/table reference validity against document structure and cross-reference consistency.The metrics verify hierarchical section correctness, citation keys and positions, and whether references resolve to existing labels with valid LaTeX syntax and semantics.
  • Evaluation Suite: End-to-End Usability measures document similarity, severe generation failures, and compilation success without manual intervention.Document-Level Similarity uses normalized character-level edit distance, while Baseline detects truncation, token loss, or abnormal characters and Compilation Success Rate tests complete-project compilation.
  • Benchmark Construction: The benchmark selects 2,135 moderate-length documents with rich LaTeX structures, then consolidates annotated projects into single .tex test files for extracting ground-truth elements.The source pool includes contemporary scientific articles and scanned materials from heterogeneous sources; parsers extract figures, tables, equations, and section headings for metric computation.

4 TEXOCR-Train Construction

TexOCR-Train is constructed from arXiv LaTeX/PDF sources through canonical source recovery, page-image/text pairing, and structured alignment and labeling. The pipeline addresses page-body alignment, float placement, and bibliography supervision to support PDF-to-LaTeX conversion training.

  • Corpus construction: The corpus collects official arXiv LaTeX sources and PDFs from January 2022 to October 2025, merges reachable .tex files, and preserves document order and provenance.A deterministic parser then recovers document structure, including section hierarchy and figure and table information.
  • Data organization: Documents are segmented into single-page screenshots and organized as paired instances of page images and textual supervision.This creates the basic training examples for PDF-to-LaTeX conversion.
  • Page–Body Alignment: For main-body pages, GPT5mini detects page-content boundaries and aligns them to crawled LaTeX, while reference markers identify the reference section.Only the first reference page is retained as the reference section.
  • Float Placement: pdf2figure detects figures and tables across the PDF, enabling global layout information to assign each element to its most appropriate page despite float placement and pagination.The method resolves mismatches between rendered PDF layout and linear LaTeX source order.
  • Bibliography Supervision: Reference pages use GPT-based OCR labels, with non-reference regions supervised by LaTeX and reference regions converted into corresponding BibTeX on the first reference page.This hybrid strategy targets structured bibliography supervision rather than the rendered bibliography list.

5 Two-Stage Training Pipeline

The model is trained in two stages: supervised fine-tuning for faithful page-level transcription, followed by reinforcement learning with verifiable rewards targeting functional LaTeX correctness. Training uses page-aligned PDF–LaTeX pairs and deterministic unit tests covering transcription, structure, usability, and compilation.

  • Reinforcement Learning with Verifiable Rewards: RLVR addresses functional failures that token-level likelihood misses, including formula rendering, table structure, numeric preservation, and cross-reference resolution.The method samples multiple completions per page and scores them with automatically constructed binary unit tests.
  • Supervised Fine-Tuning: SFT fine-tunes an instruction-tuned multimodal backbone on page-aligned pairs of single-page PDF screenshots and corresponding LaTeX or BibTeX outputs.The prompt specifies desired output format and conventions, and training uses standard next-token prediction.
  • Supervised Fine-Tuning: For the first reference page, mixed supervision assigns LaTeX to body regions and BibTeX entries to detected reference regions.This encourages a clean transition from body generation to bibliography generation near the paper’s end.
  • Reinforcement Learning with Verifiable Rewards: Group-based RL updates use within-group advantage normalization and a KL penalty that keeps the learned policy close to the SFT reference policy.The page-level reward is the fraction of unit tests passed.
  • Unit-Test Reward Design: The reward adapts all nine TEXOCR-Bench metrics into deterministic pass/fail tests spanning transcription fidelity, structural faithfulness, and end-to-end usability.Tests include complex-text anchors, numerical table overlap, formula accuracy, section and reference checks, similarity, sanity checks, and compilation.

6 Experiments

Experiments benchmark frontier multimodal models on TEXOCR-Bench under a unified page-level protocol, revealing weaknesses in syntax-sensitive reconstruction and validating design choices in inference and RL training. Single-image inference and targeted verifiable unit-test rewards improve structural reliability, while larger RL group sizes stabilize optimization.

  • Experimental Setup: The study benchmarks diverse frontier models on TEXOCR-Bench using independently rendered PDF pages whose LaTeX outputs are concatenated in document order.The evaluation reports a nine-metric suite across transcription, structure, and compilation dimensions.
  • Model Comparison: Proprietary MLLMs are most consistent, whereas many open-source models degrade on malformed equations, broken tables, and inconsistent reference formatting.These failures expose insufficient syntactic robustness and project-level consistency for reliable downstream use.
  • Inference Strategy: Single-image inference consistently outperforms multi-image and merged multi-page strategies because alternatives introduce cross-page interference, resolution loss, and visual clutter.Preserving page-level locality and visual clarity yields more accurate and stable LaTeX generation.
  • Ablation Studies: Removing any verifiable unit-test category consistently degrades its corresponding evaluation metric, with little compensation from other metrics.The results demonstrate that each unit-test reward provides targeted supervision for a specific functional capability.
  • Ablation Studies: Larger RL group sizes K ∈ {4, 8, 12, 16, 20, 24} produce more stable performance, whereas small K increases variance and makes gains less reliable.Increasing K reduces optimization noise and improves relative advantage estimates and variance reduction.

7 Conclusion

The work addresses PDF-to-LaTeX reconstruction by targeting outputs that compile while preserving document structure, introducing benchmarks and training resources for executable, structure-faithful reconstruction. Verifiable unit-test rewards reduce failures such as broken environments, malformed tables, and invalid references.

  • The work targets PDF-to-LaTeX output that both looks correct and compiles while reliably preserving document structure.
  • TEXOCR-Bench evaluates executable, structure-faithful reconstruction at scale, while TEXOCR-Train provides page-aligned supervision for training.
  • Unit-test-style verifiable rewards reduce failures including broken environments, malformed tables, and invalid references.The rewards optimize directly for functional correctness.

Limitation

The work is an initial step toward reliable page-level reconstruction of scientific documents into compilable LaTeX, but the task inherently requires document-level modeling.

  • Limitation: The page-wise setting does not fully capture cross-page structure, global consistency, and long-range dependencies inherent to document-centric reconstruction.Future work should develop more effective document-level approaches beyond isolated page reconstruction.

A Implementation Details of Evaluation Metrics … A.8 Figure and Table Reference Validity

The appendix defines deterministic, document-level metrics by comparing generated LaTeX with reference source, covering textual, tabular, formulaic, structural, citation, and cross-reference fidelity. These metrics use specialized matching and validity procedures to assess reconstruction quality and usability.

  • A Implementation Details of Evaluation Metrics: All metrics are computed deterministically from generated and ground-truth LaTeX, generally aggregating page outputs in their original document order.
  • A.1 Complex Text Preservation: Context existence recovers representative plain-text sentences from each reference section through exact substring matching against concatenated generated LaTeX.Sentences containing LaTeX commands or environments are excluded.
  • A.2 Document-Level Similarity: Document-level similarity uses normalized Levenshtein distance after removing BibTeX entries from generated output before comparison.
  • A.3 Table Accuracy: Table accuracy matches reference tables to same-page generated tables using normalized numerical entries and overlap or unique-anchor criteria.The metric is binary per table, with final accuracy equal to the proportion of successfully matched reference tables.
  • A.4 Formula Accuracy: Formula accuracy aligns normalized display-math environments to at most one reference formula and accepts matches only above a fixed similarity threshold.Normalization removes comments, layout commands, and whitespace.
  • A.5 Baseline Validity Check: The baseline validity check detects severe page-generation failures using non-empty, alphanumeric, language, emoji, and degenerate-repetition constraints.
  • A.6 Section Accuracy: Section accuracy normalizes recovered section titles and matches each predicted section to at most one reference section using bidirectional substring inclusion, reporting precision.The extracted commands include \section, \subsection, and \subsubsection.
  • A.7 Citation Coverage; A.8 Figure and Table Reference Validity: Citation coverage validates generated citation keys against numeric citation indices or generated BibTeX content, while figure and table reference precision requires matching ground-truth label-reference counts.

B OCR Benchmark Construction … D Model Configuration

The benchmark construction specifies faithful, compilable page-to-LaTeX reconstruction, while inference prompts constrain content selection, structure, figures, tables, citations, and references. Annotator compensation and evaluated foundation-model configurations are also documented.

  • B.1 PDF2Latex Instruction: Annotations must compile cleanly while preserving visible text, equations, tables, captions, page-boundary fragments, and mathematical notation without inventing content.Light cleanup is permitted only when meaning remains unchanged.
  • B.1 PDF2Latex Instruction: Standard LaTeX environments, figure placeholders with captions and labels, exact table numbers, citation conventions, and consistent labels and references define the reconstruction format.The instructions require preserving table structure and citation attachment while standardizing figure and table identifiers.
  • B.1 PDF2Latex Instruction: Without LaTeX sources, annotators reconstruct wording, mathematics, and structure as closely as possible while maintaining compilability and exact numeric table entries.Figures use placeholders with captions and labels, while tables retain grids, separators, and row breaks.
  • B.2 Annotator Information: Dataset construction documents annotator information and reports an average payment rate of 13 US dollars.This information appears in Table 6.
  • C Model Inference Prompt: Inference converts only main-body content, excludes headers and decorative elements, and uses title, author, abstract, and section commands on a paper’s first page.Outputs contain only body LaTeX, without packages or a preamble, and preserve continuation text without omission.
  • C Model Inference Prompt: Figure and table handling assigns systematic filenames, labels, and references, including numbered subfigure files and corresponding cross-reference commands.Figure files use names such as figure_x.pdf, with subfigures named figure_x_1.pdf and labels such as figure_x.
  • D Model Configuration: The study documents the multimodal foundation models evaluated, organizing them by organization and aligning their configurations with main-text performance data.The model details are presented in Table 7.

E Error Analysis

The error analysis identifies recurring reconstruction failures in paragraph text, mathematical formulas, table structures, citations and references, and compilation. Examples illustrate each error category, including paragraph truncation and missing text, formula mismatches, table corruption, citation/reference errors, and compilation failures.

  • Paragraph reconstruction: Paragraph reconstruction errors include truncation and missing text.These failures are illustrated in Figures 3 and 4.
  • Mathematical formulas: Mathematical reconstruction errors include mismatched formulas.Figures 5 and 6 provide examples of these mismatches.
  • Tables: Table reconstruction errors include corrupted table structures.Figures 7 and 8 illustrate table structure corruption.
  • Citations and references: Citation and reference reconstruction errors produce citation and reference errors.Figures 9 and 10 show examples of these failures.
  • Compilation: Some reconstruction outputs fail to compile.Figures 11 and 12 provide examples of compilation failures.
Loading 2604.22880v1…