PaperlayerView exact revision on arXiv

Source-linked AI summary

LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR

Said Taghadouini, Adrien Cavaillès, Baptiste Aubertin

arXiv:2601.14251v2cs.CV

TL;DR

Document OCR remains difficult because real pages combine ambiguous layout, tables, scientific notation, figures, and noisy scans, while multi-stage pipelines are brittle and complex. LightOnOCR-2-1B addresses this with a compact end-to-end multilingual VLM trained on expanded, curated supervision and refined with RLVR, while adding image localization through normalized bounding boxes. It achieves the highest reported OlmOCR-Bench overall score, 83.2 ± 0.9, using 1B parameters, and weight-space methods expose an OCR–bbox trade-off.

  • Problem

    Real-world document OCR still struggles with ambiguous reading order, tables, scientific notation, figures, and noisy scans, while multi-stage pipelines couple multiple components.

  • Method

    LightOnOCR-2-1B is a compact end-to-end multilingual VLM trained with expanded, cleaner supervision, higher-resolution inputs, RLVR, and bounding-box coordinate training.

  • Results

    83.2 ± 0.9 overall score on OlmOCR-Bench, the highest among evaluated systems, using 1B parameters; bbox variants enable localization with an OCR–bbox trade-off.

  • Takeaways & Limitations

    The released model provides compact end-to-end document parsing, while checkpoint averaging and task-arithmetic merging offer practical control over OCR and localization behavior.

  • Takeaways & Limitations

    Performance outside European or Latin-script languages is not fully supported, and handwritten text transcription remains inconsistent because supervision primarily uses printed or typeset sources.

Abstract

from arXiv · show

We present LightOnOCR-2-1B, a 1B-parameter end-to-end multilingual vision--language model that converts document images (e.g., PDFs) into clean, naturally ordered text without brittle OCR pipelines. Trained on a large-scale, high-quality distillation mix with strong coverage of scans, French documents, and scientific PDFs, LightOnOCR-2 achieves state-of-the-art results on OlmOCR-Bench while being 9$\times$ smaller and substantially faster than prior best-performing models. We further extend the output format to predict normalized bounding boxes for embedded images, introducing localization during pretraining via a resume strategy and refining it with RLVR using IoU-based rewards. Finally, we improve robustness with checkpoint averaging and task-arithmetic merging. We release model checkpoints under Apache 2.0, and publicly release the dataset and LightOnOCR-bbox-bench evaluation under their respective licenses.

1 Introduction

LightOnOCR-2 is introduced as a compact end-to-end multilingual VLM designed to replace brittle OCR pipelines while handling difficult document layouts. Its training expands data coverage and uses higher resolution, augmentation, explicit empty-page targets, RLVR, and localization extensions.

  • Motivation: Real-world OCR remains difficult for multi-column reading order, tables, scientific notation, figures, and noisy scans.Existing production systems often split these tasks across coupled stages such as layout analysis, recognition, table extraction, and reading-order reconstruction.
  • Approach: End-to-end VLMs learn extraction directly from pixels to structured text, reducing pipeline engineering and simplifying specialization through fine-tuning.LightOnOCR-2 extends this approach with predicted bounding boxes for document images.
  • Training: LightOnOCR-2-1B uses a 2.5× larger, cleaner training mixture with stronger coverage of scans, French documents, and scientific content.The recipe also increases resolution to a 1540px maximum longest edge and adds augmentation plus explicit empty-page targets.
  • Training: RLVR targets repetition loops, mathematical formatting errors, and layout-sensitive consistency constraints through deterministic checks.A dedicated variant also predicts bounding boxes for embedded images.
  • Contributions: The contributions include a new OlmOCR-Bench state-of-the-art result, a localization benchmark, and weight-space methods for OCR–bounding-box trade-offs.The released model is a 1B-parameter end-to-end multilingual VLM, while the bbox benchmark and dataset are also released.

2 Overview

LightOnOCR is a compact vision-language architecture that converts document images into a single reading-order-preserving structured representation. LightOnOCR-2 retains the architecture while substantially upgrading data scale, supervision, and document coverage.

  • Architecture: LightOnOCR combines a vision encoder, multimodal projector, and language-model decoder in a compact 1B-parameter VLM.OCR behavior is embedded in the weights, so inference does not require task prompts.
  • Architecture: The native-resolution Vision Transformer preserves spatial structure across variable image sizes and fine document typography.This supports documents with diverse aspect ratios.
  • Architecture: A two-layer projector maps visual features into the language embedding space after 2 × 2 spatial merging, reducing visual tokens 4×.The reduction keeps high-resolution inputs computationally tractable while retaining spatial granularity.
  • Architecture: The decoder emits one linearized page representation that preserves reading order and includes structured tokens for non-text elements.Visual tokens are provided as one contiguous block before text tokens.
  • LightOnOCR-2 changes: LightOnOCR-2 keeps the architecture but scales training from 17M to 43M pages and strengthens scanned, scientific, and European-language coverage.French receives particular emphasis, while supervision uses a stronger teacher and improved conversion and normalization.

3 Datasets and Preprocessing

The dataset combines heterogeneous OCR sources, region crops, blank pages, and TEX-derived scientific supervision, then applies normalization and validation to improve target consistency. The work also introduces a two-subset benchmark for document-image localization.

  • Data sources: LightOnOCR-2 improves distilled supervision by upgrading the teacher model and targeting more faithful mathematical notation and formatting.The corpus includes full pages and varied document-region crops annotated by multiple VLM teachers.
  • Data sources: Explicit blank-page examples establish consistent empty-input targets and mitigate looping or hallucination behaviors.Region crops expose the model to paragraphs, headers, abstracts, and snippets.
  • Supervision: Teacher-emitted figure coordinates were removed from main OCR targets but retained as separate supervision for the bounding-box procedure.This keeps the base OCR objective focused on transcription.
  • Normalization: Heterogeneous sources and teachers create superficial transcription inconsistencies, including watermarks, variable placeholders, and format drift.These artifacts motivate deterministic preprocessing before hashing and filtering.
  • Normalization: Normalization sanitizes text, removes recurring watermarks, homogenizes special cases, and validates LATEX, Markdown, tables, and KaTeX compatibility.The conversion pass records structured metadata such as partial success, unresolved references, and compatibility status.
  • Scientific supervision: The nvpdftex pipeline produces rendered pages, structured targets, pixel-accurate semantic boxes, and page metadata from TEX sources.These outputs support both scientific OCR supervision and arXiv localization experiments.
  • Localization benchmark: LightOnOCR-bbox-bench contains 290 manually reviewed OlmOCR-derived samples and 565 automatically annotated arXiv samples.It reports F1 at IoU 0.5, mean IoU, and exact count accuracy separately for the two subsets.

4 Training

Training combines filtered, augmented document images with deterministic RLVR objectives that target OCR and localization failures. Checkpoint averaging, resumed coordinate training, and bbox-focused refinement produce OCR and localization variants with controllable trade-offs.

  • Supervised training: Training renders pages at 200 DPI, resizes them to a 1540px maximum longest edge, filters problematic samples, and applies document augmentations.Long completions are capped at 3100 tokens.
  • RLVR: RLVR uses deterministic automatic checks to optimize specific OCR failure modes without additional annotation.Rewards can include binary unit tests on synthetic documents.
  • RLVR: The training setup uses GRPO for one epoch with AdamW, learning rate 4 × 10^-5, KL strength β = 0.01, and multiple rollouts per prompt.The rollout counts are 28 for OCR and 14 for bounding-box training.
  • RLVR: OCR-focused and bbox-focused RLVR recipes separately optimize scientific OCR checks and IoU-based localization rewards.The OCR recipe addresses repetition, mathematical rendering, and formatting constraints.
  • Localization: Bounding-box training extends image outputs with normalized coordinates in [0, 1000] and resumes pretraining to learn localization.The model already learns whether an image exists; coordinate training primarily adds where it is.
  • Model variants: Checkpoint averaging over the last five checkpoints consistently outperforms any single checkpoint for the supervised OCR baseline.RLVR is then applied to this averaged base to obtain the best OCR model.

5 Results

LightOnOCR-2-1B achieves the strongest reported OlmOCR-Bench performance with a compact end-to-end model, while also supporting embedded-image localization and high-throughput inference.

  • OlmOCR-Bench: 83.2 ± 0.9 overall score is the highest among evaluated OlmOCR-Bench systems, despite using only 1B parameters.Headers/footers are excluded from this benchmark definition, and evaluations use no test-time heuristics.
  • OlmOCR-Bench: Improvements are broad across categories, especially for ArXiv, old scans math, and table-heavy documents.The reported gains are associated with higher-quality data, stronger scientific coverage, and higher-resolution training.
  • Ablations: RLVR improves overall performance and reduces generation failures such as repetition loops relative to the base checkpoint.The comparison isolates the effect of RLVR between LightOnOCR-2-1B-base and LightOnOCR-2-1B.
  • Ablations: Bounding-box prediction causes a small OCR-quality drop but enables localization, while task-arithmetic merging partially recovers OCR performance.The merged checkpoint exposes an explicit OCR–bbox trade-off.
  • Image Bounding Box Detection: LightOnOCR-2-1B-bbox improves F1@0.5 and count accuracy over the 9B baseline on both bbox-bench subsets, with comparable mean IoU.The evaluation is restricted to visual elements such as figures and images.
  • Efficiency: LightOnOCR-2 provides substantially higher throughput than larger end-to-end baselines, supporting high-volume document processing.Throughput is measured as pages/sec over the full 1,403-page OlmOCR-Bench evaluation.

6 Scope and Limitations

LightOnOCR-2-1B is aimed at printed documents, especially scientific PDFs, scans, European languages, and layout-heavy pages. Its main scope boundaries are non-Latin multilingual coverage and handwritten text.

  • Scope: The model is designed for printed document understanding, with particular strengths on scientific PDFs, typed scans, European languages, and layout-heavy pages.Targeted examples include dense typography, LATEX math, moderately degraded scans, multi-column documents, and long-form tables.
  • Limitations: Non-Latin multilingual performance is not fully supported and can show degraded fidelity or inefficient tokenization.The training mix and normalization pipeline prioritize Latin-script documents, with CJK and Arabic identified as examples of affected scripts.
  • Limitations: Handwritten text transcription remains inconsistent because supervision is primarily derived from printed or typeset sources.Cursive or unconstrained handwriting is not a target use-case for the released checkpoints.

7 Conclusion

LightOnOCR-2-1B combines a compact end-to-end OCR architecture with improved data, normalization, resolution, localization training, and weight-space techniques. It sets a new OlmOCR-Bench state of the art and releases resources for reproducible document extraction research.

  • Conclusion: LightOnOCR-2-1B is a 1B-parameter end-to-end OCR VLM that sets a new state-of-the-art result on OlmOCR-Bench.The paper attributes improved performance to changes in data, normalization and conversion, and higher-resolution training with targeted augmentations.
  • Conclusion: An image-localization variant predicts bounding boxes using coordinates during pretraining and RLVR refinement without degrading OCR quality.The conclusion describes this as the localization-training strategy used for the bbox variant.
  • Conclusion: Checkpoint averaging and task-arithmetic merging provide practical gains and explicit control of the OCR–bbox trade-off.The techniques operate in weight space to combine or adjust specialized model behavior.
  • Conclusion: The authors release model weights, datasets, and the LightOnOCR-bbox-bench benchmark for reproducible research.The released resources target high-fidelity document extraction and localization.

A Vocabulary Pruning

Vocabulary pruning reduces model size and can improve speed for Latin-script OCR, but removes script-specific tokens and harms tokenization for non-Latin languages.

  • Method: 51k, 32k, and 16k vocabulary variants are evaluated using frequency-based pruning with recursive sub-token frequency propagation.The experiments target English/French documents while preserving tokenizer integrity.
  • Results: 13.8% parameter reduction occurs at 16k tokens with minimal English OCR degradation, from 76.1% to 75.4% on OlmOCR-Bench.These experiments were conducted on LightOnOCR-1.
  • Results: 11.6% faster inference at 32k tokens retains 96% of base performance, giving the best reported speed–accuracy balance.The result is reported for the 32k variant in the vocabulary-pruning trade-off study.
  • Limitations: Arabic and Chinese experience approximately 3× token-count inflation when script-specific tokens are removed.The table caption summarizes the broader trade-off as improved Latin-script speed but degraded Chinese and other non-Latin tokenization.

C Additional Results

The original headers_footers metric rewards omitting visible header and footer content, unlike the models’ full-page transcription objective.

  • C.1 OlmOCR Headers/Footers: The benchmark rewards absence of header and footer text, while LightOnOCR models retain all visible page content.This creates a metric–objective mismatch for full-page transcription.
  • C.1 OlmOCR Headers/Footers: Under this scoring definition, full-page extraction can receive lower headers_footers scores despite preserving headers, titles, page numbers, and footers.

C.2 OmniDocBench Results

OmniDocBench provides a complementary evaluation of reading order and structured layout fidelity, but its formatting-sensitive, English-and-Chinese focus makes it secondary here.

  • C.2 OmniDocBench Results: The benchmark offers a complementary view of reading order and structured layout fidelity.
  • C.2 OmniDocBench Results: Its edit-distance metrics are sensitive to formatting conventions, and its documents primarily target English and Chinese, so results are treated as secondary.
  • C.2 OmniDocBench Results: OmniDocBench reports EN/ZH results and shows LightOnOCR-2 improving over LightOnOCR-1 while ranking among the strongest models in its size class.
  • C.2 OmniDocBench Results: Throughput comparisons use full inference runs and pages/sec rather than tokens/sec because tokenization and output lengths differ across models.
  • C.2 OmniDocBench Results: RLVR reduces the fraction of OlmOCR-Bench generations flagged as low-entropy repetition loops under a ZLIB compression-ratio detector.

D.1 Task-Arithmetic Model Merging

Task-arithmetic merging combines OCR- and bounding-box-specialized checkpoints to expose a controllable trade-off between transcription quality and localization. The paper illustrates this model family on complex tables, bounding boxes, scientific documents, and out-of-distribution examples.

  • D.1 Task-Arithmetic Model Merging: Task-arithmetic merging interpolates between OCR- and bbox-specialized checkpoints to control the OCR–localization trade-off.The interpolation is θ = θ_bbox + α(θ_ocr − θ_bbox), with α ∈ [0, 1].
  • D.1 Task-Arithmetic Model Merging: The merged models are evaluated on both OCR and bounding-box localization metrics.
  • D.1 Task-Arithmetic Model Merging: Examples demonstrate transcription of complex tables, including rendered outputs alongside the original document images.
  • D.1 Task-Arithmetic Model Merging: Bounding-box examples pair original pages with rendered transcriptions and crops corresponding to generated image boxes.
  • D.1 Task-Arithmetic Model Merging: Additional examples cover scientific documents and transcription outside the training distribution.
  • D.1 Task-Arithmetic Model Merging: A scanned historical document example highlights the model’s coverage of scanned files.
Loading 2601.14251v2…