PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pixels to Structure: Lightweight Vision-Language Models for Document OCR and Structured JSON Extraction

Uddipan Basu Bir, Vincent Christlein, Andreas Maier, Mathias Zinnen

arXiv:2610.11818v1cs.CVcs.AI

TL;DR

Institutions face open questions about lightweight VLM reliability for structured OCR on specialized archival records, alongside concerns about commercial API dependence. This paper benchmarks eight models across three collections under interface-aware evaluation and studies adaptation strategies; instruction-tuned general-purpose VLMs lead overall, while joint training’s effects vary by model.

  • Problem

    Evidence remains limited on lightweight VLM performance for structured archival OCR, JSON validity, interface-aware comparisons, and practical adaptation strategies.

    Source: §2 · p. 4

  • Method

    The paper benchmarks eight lightweight VLMs across three collections under a constraint-aware protocol and evaluates prompting, fine-tuning, and additional adaptation strategies.

    Source: §1 · p. 3

  • Results

    Instruction-tuned general-purpose VLMs Qwen2.5-VL, Phi-3.5-Vision, and Gemma-3 are strongest overall; joint training improves Gemma-3 across collections but shows negative transfer for Phi-3.5-Vision and especially Qwen2.5-VL-7B.

    Source: §6 · p. 15

  • Takeaways & Limitations

    Deployment choices should weigh one cross-collection checkpoint against dataset-specific checkpoints for maximum accuracy, because joint training is not uniformly beneficial.

    Source: §6 · p. 15

    Hyperparameter optimization, preprocessing, and multi-stage training were evaluated independently, and findings apply to the evaluated model families rather than all emerging OCR/document models.

    Source: §6 · p. 15

Abstract

from arXiv · show

While massive, closed-source Vision-Language Models (VLMs) set strong benchmarks for document understanding, their dependence on commercial APIs limits adoption in institutional archives due to data autonomy concerns, recurring costs, and the environmental footprint of hyperscale computing. This is especially acute in heritage digitization, where documents include historical handwriting, domain-specific terminology (e.g., jewelry, prehistory, architecture), and non-standard layouts requiring high-dimensional structured extraction. We present a comparative study of eight open-source lightweight VLMs (up to 7B parameters) for Optical Character Recognition (OCR)-to-structure across three university heritage collections. Given a document image, models must extract text and generate schema-compliant JSON, enabling automatic validation and downstream use. We evaluate models under a constraint-aware protocol across zero-shot, few-shot, and fine-tuning settings, measuring extraction fidelity and structured-output quality using Character Error Rate (CER), Approximate Normalized Levenshtein Similarity (ANLS*), and mean Average Precision F1 (mAP-F1). Against a fine-tuning baseline, we further test the independent impact of (i) hyperparameter optimization, (ii) classical image preprocessing (illumination flattening, denoising, and CLAHE), and (iii) multi-stage training. Finally, we analyze the trade-off between dataset-specific fine-tuning and a single multi-dataset checkpoint, where joint training enables one model to operate across collections but can shift performance between datasets. Overall, we show that carefully adapted VLMs with up to 7B parameters can provide a sustainable, private, high-performing alternative to manual transcription or commercial black-box systems, and we offer actionable guidance for heritage institutions seeking institution-controlled OCR-to-JSON extraction.

1 Introduction

The paper addresses whether lightweight VLMs can reliably turn specialized archival documents into structured JSON, a task shaped by institutional data-autonomy and resource concerns. It benchmarks these models and studies practical adaptation strategies for institution-controlled OCR-to-structure.

1 Introduction

  • The task extends beyond plain-text OCR: models must return document content as structured JSON that can be searched, validated, or used downstream.The paper focuses on specialized historical records rather than standard invoice or ID-card benchmarks.Source: §1 · p. 2
  • Commercial API dependence raises data-autonomy and environmental concerns, while institutions may prefer locally deployable models under cost or GPU constraints.The reliability of lightweight models on specialized schemas remains unclear.Source: §1 · p. 2
  • The study evaluates zero-shot, few-shot, and fine-tuning settings while respecting whether each model accepts fixed or custom prompts.It selects the top three models for additional investigations and compares single-dataset with multi-dataset fine-tuning.Source: §1 · p. 3
  • The stated goal is to guide GLAM practitioners toward small VLMs and adaptation strategies that avoid commercial APIs and excessive resources.The paper frames its contributions as benchmarking models, identifying strong choices, and studying practical adaptation.Source: §1 · p. 3
  • Eight lightweight VLMs (≤7B) are benchmarked on three specialized archival collections for structured OCR-to-JSON as a private, sustainable alternative to commercial APIs.The study also proposes an evaluation protocol that accounts for models’ differing interface constraints.Source: §1 · p. 3
  • Further studies test hyperparameter optimization, classical image preprocessing, and multi-stage fine-tuning for their effects on extraction accuracy and schema adherence.These investigations address practical questions for institutional deployments.Source: §1 · p. 3

2 Related Work

Prior document-understanding work spans classical OCR pipelines, OCR-free extraction, and general-purpose VLMs, but the supplied review identifies gaps in structured-output evaluation and lightweight-model comparisons. This study addresses those gaps with a controlled benchmark and adaptation studies.

Traditional OCR and Document Understanding

  • Classical OCR pipelines can be brittle on complex layouts and degraded images, while downstream KIE approaches often depend on external OCR and task-specific labeled-box fine-tuning.The review names Tesseract and LayoutLM, LayoutLMv2, and LayoutLMv3.Source: §2 · p. 3

End-to-End Document Understanding without OCR

  • OCR-free systems map images to structured sequences or other formats, but are typically evaluated for accuracy rather than institutional schema compliance.Examples include Donut, Pix2Struct, Nougat, DAN, and GOT-OCR 2.0 [29].Source: §2 · pp. 3–4

Vision-Language Models for Document Tasks

  • VLM document evaluations largely emphasize question answering rather than JSON validity and schema compliance, despite advances on DocVQA and FUNSD.The review also notes open-deployment models including Pixtral [1] and DeepSeek OCR [30].Source: §2 · p. 4

Lightweight VLMs and Efficient Adaptation

  • Lightweight models and LoRA or QLoRA support adaptation on accessible hardware, but systematic comparisons on complex archival JSON extraction remain scarce.Some specialized extraction models have rigid interfaces that limit complex conversational prompting.Source: §2 · p. 4

Research Gap

  • Prior benchmarks leave uncertainty about lightweight VLMs (≤7B) on structured OCR, JSON validity, interface constraints, preprocessing, and practical adaptation strategies.The paper responds with a controlled benchmark of eight models and studies of preprocessing, hyperparameter optimization, and multi-stage training.Source: §2 · p. 4

3 Dataset, Task, and Evaluation

The evaluation uses three university collections spanning handwritten historical records, printed catalog cards, and complex architectural forms. Models produce dataset-specific JSON, assessed for both text fidelity and structured-output reliability.

3.1 Datasets and Splits

  • The three collections span archaeology, design history, and architecture, with challenges from historical handwriting to printed cards and nested forms.The datasets were developed with their institutions to support digitization of historical records.Source: §3.1 · p. 5
  • Erlangen-Prehistoric contains 308 pages of predominantly handwritten inventory-book records, with artifact fields such as inventory number, dimensions, find location, and description.Historical script variation and occasional degradation make it particularly challenging for OCR.Source: §3.1 · p. 5
  • Pforzheim Jewelry contains 590 inventory cards, and extraction requires a dense 17-field schema including material, weight, and insurance value.The cards describe jewelry from different periods and cultural areas.Source: §3.1 · pp. 5–6
  • Regensburg Scalalogy comprises 164 survey forms with printed labels and handwritten entries; its target structure has nested objects and boolean technical attributes.The dataset is presented as a high-complexity stress test for hierarchical schema compliance.Source: §3.1 · p. 6

3.2 Task Definition and Metrics

  • Each image must produce JSON matching a fixed dataset-specific schema; missing or illegible text fields are empty strings and boolean fields are false.Erlangen and Pforzheim use flat key-value dictionaries, whereas Regensburg uses nested objects and boolean fields.Source: §3.2 · p. 6
  • CER and ANLS∗ measure value-matching fidelity, while mAP-F1 and JSON validity assess whether outputs form usable JSON under strict parsing.For failed parsing, CER and ANLS∗ use raw-text comparisons, but mAP-F1 requires valid JSON and is set to zero otherwise.Source: §3.2 · pp. 7–8
  • CER compares canonical JSON strings, while structured ANLS∗ scores parsed objects at τ = 0.5 and falls back to raw-text similarity when parsing fails.Parsed predictions are serialized in the ground truth’s canonical format before string-based scoring.Source: §3.2 · pp. 7–8
  • mAP-F1 flattens valid JSON into dot-notated leaf keys and averages field-level F1 over overlap thresholds θ ∈ {0.3, 0.5, 0.7, 0.8}.Empty ground-truth fields with non-empty predictions count as false positives; invalid JSON yields mAP-F1 = 0.Source: §3.2 · p. 8

4 Experimental Setup and Methods

The experiments compare eight VLMs under their supported prompting interfaces, then evaluate few-shot and fine-tuning approaches alongside three independent fine-tuning studies. The protocol spans fixed and custom prompts, while additional studies test hyperparameter optimization, image preprocessing, and multi-stage training.

4.1 Models

  • Models span 200M–7B parameters, and four accept only fixed OCR-style prompts without few-shot demonstrations.The models are described as encoder-decoder and instruction-tuned architectures.Source: §4.1 · p. 8

4.2 Constraint-aware prompting

  • To compare models fairly, the study uses each model’s supported interface: fixed OCR prompts for restricted models and custom schema-guided prompts where available.Fixed-prompt outputs receive minimal cleanup before JSON parsing.Source: §4.2 · pp. 8–9, Table 2 · p. 9

4.3 Zero-shot inference

  • Zero-shot evaluation uses no demonstrations; custom-prompt models receive the target JSON skeleton, while fixed-prompt models receive their designated OCR prompt.All models use greedy decoding and their native image preprocessing.Source: §4.3 · p. 9

4.4 Few-shot inference

  • Few-shot evaluation uses one or two training-set image–JSON demonstrations, with k=1 for Pforzheim and k=2 for Erlangen and Regensburg.Only models supporting custom prompting are included.Source: §4.4 · p. 9

4.5 Standard fine-tuning

  • Models below 1B parameters receive full-parameter fine-tuning, while larger models use QLoRA with frozen base weights and trainable adapters.The best checkpoint is selected by validation CER and evaluated through autoregressive generation.Source: §4.5 · pp. 9–10

4.6 Additional Study Configurations

  • Three independent studies compare baseline fine-tuning with Optuna-selected hyperparameters, classical image preprocessing, and curriculum-based two-stage training.Optimization uses 20 five-epoch TPE trials; preprocessing combines illumination flattening, denoising, CLAHE, and letterbox resizing; multi-stage training uses a higher-learning-rate warm-up followed by refinement.Source: §4.6 · p. 10

5 Results

Across the evaluated settings, Phi-3.5-Vision, Qwen2.5-VL-7B, and Gemma-3-4B emerge as strong structured-extraction candidates, but adaptation effects vary by model and dataset. Joint training helps Gemma across collections, whereas Phi and Qwen show negative transfer.

5.1 Benchmark of eight VLMs Across Settings

  • Lower CER or higher ANLS∗ does not ensure structured accuracy: Donut can score well on those metrics yet receive 0 mAP-F1 when outputs are malformed or schema-incompatible.CER and ANLS∗ can fall back to raw-text comparison when JSON parsing fails.Source: §5.1 · p. 11
  • ≈95%–100% JSON validity is maintained by Phi-3.5-Vision, Qwen2.5-VL-7B, and Gemma-3-4B across settings, and these three are selected as top candidates for reliable structured extraction.Their selection reflects low CER, high ANLS∗, high mAP-F1, and stable JSON generation across datasets.Source: §5.1 · p. 11, Table 5 · p. 12

5.2 Secondary Studies on the Selected Top-3 Models

  • Hyperparameter optimization generally improves performance, with the largest gains on Regensburg, but its effects vary across models, datasets, and metrics.On Erlangen, Qwen benefits clearly, while Gemma and Phi show mixed effects, including some mAP-F1 declines.Source: §5.2 · p. 12, Table 6 · p. 12
  • Image preprocessing has mixed, model- and dataset-dependent effects: Phi’s Regensburg CER changes from 6.3% to 5.5%, while structured extraction does not always improve.Preprocessing also harms several settings, including Phi on Erlangen and Gemma and Qwen on Pforzheim and Regensburg.Source: §5.2 · pp. 12–13, Table 7 · p. 13
  • Multi-stage training consistently benefits Gemma across all three datasets, but its effects on Phi and Qwen range from gains to declines by dataset and metric.Qwen improves slightly on Erlangen but degrades on Pforzheim and Regensburg.Source: §5.2 · p. 13, Table 8 · p. 13

5.3 Single-Dataset vs. Multi-Dataset Fine-tuning

  • In a Gemma-3-4B Erlangen example, fine-tuning reduces sample CER from 26.8% to 13.9%, yet longer handwriting and Fundort/Fundzeit separation remain challenging.The example shows corrections to the inventory number and measurements.Source: Figure 4 · p. 14, §5.3 · p. 14
  • Multi-dataset fine-tuning improves Gemma across all three datasets, whereas Phi and Qwen degrade; Qwen’s Regensburg CER shifts from 7.6% to 24.90%.Qwen’s Regensburg ANLS∗ shifts from 87.0% to 65.2%; Phi’s decline is smaller, especially on Pforzheim.Source: §5.3 · p. 14, Table 9 · p. 14

6 Conclusion

The study finds that instruction-tuned general-purpose VLMs are strong candidates for structured OCR, while adaptation results vary across collections and models. Its findings are guidance for the evaluated model families, not an exhaustive account of a rapidly changing landscape.

Conclusion

  • Instruction-tuned Qwen2.5-VL, Phi-3.5-Vision, and Gemma-3 are the strongest candidates among eight lightweight VLMs, outperforming several OCR-specialized alternatives in structured generation.Reliable extraction depends on instruction following, field binding, and schema-conformant decoding as well as text recognition.Source: §6 · p. 15
  • Fine-tuning is the most effective adaptation strategy, while few-shot prompting, image preprocessing, and multi-stage training have varying effects across document types and models.Hyperparameter optimization provides the most reliable modest gains, particularly on Regensburg Scalalogy.Source: §6 · p. 15
  • Joint training improves Gemma-3-4B across all three collections but shows signs of negative transfer for Phi-3.5-Vision and especially Qwen2.5-VL-7B.Deployment choices therefore depend on whether institutions prioritize one cross-collection checkpoint or maximum accuracy from dataset-specific checkpoints.Source: §6 · p. 15
  • The study evaluates hyperparameter optimization, preprocessing, and multi-stage training independently, and its findings are bounded to evaluated model families in a rapidly evolving field.The authors identify constrained decoding, layout-aware binding, handwriting adaptation, broader multilingual datasets, and matched-compute fine-tuning comparisons as future work.Source: §6 · p. 15

Related papers

Loading 2610.11818v1…