Source-linked AI summary
Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application
Elias Schubert, Felix Bießmann
TL;DR
Realistic, systematic evaluations of open-source extraction pipelines for high-risk public-sector tasks remain limited. This paper benchmarks OCR+LLM pipelines and standalone VLMs on student applications, finding that none is reliable out of the box and roughly 75% of configurations score below F1 0.25.
Problem
Systematic empirical evidence on freely available models for realistic, complex extraction tasks in high-risk public-sector settings remains limited.
Method
The paper benchmarks open-source OCR+LLM pipelines and standalone VLMs for extracting structured information from heterogeneous student-application PDFs.
Results
Roughly 75% of configurations scored below F1 0.25, while VLMs generally outperformed OCR+LLM pipelines and the best OCR engine matched the top VLM at F1 0.509.
Takeaways & Limitations
Reliable extraction depends strongly on preserving document structure in the input, making data preparation at least as important as downstream model capability.
Takeaways & Limitations
The findings apply specifically to zero-shot, off-the-shelf models evaluated with best available configurations.
Abstract
from arXiv · showhide
The extraction of structured information from unstructured documents represents a critical component of digital transformations in all sectors. While proprietary solutions dominate commercial applications, a rapidly growing ecosystem of open-source Optical Character Recognition (OCR) engines, Large Language Models (LLMs), and Vision-Language Models (VLMs) offers accessible alternatives. However, systematic evaluations on realistic, multi-step extraction pipelines remain scarce. Responsible usage of such extraction tools require comprehensive evaluations on realistic tasks, especially as these solutions will be key components of applications in the public sector that the EU AI act categorizes as high risk. To address this gap we present a comprehensive benchmark assessing the end-to-end performance of open-source systems on a complex real-world document processing task classified as high risk: Student applications for an international study program. We conduct a comprehensive empirical evaluation with state-of-the-art OCR engines, LLMs and VLMs. Our results reveal that while VLMs generally outperform OCR+LLM pipelines, even state-of-the-art open-source models struggle to handle such tasks reliably in zero-shot settings. Only 4 of 35 configurations achieved F1 scores above 0.5, with the best OCR+LLM pipeline matching top VLM performance, though most OCR+LLM combinations performed substantially worse. Roughly 75\% of all configurations scored below 0.25. Model scale influences performance, yet the relationship is non-linear: substantially larger models do not guarantee proportionally better results. Input quality, particularly the structural preservation of OCR output, emerges as a critical factor independent of downstream model capability.
1 Introduction
Structured information extraction from documents is increasingly central to digital transformation, but realistic evaluations of freely available models remain insufficient, especially for high-risk public-sector applications. This work addresses that gap by introducing an evaluation protocol for extracting structured information from heterogeneous student-application PDFs.
- Motivation: Document-based structured extraction increasingly supports data-driven workflows and digital transformation across sectors.These workflows must integrate information from scanned documents and digitally edited forms.
- Motivation: Realistic empirical evaluation is needed because many existing benchmarks address simple or non-high-risk tasks.The need is heightened by rapid technological development relative to regulation, particularly for high-risk settings.
- Contributions: The study devises an evaluation protocol for a high-risk public-sector use case involving student applications for an international study program.The task processes heterogeneous PDFs containing structured and unstructured data, extracts targeted information, and returns a defined structured format.
2 Related Work
Prior benchmarks largely evaluate OCR, extraction, table processing, or structured output in isolation, leaving complex zero-shot, end-to-end pipelines insufficiently assessed. This work addresses that gap by evaluating the complete three-step process with open models.
- Benchmark gap: Existing benchmarks rarely assess complex, zero-shot extraction pipelines that convert documents to text, extract information, and return strictly structured outputs.Such end-to-end tasks reflect practical deployment, whereas existing benchmarks tend to isolate individual components.
- Component-specific benchmarks: OCR research has compared engines using accuracy, processing time, and error rates, with PaddleOCR and EasyOCR informing this work’s model selection.Khan et al. evaluated these engines on the CBC Reports Dataset.
- Component-specific benchmarks: Related studies have examined OCR and multimodal models for table extraction or structured generation, often enforcing predefined schemas or valid JSON outputs.These studies address important components but do not constitute a complete end-to-end extraction pipeline.
- Model and prompting limitations: Closed-weight models such as GPT-4o have outperformed open-weight alternatives, while visual nuance handling and performance remain sensitive to prompt formulation.These findings suggest standardized zero-shot prompting may be suboptimal for visual structured extraction.
- Research motivation: The literature therefore lacks holistic benchmarks of free and open models across document ingestion, information extraction, and structured output generation.This paper directly addresses the complete three-step problem.
3 Methods
The benchmark evaluates structured extraction from 100 academic transcripts using standardized documents, manually defined ground truth, and open-model OCR+LLM and VLM pipelines. It controls model and prompting variables through pretrained models, shared schemas, deterministic generation, and comparable evaluation settings.
- Dataset: 100 academic transcripts from applications to a German Data Science Master’s programme form the benchmark dataset, mostly originating from Asian countries.94 documents are entirely in English, while six are multilingual or German; over 90% of applicants originate from Asian countries.
- Ground truth: 952 unique ground-truth rows contain academic field, course name, grade, and awarded credits, with 0–44 extracted rows per document.Rows were manually extracted using rules for course relevance, language preference, grade formats, missing values, and ambiguous credit columns.
- Extraction workflow: The three-step workflow compares six OCR engines combined with Qwen3 LLMs against five standalone VLMs for structured information extraction.All models use pretrained weights, and structured outputs are enforced through a shared Pydantic schema via Ollama.
- Prompting and execution: Prompts use schemas, verification, delimiters, concise instructions, examples, and context, with temperature set to 0 for deterministic structured generation.Experiments ran primarily on one NVIDIA H200 GPU with 141 GB VRAM, while Qwen3:235b briefly used an NVIDIA B200 GPU with 192 GB VRAM.
4 Conclusion
The benchmark finds that no tested open-source approach is reliable out of the box for complex zero-shot extraction, with performance strongly shaped by input quality. VLMs generally outperform OCR+LLM pipelines, but model scale and recency do not guarantee better results, and practical deployment requires further refinement.
- 4 Conclusion: 75% of configurations scored below an F1 of 0.25, showing that none of the tested approaches is reliable out of the box.The evaluation covered open-source OCR engines, LLMs, and VLMs in pretrained states with best available configurations.
- 4 Conclusion: OCR output that preserves spatial and structural layout yields dramatically better downstream extraction than unstructured, block-fragmented output.MinerU preserves structure through row-wise text grouping and HTML-encoded tables, making input preparation at least as important as model capability.
- 4 Conclusion: VLMs generally outperform OCR+LLM pipelines, yet several models produce near-zero scores because of malformed outputs.VLMs bypass OCR and operate directly on document images, but their performance remains far from reliable.
- 4 Conclusion: Qwen2.5-VL outperformed Qwen3-VL, while larger LLM parameter counts did not guarantee proportionally better results.The conclusion also notes that newer models are not always better, although very small models are discussed as an exception in the supplied passage.
- 4 Conclusion: Improved preprocessing, data quality, prompting, and schema definitions could meaningfully improve results, but reliable deployment requires additional effort and experimentation.These directions concern the zero-shot, off-the-shelf setting evaluated in the benchmark and do not establish an inherent performance ceiling.