Source-linked AI summary
Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild
Changda Zhou, Ziyue Gao, Xueqing Wang, Tingquan Gao, Cheng Cui, Jing Tang, Yi Liu
TL;DR
Real-world physical distortions remain insufficiently evaluated because existing document benchmarks lack either realistic capture or precise digital correspondence. Real5-OmniDocBench reconstructs all 1,355 OmniDocBench v1.5 pages across five physical scenarios with inherited annotations, revealing persistent robustness gaps and enabling factor-wise diagnosis.
Problem
Existing evaluations do not systematically measure how document-parsing models respond to realistic physical perturbations with exact digital correspondence.
Method
The benchmark creates one-to-one physical variants of all 1,355 OmniDocBench v1.5 pages across Scanning, Warping, Screen-Photography, Illumination, and Skew.
Results
The evaluation reveals substantial performance degradation and persistent vulnerabilities, especially under complex 3D distortions and localized illumination variations.
Takeaways & Limitations
Real5-OmniDocBench provides a diagnostic stress test for developing document-parsing systems robust to physical-world conditions.
Takeaways & Limitations
The benchmark filters unusable captures while retaining representative clarity degradation and authentic artifacts from handheld capture.
Abstract
from arXiv · showhide
While Vision-Language Models (VLMs) achieve near-perfect scores on digital document benchmarks like OmniDocBench, their performance in the unpredictable physical world remains largely unknown due to the lack of controlled yet realistic evaluations. We introduce Real5-OmniDocBench, the first benchmark that performs a full-scale, one-to-one physical reconstruction of the entire OmniDocBench v1.5 (1,355 images) across five critical real-world scenarios: Scanning, Warping, Screen-Photography, Illumination, and Skew. Unlike prior benchmark that either lack digital correspondence or employ partial sampling, our complete ground-truth mapping enables, for the first time, rigorous factor-wise attribution of performance degradation-allowing us to pinpoint whether failures stem from geometric distortions, optical artifacts, or model limitations. Our benchmark establishes a challenging new standard for the community, demonstrating that the 'reality gap' in document parsing is far from closed, and provides a diagnostic tool to guide the development of truly resilient document intelligence.
1. Introduction
Real5-OmniDocBench addresses the lack of controlled, realistic evaluation for document parsing in physical conditions. It reconstructs the full OmniDocBench test set across five scenarios, enabling comparable attribution of degradation to physical factors.
- Real-world document perturbations include warping, perspective distortion, non-uniform illumination, and optical artifacts from screen photography.
- Prior in-the-wild datasets lack digital ground-truth correspondence, while partial physical simulations provide limited coverage and diagnosis.
- Real5-OmniDocBench physically reconstructs all 1,355 OmniDocBench v1.5 pages across Scanning, Warping, Screen-Photography, Illumination, and Skew.
- Each digital source receives five physical variants that inherit the original annotations, converting physical distortions into controlled variables for comparable scenario analysis.
- The benchmark provides a diagnostic tool for identifying performance degradation associated with specific physical factors rather than only reporting model failure.
2. Related Work
Prior document-parsing benchmarks broadened evaluation beyond OCR but generally emphasize digital images or uncontrolled real-world samples. Real5-OmniDocBench provides complete digital–physical alignment and decomposes five physical factors for end-to-end parsing evaluation.
- Document benchmarks progressed from single-task OCR toward end-to-end structured parsing, with OmniDocBench evaluating nine document types at full-page, module, and attribute levels.
- Recent benchmarks target formula and table verification, cross-modal layout alignment, or natural-environment documents, but do not jointly provide comprehensive physical scenario coverage and correspondence.
- Real5-OmniDocBench differs from restoration-focused work by evaluating end-to-end parsing and from digital VLM benchmarks by physically reconstructing the complete OmniDocBench corpus.
- Its full-set alignment enables systematic decomposition of Scanning, Warping, Screen-Photography, Illumination, and Skew.
3. The Real5-OmniDocBench Benchmark
The benchmark reconstructs OmniDocBench v1.5 with one-to-one physical variants across five scenarios while preserving annotations and applying quality controls. Its scenarios span digitization artifacts, geometric distortions, display artifacts, lighting changes, and 3D pose variation.
- Design Principles and Overall Architecture: Each of 1,355 source pages maps to five physical variants, producing 6,775 samples with unchanged layout, table, formula, text, and reading-order annotations.
- Scanning: Scanning covers standard, low-quality, slanted, stapled, and bound configurations representing resolution loss, alignment errors, shadows, occlusions, and curvature.
- Warping: Warping models folding, cylindrical curvature, crumpling, corner curling, and book-spine arcs to test spatial robustness under non-rigid deformation.
- Screen-Photography: Screen-Photography varies office monitors, professional displays, laptops, tablets, and mobiles to assess pixel structures and moiré patterns.
- Illumination and Skew: Illumination uses low-light, shadow, color-cast, flashlight, and refraction environments, while Skew varies pitch, roll, yaw, compound rotation, and extreme tilt.
- Quality Audit and Refinement: Quality control combines VLM-based anomaly detection, expert audits, correspondence checks, structural checks, and preservation of representative degradation while removing unusable captures.
- Dataset Completeness: The final dataset maintains 100% coverage of the original 1,355 samples across all five scenarios.
4. Evaluation Methodology and Metrics
The benchmark uses OmniDocBench-compatible metrics and an alignment-first model evaluation protocol to compare parsing performance across digital and physical domains.
- Real5-OmniDocBench remains fully compatible with OmniDocBench for cross-domain diagnostic analysis.
- The evaluation framework separates layout, content, and structural logic through multiple metrics.
- Normalized Edit Distance evaluates character-level precision and reading-order consistency using Levenshtein distance.
- Formula recognition uses CDM, while table evaluation reports TEDS and TEDS-Struct for content and structural reconstruction.
- The overall score combines text NED, table TEDS, and formula CDM, while models are screened for reproducibility against official digital metrics.
5. Experiments and Analysis
Experiments show that physical distortions produce scenario-specific failures, while PaddleOCR-VL-1.6 maintains strong and consistent performance across the benchmark’s five conditions.
- 5.1. Main Results Overview: 93.19 Overall: PaddleOCR-VL-1.6 outperforms Qwen3-VL-235B at 88.90 and Gemini-3 Pro at 89.24.The specialized model uses 0.9B parameters and remains competitive with substantially larger models.
- 5.1. Main Results Overview: 91.25%–94.74%: PaddleOCR-VL-1.6 ranges from Warping to Scanning across the five physical scenarios.The reported range indicates stable performance across diverse physical conditions.
- 5.1. Main Results Overview: Qwen3-VL-235B reaches 89.99% in Warping but falls to 89.27% in Screen-Photography and 86.56% in Skew.
- 5.1. Main Results Overview: The evaluation decomposes robustness into Full-text OCR, Formula Recognition, Table Reconstruction, and Reading Order consistency.
- 5.2. Scenario-Level Diagnostic Analysis: Warping reduces structural fidelity because paper curvature alters spatial relationships between text lines and complicates coordinate-based parsing.
- 5.2. Scenario-Level Diagnostic Analysis: Moiré patterns and localized overexposure can trigger layout fragmentation, causing pipeline tools to misinterpret reflections as structural boundaries.
- 5.2. Scenario-Level Diagnostic Analysis: 0.058 ROE: PaddleOCR-VL-1.6 maintains robust reading-order recovery under large-angle Skew.The result exceeds several large-scale general models in the reported Skew analysis.
6. Discussion
The discussion identifies a pronounced reality gap between digital-born and physical captures, with robustness depending more on visual alignment and domain-specific optimization than parameter scale alone.
- Large general VLMs retain strong semantic priors, but transfer to non-rigid transformations is not strictly correlated with parameter scale.
- The reported bottleneck is visual feature alignment under stochastic distortions rather than linguistic modeling.
- Specialized models can provide higher structural fidelity in constrained-resource settings despite smaller parameter counts.
- Failures of traditional pipeline tools under Skew and Illumination motivate unified architectures that jointly optimize optical correction and structural parsing.
7. Conclusion
Real5-OmniDocBench provides a standardized benchmark for document-parsing robustness across five real-world scenarios, using 1,355 reconstructed images. Its results reveal vulnerabilities in current technologies, especially under complex 3D distortions and localized illumination variations.
- Real5-OmniDocBench evaluates document-parsing robustness across five real-world scenarios using 1,355 reconstructed high-fidelity images.
- The benchmark provides a standardized platform for assessing how physical artifacts affect model performance.
- Current parsing technologies show significant vulnerabilities, particularly under complex 3D distortions and localized illumination variations.
- Compact specialized models achieve high efficiency, but universal robustness remains substantially limited.