RaV-IDP: A Reconstruction-as-Validation Framework for Faithful Intelligent Document Processing

Pritesh Jha

Source-linked AI summary

RaV-IDP: A Reconstruction-as-Validation Framework for Faithful Intelligent Document Processing

Pritesh Jha

arXiv:2604.23644v1Apr 26, 2026cs.CVcs.AI

TL;DR

Existing intelligent document processing pipelines extract structured content without directly verifying fidelity to the source, while model confidence measures certainty rather than correspondence. RaV-IDP reconstructs each entity, compares it with an immutable original crop, and triggers structured GPT-4.1 vision fallback when fidelity is low. Fidelity correlates with ground-truth quality, fallback recovers failed extractions, and the full pipeline outperforms evaluated open-source systems on DocVQA, although its aggregate DocVQA improvement is small.

Problem
Existing IDP pipelines lack intrinsic verification that extracted tables, images, and text faithfully represent document regions, while model confidence reflects inference certainty rather than source correspondence.
Method
RaV-IDP reconstructs each extracted entity, compares the reconstruction with the unmodified source crop, and routes low-fidelity outputs to structured GPT-4.1 vision fallback re-extraction.
Results
Fidelity scores correlate with ground-truth table quality at Spearman ρ = 0.800 and native-PDF quality at ρ = 0.877, while fallback recovers 38.1% of failed table extractions.
Takeaways & Limitations
The framework supports label-free, source-grounded validation and routes failed entities for another extraction attempt rather than only filtering them.
Takeaways & Limitations
The aggregate full-versus-no-RaV DocVQA improvement is small at +0.0018 ANLS on 300 questions, despite per-question gains and losses.

Abstract

from arXiv · show

Intelligent document processing pipelines extract structured entities (tables, images, and text) from documents for use in downstream systems such as knowledge bases, retrieval-augmented generation, and analytics. A persistent limitation of existing pipelines is that extraction output is produced without any intrinsic mechanism to verify whether it faithfully represents the source. Model-internal confidence scores measure inference certainty, not correspondence to the document, and extraction errors pass silently into downstream consumers. We present Reconstruction as Validation (RaV-IDP), a document processing pipeline that introduces reconstruction as a first-class architectural component. After each entity is extracted, a dedicated reconstructor renders the extracted representation back into a form comparable to the original document region, and a comparator scores fidelity between the reconstruction and the unmodified source crop. This fidelity score is a grounded, label-free quality signal. When fidelity falls below a per-entity-type threshold, a structured GPT-4.1 vision fallback is triggered and the validation loop repeats. We enforce a bootstrap constraint: the comparator always anchors against the original document region, never against the extraction, preventing the validation from becoming circular. We further propose a per-stage evaluation framework pairing each pipeline component with an appropriate benchmark. The code pipeline is publicly available at https://github.com/pritesh-2711/RaV-IDP for experimentation and use.

1. Introduction

RaV-IDP addresses the gap between model confidence and extraction fidelity by reconstructing extracted entities and comparing them with original document regions. Its validation loop supports entity-specific scoring, fallback re-extraction, enrichment, and staged evaluation.

Motivation: Reconstruction tests fidelity by rendering an extracted representation and measuring its resemblance to the original document region without labels, model internals, or domain-specific rules.A divergence between reconstruction and source indicates information loss or distortion.
Motivation: Extraction errors can silently propagate because existing IDP pipelines provide no intrinsic signal of whether structured output faithfully represents the source.Model confidence measures inference certainty rather than correspondence to the document.
Contributions: The comparator anchors against the unmodified source crop rather than the extraction, preventing a wrong extraction that reconstructs cleanly from passing validation.This bootstrap constraint makes the fidelity signal non-circular.
Contributions: RaV-IDP inserts dedicated reconstructors and comparators after extraction for tables, images, and text.The framework treats reconstruction-as-validation as a pipeline-level architectural pattern applicable across entity types.
Contributions: Low-fidelity entities trigger structured GPT-4.1 vision re-extraction on the same source region, after which validation runs again.Entities that fail both passes remain available with fidelity information for downstream filtering or escalation.
Contributions: Image enrichment runs after extraction to add descriptions, extracted text, and structured data, while a per-stage evaluation framework matches components with appropriate benchmarks.The evaluation includes whether fidelity scores correlate with ground-truth extraction quality.

2. Related Work

Prior document systems advance layout analysis and extraction but generally lack an internal mechanism for verifying source fidelity. RaV-IDP differs by using reconstruction-based, label-free validation and image enrichment for downstream retrieval.

Layout Analysis: Large annotated corpora established modern benchmarks for document layout analysis, including mAP@IoU on PubLayNet and broad manual annotation in DocLayNet.DocLayNet contains 80,863 manually annotated pages across six domains and eleven element classes.
Extraction Systems: Existing extraction pipelines span detectors, document transformers, image-only transformers, and commercial APIs, but remain extraction-only.They do not verify whether extracted output is faithful to the source document.
Table Evaluation: Table evaluation methods such as TEDS and graph-based approaches require labeled ground-truth annotations, whereas RaV-IDP computes table fidelity without labels.The reconstruction-based score is intended for production use on extracted tables.
Confidence Signals: OCR confidence and human-escalation workflows rely on model-internal probability signals rather than direct comparison with the document.RaV-IDP instead uses the document as the reference for fidelity.
Image Retrieval: Existing systems return opaque figure crops, while RaV-IDP enriches images with descriptions, visible text, and structured data for retrieval-augmented generation.The enrichment addresses the limited usefulness of raw pixels to vector stores and language models.

3. Problem Formulation

RaV-IDP formalizes extraction, reconstruction, and comparison around immutable original crops, then applies entity-specific fidelity metrics and acceptance rules. The design explicitly prevents circular validation and adapts comparison to tables, images, and text.

3.1 Document and Region Model: Each detected region contains a bounding box, entity type, and immutable crop from the unmodified page, which serves as the validation anchor.The crop is stored when layout detection occurs and is never altered downstream.
3.2 The Bootstrap Constraint: The extractor maps a region to a structured entity, the reconstructor maps that entity to a comparable representation, and the comparator scores fidelity against the source crop.The formal pipeline is ε(rᵢⱼ) = Eᵢⱼ, ρ(Eᵢⱼ) = R̃ᵢⱼ, and fᵢⱼ = σ(R̃ᵢⱼ, cᵢⱼ) ∈ [0, 1].
3.2 The Bootstrap Constraint: The comparator must receive the original crop cᵢⱼ rather than extraction Eᵢⱼ, because comparing against the extraction would let a wrong output pass if it reconstructs cleanly.The constraint is enforced at the interface level by denying the comparator access to the extraction.
3.3 Fidelity Formulas: For standalone image crops, SSIM is excluded and fidelity is defined as f = f_struct.This evaluation setting differs from production table fidelity with full-page context.
3.5 Validity of Per-Entity Comparison Metrics: Table fidelity combines visual and structural comparison, with SSIM assessing alignment and structural CER assessing cell-content agreement against independent OCR.The two channels separately test table shape and content fidelity.
3.5 Validity of Per-Entity Comparison Metrics: Image fidelity uses pHash and sharpness because extraction errors appear as crop misalignment, truncation, or resolution loss.pHash tolerates minor rendering differences while remaining sensitive to content changes.
3.5 Validity of Per-Entity Comparison Metrics: Text fidelity compares extracted text with an independent OCR rereading or embedded native-PDF text stream, avoiding circular reference to the extraction.Native PDF text streams avoid reintroducing OCR error into the reference.
3.5 Validity of Per-Entity Comparison Metrics: Model confidence is not a substitute for fidelity because a systematically wrong model can remain confidently wrong.Reconstruction-based validation uses the document rather than the model’s own probability assessment.

4. Architecture

RaV-IDP routes documents through quality-aware preprocessing, entity extraction, reconstruction, comparison, fallback, and enrichment while preserving original crops as validation references. Entity-specific reconstructors and comparators produce fidelity and provenance for downstream use.

Pipeline Flow: The pipeline sequences eight components from quality classification and layout detection through extraction, reconstruction, comparison, fallback, and context enrichment.Layout detection precedes preprocessing because it requires the raw document; preprocessing operates on detected regions or classifier-selected inputs.
Document Quality Classifier: Quality classification is limited in the current release to basic clean/degraded checks based on skew angles, with a more detailed classifier deferred to future scope.The planned classifier includes additional page-level classes and preprocessing routes.
Document Quality Classifier: Each quality class maps to a preprocessing track, while clean pages bypass preprocessing and original crops remain sourced from the unprocessed page.Degraded, photographed, handwritten, and overlapping content receive class-specific transformations.
Layout Detection: Docling detects regions and stores immutable pixel crops from the original page before filtering enclosed text regions.Every comparator receives these crops as its reference.
Entity Extractors: Table, image, and text extractors produce structured records using DataFrames, high-resolution crops, parsed strings, formulas, URLs, and handwritten-text routing as applicable.Standalone table-crop evaluation uses TableTransformer instead of Docling because Docling requires full PDF context.
Reconstructors: The table reconstructor renders DataFrames to HTML and rasterized images while extracting row, column, header, and cell-text signatures for structural comparison.The structural channel also OCRs the original crop.
Reconstructors: The image and text reconstructors use pHash, sharpness, caption adjacency, and independent OCR or embedded text streams to generate comparable references.Native PDFs use their embedded text stream instead of OCR.
Comparators and Fallback: Comparators use original crops, and low-fidelity entities are sent to structured GPT-4.1 vision fallback extraction before validation repeats.Fallback prompts request entity-type-specific JSON fields and include neighboring context as a grounding signal.

5. Evaluation Framework

The evaluation framework isolates each pipeline component with a dataset and metric suited to its function, rather than relying on end-to-end scores. It combines component-specific benchmarks with an empirical test of whether fidelity scores reflect extraction quality.

Evaluation Design: Each pipeline component is evaluated independently with a dataset and metric appropriate to what it does.The framework is designed to prevent end-to-end benchmarks from masking component failures.
Dataset Selection: DocLayNet evaluates layout analysis across six document domains and eleven manually annotated element classes.It supports the layout-analysis stage with multi-domain annotations.
Dataset Selection: PubTabNet provides table image crops and HTML annotations for evaluating table structure detection with structural CER.The image crops permit extractor evaluation without full PDF context.
Dataset Selection: ScanBank, FUNSD, and native arXiv PDFs support image and text extraction evaluation across scanned and born-digital documents.Their annotations or embedded text enable component-isolated evaluation under distinct document conditions.
Validation and Ablation: Stage 4 computes Spearman ρ between fidelity scores and ground-truth extraction quality, testing whether fidelity is a reliable quality signal.Stage 6 additionally uses an ablation design derived from shared full-mode traces without redundant pipeline runs.

6. Results

Across layout, table, image, text, and end-to-end evaluations, RaV-IDP measures extraction fidelity against source regions, routes failures to fallback, and improves structured document processing outcomes.

Entity Extraction: Table structure recognition showed row accuracy 0.596, column accuracy 0.584, exact shape accuracy 0.334, and cell text CER 0.405.Column merging was the dominant failure mode, while image extraction achieved 100% semantic coverage for descriptions and extracted text.
Fidelity Reliability: ρ = 0.800 for table fidelity and ρ = 0.877 for native-PDF text fidelity, demonstrating correspondence with extraction quality.Native PDFs also achieved a 97.1% pass rate, while table gating reached F1 = 0.914 at τ = 0.43.
Fallback Recovery: 38.1% of failed table extractions were recovered through the GPT-4.1 fallback path.Recovered samples reached mean fidelity 0.609, crossing the acceptance threshold despite a small aggregate mean Δ fidelity of +0.005.
End-to-End Performance: Excluding failed entities without fallback reduced answerable rate from 44.7% to 14.3% and ANLS from 0.4224 to 0.1408.This ablation identifies fallback routing, rather than filtering alone, as the mechanism preserving downstream coverage.
End-to-End Performance: 0.4224 ANLS on DocVQA exceeded all open-source extraction pipelines, while direct GPT-4.1 vision achieved 0.9372.RaV-IDP is evaluated as a structured IDP pipeline producing EntityRecord outputs, whereas direct vision reading bypasses structured extraction.
Fallback Efficiency: Selective fallback increased pass rate from 61.2% to 76.0% using 145 API calls instead of 500, reducing API cost by 71%.The full pipeline's mean fidelity was 0.541 versus 0.539 without RaV-IDP.

7. Limitations

RaV-IDP shows strong table-fidelity validation but has important scope, calibration, dataset, and dependency limitations. Several failure modes remain, including shared OCR errors, missing formula detection, and weak end-to-end gains.

TATR misses boundary-adjacent table rows in approximately 40% of PubTabNet samples, while column merging reduces exact shape accuracy to 0.334 versus row accuracy of 0.596.The row issue is specific to standalone image crops and does not reproduce on full PDFs with natural margins.
The full-versus-no-RaV DocVQA aggregate ANLS delta is only +0.0018 across 300 questions, despite improvements on 9% of questions.Wins and losses occur with roughly equal frequency, so the aggregate nearly cancels.
Table fidelity correlates strongly with ground-truth quality, but image fidelity reliability was not measured because ScanBank crops were trivially identical.A diverse degraded-image benchmark is needed for image validation.
Threshold calibration remains necessary: FUNSD achieves only a 6% pass rate at τ = 0.85, while τ = 0.43 is identified as the optimal table gate point.The lower threshold is expected to increase recoveries while reducing incorrect fallback triggers.
The bootstrap constraint has a blind spot when extractor and reconstructor share the same systematic error, and formula detection is absent with F1 = 0.0.External vision-model access is also required for fallback and image enrichment; without it, those layers are unavailable.

8. Conclusion

RaV-IDP makes reconstruction-based fidelity validation a first-class pipeline component, using source-anchored comparison and fallback routing. Results support the approach for table and native-PDF validation, while ablations show that recovery—not filtering alone—drives the end-to-end benefit.

RaV-IDP validates each extracted entity by reconstructing it, comparing it with the unmodified source crop, and routing low-fidelity outputs to structured vision fallback.The bootstrap constraint prevents circular validation by ensuring the comparator never anchors against the extraction.
Fidelity scores correlate with ground-truth table quality at ρ = 0.800 and with native-PDF quality at ρ = 0.877; table gating reaches F1 = 0.914.The fallback recovers 38.1% of failed table extractions and 24.5% of failed text regions.
The full pipeline achieves 0.4224 ANLS on DocVQA, outperforming all evaluated open-source pipelines by 8-16%.
Gate-only processing collapses to 0.1408 ANLS with a 29.7% pipeline error rate, showing that exclusion without replacement is harmful.The reported improvement comes from routing rejected entities to fallback rather than merely filtering them.
Semantic enrichment produces descriptions, extracted text, and structured chart data for every image entity, supporting image retrieval in RAG systems.Future work includes learned quality classification, diverse image benchmarks, threshold calibration, and native-PDF QA evaluation.

Appendix A: End-to-End Pipeline Walkthrough

The appendix traces RaV-IDP on a real LLaMA 2 page containing tables, figures, and text, illustrating the pipeline across all three entity types.

The walkthrough uses page 6 of the LLaMA 2 paper as a complete trace of RaV-IDP on a document containing all three entity types.The page includes a multirow table, a training-loss figure, and multiple text blocks.

A.1 Input Document

The input is a one-page LLaMA 2 PDF page containing a table, figure, caption, and text sections for end-to-end processing.

The page contains Table 1 with merged row labels, Figure 9 with training-loss curves for four model sizes, a caption, and three text sections.
Figure A1 describes an input page containing 9 detectable entities: 7 text blocks, 1 table, and 1 additional entity.The supplied passage truncates the final entity description.

A.2 Stage 1 — Layout Detection

Stage 1 detects and labels document regions, stores their original pixel crops, and applies containment filtering before downstream processing.

Docling detects 9 regions: 7 text blocks, 1 table, and 1 image, recording each bounding box.
Original pixel crops are extracted from the unmodified page and stored immutably as bootstrap anchors for downstream comparators.
Text regions enclosed within the table bounding box are suppressed by the spatial containment filter.

A.3 Stage 2 — Quality Classification

The pipeline evaluates each extracted entity by reconstructing and comparing it with its original crop, routing low-fidelity cases through fallback while preserving explicit uncertainty.

Each entity follows an Extract → Reconstruct → Compare loop, with the primary fidelity score determining whether the gate fires.If triggered, GPT-4.1 vision fallback runs and the final score and low-confidence flag are attached to the EntityRecord.
All seven text regions score between 0.968 and 1.000, avoid fallback, and achieve median CER below 0.01.The native PDF benchmark reports a 97.1% pass rate and mean CER = 0.048 across 10,028 regions.
The image comparator reports pHash similarity = 0.981 for the training-loss chart crop, so no fallback is triggered.
The table extraction has the correct 8-row, 7-column shape but misses or truncates row-span cell content, producing primary fidelity f = 0.322 < τ = 0.75.Low SSIM and high structural CER expose misaligned span structure, causing the gate to fire.
GPT-4.1 correctly identifies the table’s row-span cells, improving fidelity from 0.322 to 0.387, but the result remains below the 0.75 threshold.The complex layout, small font, and fine ruling lines make the table difficult to reconstruct faithfully.
The failed table is emitted with low_confidence = True and re_extraction_count = 1, while the higher-fidelity fallback output remains available to downstream consumers.The pipeline reports the uncertainty instead of silently promoting the fallback to a passing result.