PaperlayerView exact revision on arXiv

Source-linked AI summary

LAVA: Logic-Aware Validation and Augmentation Framework for Large-Scale Financial Document Auditing

Ruoqi Shu, Xuhui Wang, Isaac Wang, Yanming Mai, Bo Wan

arXiv:2608.16763v1cs.AI

TL;DR

Financial document validation must handle heterogeneous layouts, context-dependent content, and embedded business rules reliably for high-stakes workflows. LAVA addresses this with modular extraction, augmentation, and symbolic verification, achieving strong factual and numerical reliability with lower computational overhead on a large real-world benchmark.

  • Problem

    Existing document-understanding evaluations focus largely on perceptual and question-answering tasks, leaving multi-document financial validation underrepresented despite its accuracy and auditability requirements.

  • Method

    LAVA is a backbone- and domain-agnostic pipeline combining layout-preserving extraction, domain-aware augmentation, and hybrid symbolic and arithmetic verification.

  • Results

    LAVA achieves high factual alignment and numerical reliability with lower computational overhead than monolithic prompting on a large real-world benchmark.

  • Takeaways & Limitations

    Structured, multistage reasoning supports fine-grained error attribution, robustness in compliance-sensitive workflows, and transfer to structurally similar data.

  • Takeaways & Limitations

    The proprietary dataset limits direct assessment of demographic bias, so production deployment requires rigorous ongoing audits for fair and equitable outcomes.

Abstract

from arXiv · show

Financial document validation in production, such as payroll auditing, tax compliance, and loan underwriting, demands exceptional accuracy, consistency, and reproducibility under strict enterprise constraints. In practice, documents arrive with heterogeneous layouts and formats, semantically rich and context-dependent content, and embedded business rules that current pipelines struggle to process reliably. We introduce LAVA (Logic-Aware Validation and Augmentation), a modular, backbone-agnostic pipeline built on multimodal large language models, that integrates a four-stage design: document-rule retrieval, layout-preserving information extraction, auxiliary metadata enrichment, and auditable symbolic/arithmetic verification. LAVA supports robust rule grounding, fine-grained error attribution, and consistent, traceable end-to-end execution, capabilities essential for high-stakes deployment. Evaluated on a large real-world benchmark with diverse financial documents and dozens of expert-curated validation rules, LAVA outperforms baselines in hallucination control and edge-case handling while maintaining efficient token usage, demonstrating practicality for high-volume, time-critical validation.

1 Introduction

Financial document validation requires accuracy, consistency, auditability, and robust rule enforcement across heterogeneous, semi-structured documents, yet existing models and benchmarks incompletely support compliance-critical validation. LAVA addresses this gap with a modular, backbone-agnostic framework combining layout-informed extraction, augmentation, hybrid symbolic reasoning, and auditable verification.

  • Motivation: Financial institutions process millions of semi-structured documents daily, where a single validation error can cause regulatory penalties, financial losses, or reputational damage.These documents include statements, invoices, and tax slips with multiple pages, irregular layouts, noisy scans, and non-standard PDFs.
  • Problem: Enterprise validation requires symbolic rule enforcement, cross-field consistency, and multi-step logical coherence beyond perception or reasoning alone.Compliance-critical validation logic remains underexplored in existing benchmarks.
  • Problem: Existing models applied directly to validation still produce hallucinations, offer limited reasoning traceability, and adapt fragily across regulatory schemas.Enterprises often combine general-purpose models with rigid rule-based modules to address this gap.
  • Our Work: LAVA formalizes financial document validation as a multi-document reasoning task and provides a modular, backbone-agnostic framework for accurate, auditable validation of layout-complex financial documents.Its design targets fine-grained error attribution, rapid adaptation, and reproducibility across structurally similar collections.
  • Evaluation: LAVA achieves competitive factual accuracy and symbolic correctness with lower computational overhead than baseline MLLM pipelines on a large-scale industrial benchmark curated by senior industry experts.The benchmark reflects real regulatory conditions and the results demonstrate robustness and cost-effectiveness in realistic financial validation scenarios.
  • Reasoning Strategy & Auditability: LAVA unifies factual and contextual templates with symbolic and arithmetic reasoning through explicit formula generation and an external checker fallback.The evaluation covers symbolic correctness, factual alignment, and hallucination control in realistic validation workflows.

2 Related Work

Related work has advanced visually rich document understanding and layout-guided encoding, while LLM verification and traditional rule engines remain insufficient for noisy documents with embedded business rules. LAVA addresses this gap through layout-derived structures, modular rule checks, reproducible logic tracing, and metadata-informed prompting.

  • Visually Rich Document Understanding: Recent document-understanding models integrate layout and visual cues through visual patches, 2D positional tokens, multimodal retrieval, consistency constraints, multi-task distillation, and instruction tuning.Examples include DocLayLLM, VisDoM, 3MVRD, and LayoutLLM.
  • Layout-Guided Document Encoding: LayoutLMv3, FormNet, and DocFormer combine textual, spatial, and visual cues through large-scale pretraining for form-style entity extraction.These backbones remain embedding-level and are not optimized for symbolic validation.
  • Layout-Guided Document Encoding: LAVA uses layout-derived structures for modular business-rule checks and reproducible logic tracing, combining rule transparency with neural adaptability through metadata-informed prompting.This design targets compliance-critical validation.
  • LLM Verification and Rule-based Validation: LLM factuality methods improve reliability in clean text and formal reasoning, but falter on noisy heterogeneous documents with long-range dependencies and embedded business rules.The cited approaches include self-checking, retrieval-augmented prompting, symbolic grounding, and rectification.

3 Method

LAVA formulates financial document validation as a human-in-the-loop task that maps document sets and natural-language intents to auditable, logic-grounded verification outputs. Its modular pipeline retrieves relevant document-rule pairs, preserves layout-aware information, enriches metadata, and delegates arithmetic verification to deterministic tools.

  • Problem Formalization: LAVA assists users in verifying whether heterogeneous financial document sets satisfy business rules under complex layouts, domain logic, and noisy inputs.The task is explicitly framed for underwriters, compliance officers, and fraud analysts.
  • Inputs and Objective: Given scanned documents D and a natural-language validation intent q, the system produces validation outputs containing supporting documents, retrieved rules, Pass/Fail labels, and explanation traces.This formulation supports multi-document, logic-grounded reasoning with auditability and interpretability.
  • Document-Rule Retrieval: Bidirectional filtering selects temporally valid documents and executable rules tailored to q, using semantic relevance, document compatibility, temporal constraints, and rule metadata.The closed-loop process reduces irrelevant candidates and downstream reasoning load while preserving interpretability.
  • Structured Modality Conversion: Layout-preserving extraction converts filtered documents into compact hybrid markup that retains visual structure, recovers noisy content, groups fragmented tokens, preserves visual regions, and removes boilerplate.The resulting representation bridges scanned formats and language-model input while reducing token usage and maintaining interpretability.
  • Auxiliary Metadata Enrichment: Auxiliary metadata routes each retrieved rule as symbolic or arithmetic and augments prompts with document language, types, and domain-specific terminology.Rule classification uses a lightweight LLM, while metadata is extracted from layout-preserving outputs with token-based, lexical, and structural heuristics.
  • Arithmetic Processor: Arithmetic rules generate task-specific formulas for execution by deterministic external engines rather than relying on LLM direct computation, reducing hallucinations and numeric instability.A checker LLM validates formula-result alignment, with failures triggering fallback handling.

4 Experiments

LAVA is evaluated on a heterogeneous real-world Canadian mortgage-document benchmark using rule-level testing and task-specific metrics for hallucination, edge-case handling, and token cost. It achieves the lowest reported failure rates across metrics and categories, with particularly strong gains in complex reasoning and constrained arithmetic, and transfers effectively to public DocVQA data.

  • Evaluation Setup: The benchmark contains around 1,000 scanned Canadian mortgage documents spanning tax forms, financial statements, and legal agreements with diverse layouts and logical dependencies.Documents were sampled from a proprietary database and selected for semi-structured formats and rich logical dependencies.
  • Evaluation Setup: Evaluation uses atomic stakeholder-defined validation rules matched to applicable document types, enabling rule-level testing rather than intent-level assessment.Document-type metadata is included because correct rule-set matching is required for fair validation assessment.
  • Evaluation Metrics: The metrics target factual hallucinations, numerical infidelity, edge-case failures, and per-rule-check token cost, emphasizing false-positive suppression in compliance-sensitive validation.Edge cases comprise about 10%–25% of document–rule pairs, while token counts are aggregated across the evaluation set.
  • Main Results: LAVA also performs strongly on a manually ruled DocVQA subset, supporting transfer to public data despite the greater semantic and reasoning complexity of real-world validation.The DocVQA subset used five manually defined rules per document.
  • Ablation Study: The ablation study compares full LAVA with Knowledge Extraction variants and versions removing Info Augmentation or the Arithmetic Processor using exact ground-truth match failure as a unified metric.The Knowledge Extraction variants progressively reduce structural detail and alter the input format.

3. No KE omits textual knowledge entirely,

Ablations show that KE, IA, and AP provide complementary benefits, with KE most critical for preserving structure and multi-step reasoning. LAVA also reduces token usage while adding a small per-rule overhead that supports efficient, stable validation.

  • Ablation results: Removing KE raises C2, C3, and C5 failure rates to 0.65–0.67, making it the most critical component for structural and relational reasoning.Without KE, the model relies almost entirely on visual reasoning; the resulting C3 error is over twice LAVA’s and C5 error nearly tenfold.
  • Ablation results: Removing IA increases C3 error from 0.28 to 0.45, showing that metadata and domain cues support field semantics and cross-step linking.IA primarily affects logical reasoning rather than computation.
  • Ablation results: Removing AP leaves C1–C3 unchanged but increases C4/C5 error to 0.10/0.56, demonstrating the value of explicit calculation and fallback validation.AP’s effect is concentrated on computation-focused tasks.
  • Operational efficiency: LAVA cuts input tokens by 25%–45% versus VLM and LLM baselines with Field-Level OCR by extracting structured knowledge once and reusing it across rules.The token reduction lowers inference costs and supports faster turnaround while also achieving fewer errors than the baselines.
  • Operational efficiency: Across all ablations, full LAVA adds under 3k tokens per rule, costing ∼$0.009 with Claude 3.7 or ∼$0.006 with GPT-4.1.The overhead is described as outweighed by gains in accuracy and reasoning stability.

5 Conclusion

LAVA is presented as a modular, interpretable, backbone- and domain-agnostic system for enterprise-grade financial document validation. Its multistage design combines layout-preserving extraction, domain-aware augmentation, and symbolic/arithmetic verification to support factual alignment and numerical reliability with lower computational overhead than monolithic prompting.

  • 5 Conclusion: LAVA is a modular, interpretable, backbone- and domain-agnostic system for enterprise-grade financial document validation.The system is designed for broad applicability across financial-document validation settings.
  • 5 Conclusion: Its pipeline combines layout-preserving extraction, domain-aware augmentation, and symbolic and arithmetic verification.These components form a structured, multistage approach to validation.
  • 5 Conclusion: LAVA achieves high factual alignment and numerical reliability on a large real-world benchmark with lower computational overhead than monolithic prompting.The conclusion attributes these outcomes to the structured multistage reasoning evaluated through factual hallucination rate, numerical infidelity rate, and edge-case handling.

6 Ethical Considerations

The study uses anonymized proprietary financial documents under strict institutional privacy protocols and acknowledges algorithmic bias as a risk in financial decision-making. LAVA emphasizes transparency and human accountability, while production deployment would require rigorous ongoing audits.

  • The proprietary financial-document dataset was fully anonymized and handled under strict institutional privacy protocols.
  • The authors acknowledge algorithmic bias as a potential risk in financial decision-making.
  • LAVA is designed to mitigate this risk through accuracy, scalability, auditability, transparency, and human accountability.
  • Production deployment would require rigorous, ongoing audits.

A Appendix · A.1 Prompt used in LAVA

The appendix specifies LAVA’s prompts for symbolic verification and arithmetic processing. These prompts constrain extraction, formula generation, correction, and machine-readable outputs using document knowledge and validation rules.

  • A.1.1 Prompt for Symbolic Reasoner: The symbolic reasoner verifies financial documents against validation rules using document types, languages, and extracted knowledge without inventing values.It is instructed to use structured representations for document layout and semantic fields for contextual information.
  • A.1.1 Prompt for Symbolic Reasoner: The symbolic prompt supplies validation rules and requires outputs to contain only requested fields with normalized keys and YYYYMMDD dates.Missing fields are assigned empty values, and the response must be solely a JSON object.
  • A.1.2 Prompt for Arithmetic Processor: The formula-generation prompt identifies the requested field, extracts its stated numeric value, and records missing or empty values as "NaN".The field name is stored under "field_name" and the stated value under "stated".
  • A.1.2 Prompt for Arithmetic Processor: Formula generation derives a Python-executable expression from relevant numerical fields and stores it under "formula" without computing the result.The prompt permits only raw numerical operations and excludes functions such as "round(...)".
  • A.1.2 Prompt for Arithmetic Processor: Both arithmetic prompts receive extracted knowledge and a validation rule, while formula correction additionally receives the wrong calculation expression.The inputs are delimited as knowledge, validation_rule, and wrong_calculation fields.
  • A.1.2 Prompt for Arithmetic Processor: The arithmetic processor’s correction prompt treats the supplied calculation expression as wrong and searches the extracted knowledge for another correct expression.The corrected expression is returned without computing it or adding explanatory text.

A.2 Representative Validation Rules and Output Examples Used in Experiment

The section illustrates representative validation rules for extracting payroll and tax-document fields, alongside structured outputs indicating field presence and values.

  • Representative Validation Rules: Content-extraction rules check whether current and YTD regular pay or salary amounts appear in documents.
  • Representative Validation Rules: Additional rules verify current and YTD CPP/QPP, potentially labeled Government Pension, in paystubs.
  • Representative Validation Rules: The rules also check whether a Social Insurance Number appears in T4 documents.
  • Output Examples: Example output records field presence and extracted values, marking Employer_Name as present with value organization.
  • Output Examples: The example marks Current_Regular_Pay as absent with an empty value.

2. Conditional Logic Reasoning

The section defines conditional validation rules for EI, numerical cells, pay-period consistency, and arithmetic calculations. These rules specify when checks are skipped, values are valid, and stated figures are compared with expected results.

  • Conditional Logic Reasoning: EI validation branches on Current and YTD EI values, accepting non-zero Current EI, rejecting both blank or zero values, and comparing YTD EI with the cap otherwise.When Current EI is blank or zero, validity is true only if YTD EI is at most the EI cap.
  • Conditional Logic Reasoning: Numerical validation marks empty or symbol-only cells valid, alphabetic cells invalid, and returns {} when all fields pass.The rule applies to non-header table cells.
  • Conditional Logic Reasoning: Pay-period validation computes an expected period number from pay frequency, dates, and frequency caps, then compares it with the stated period number.Missing dates or pay-period values produce an empty stated value and equal=None.
  • Conditional Logic Reasoning: Unconstrained arithmetic checks whether YTD gross pay, net income, and ending balance follow their specified summation and balance equations.The checks cover earnings aggregation, income minus deductions, and starting balance plus deposits minus withdrawals.
  • Conditional Logic Reasoning: Constrained arithmetic validates regular pay as rate × hours and net pay as gross pay minus deductions, rounding results to 2 decimal places.The regular-pay formula becomes empty when rate or hours is missing or zero; a Total Deduction field is used directly when present.

A.3 Evaluation Metrics … A.4 Evaluation Dataset

The evaluation defines three complementary metrics to distinguish factual hallucination, numerical inconsistency, and edge-case errors, then tests them on heterogeneous production-level mortgage documents. The examples show that these failure modes arise from unsupported evidence, arithmetic inconsistency, or misinterpretation of atypical structures.

  • A.3.1 Factual Hallucination Rate (FHR).: FHR marks outputs as factually hallucinated when predicted evidence is unsupported by the document or the predicted formula is not equivalent to the gold formula.Evidence containment normalizes case, punctuation, numeric formatting, and units; formula equivalence uses randomized polynomial identity testing.
  • A.3.1 Factual Hallucination Rate (FHR).: Factual hallucination can fabricate a variable to force arithmetic agreement or repurpose another field as a missing value, despite correct arithmetic or reasoning.Examples change 660.50 to 560.50 or invent Start_Date as 20111014, showing errors caused by introducing absent evidence.
  • A.3.2 Numerical Infidelity Rate (NIR).: NIR isolates numerical inconsistencies after formula or reasoning generation, counting results that deviate from the gold numerical value within absolute or relative tolerances.Formula-generation hallucinations are already counted by FHR, while NIR captures failures to preserve quantitative consistency.
  • A.3.2 Numerical Infidelity Rate (NIR).: The numerical-infidelity example uses faithfully extracted evidence and a correct formula but computes 7633.44 instead of the correct 7733.44.The error reflects arithmetic or quantitative-reasoning inconsistency rather than fabricated or misattributed values.
  • A.3.3 Edge Case Handling (ECH).: ECH evaluates robustness on predefined complex or exception-driven cases, typically 10%–25% of document–rule pairs, where higher ECH indicates weaker generalization.These cases include missing or abnormal values, atypical field combinations, and boundary conditions requiring adaptive reasoning.
  • A.3.3 Edge Case Handling (ECH).: The edge-case example requires excluding a subtotal and subtracting a negative adjustment, but the pipeline instead reuses the subtotal and treats the adjustment as positive.This reflects structural misinterpretation rather than arithmetic error, factual fabrication, or numerical miscomputation.
  • A.4 Evaluation Dataset: The evaluation dataset is a proprietary collection of production-level mortgage application documents from an active industrial workflow.It covers proof of income, property appraisal, account statements, tax forms, and legal agreements, with documents ranging from single-page forms to multi-dozen-page reports.
  • A.4 Evaluation Dataset: Unlike synthetic or visually uniform public datasets, the corpus preserves broad document-category and layout heterogeneity.Table 3 summarizes representative mortgage document types and their key characteristics.
Loading 2608.16763v1…