PaperlayerView exact revision on arXiv

Source-linked AI summary

FinRCA-Bench: Benchmarking Evidence Retrieval and Reasoning for Financial AI Systems

Pratik Ghawate

arXiv:2608.18534v1cs.AIcs.IR

TL;DR

Financial reconciliation tests whether AI systems can reason over distributed transactional evidence when end-to-end accuracy conflates retrieval with reasoning. FinRCA-Bench separates these factors and shows that changing retrieval alone raises exact 16-class accuracy from 2.05% to 72.44%.

  • Problem

    End-to-end accuracy cannot distinguish reasoning errors from failures to retrieve the distributed, relationship-linked records required for financial reconciliation.

  • Method

    FinRCA-Bench evaluates outcome correctness, evidence access, and evidence sufficiency independently across synthetic reconciliation cases using typed relational retrieval and hidden evidence contracts.

  • Results

    Changing only retrieval raises macro required-record recall from 0.83% to 77.70% and exact 16-class accuracy from 2.05% to 72.44%.

  • Takeaways & Limitations

    Retrieval architecture strongly shapes observed performance, while correct root-cause labels remain insufficient proxies for auditable investigations.

  • Takeaways & Limitations

    FinRCA-Bench is synthetic and models a selected accounts-payable-to-bank lifecycle, so reported absolute accuracies are not production estimates.

Abstract

from arXiv · show

Large language models are increasingly used to support financial operations, but their apparent reasoning performance can depend on whether they receive the right evidence. In financial reconciliation, the evidence needed for diagnosis is distributed across invoices, purchase orders, approvals, allocations, payments, ledger entries, and bank activity, linked by transactional relationships rather than textual similarity. End-to-end accuracy can therefore conflate evidence access with reasoning quality. We introduce FinRCA-Bench, a deterministic synthetic benchmark of 2,250 accounts-payable-to-bank reconciliation cases spanning 14 operational tables, including 1,500 injected failures across 15 causal categories and 750 legitimate or hard-negative cases. Root-cause labels and record-level evidence contracts are hidden from the model, allowing retrieval to be evaluated independently of answer correctness. We compare Rules/SQL, classical machine learning, dense semantic retrieval, deterministic relational expansion, and Typed Provenance Graph Retrieval (TPGR), a typed traversal restricted to persisted transaction relationships. Rules/SQL reaches 84.97% held-out exact accuracy and classical ML reaches 95.44%. Holding the reasoning model, prompt, and generation settings fixed while changing only retrieval increases macro required-record recall from 0.83% to 77.70% and exact 16-class accuracy from 2.05% to 72.44%. Structural retrieval failures outnumber reasoning failures with sufficient retrieval by 95 to 15; 254 correct predictions occur despite incomplete retrieval, and strict returned-evidence contract accuracy is only 5.72%. On FinRCA-Bench, retrieval architecture strongly shapes observed AI-system performance, and a correct root-cause label is a weak proxy for an auditable diagnosis.

1. Introduction

FinRCA-Bench frames financial reconciliation as a transactional evidence-retrieval problem in which records are linked by persisted relationships rather than textual similarity. Its controlled evaluation separates retrieval from reasoning and shows that retrieval architecture strongly shapes diagnosis accuracy and auditability.

  • Problem setting: Financial reconciliation evidence is distributed across operational records connected by persisted transactional relationships, so textual relevance alone may miss decisive records.An allocation can bridge an invoice and payment while sharing almost no text with either.
  • Benchmark contributions: FinRCA-Bench contains 2,250 deterministic cases over a 14-table, 155,391-row accounts-payable-to-bank corpus, spanning 15 injected causal failure categories and 750 hard negatives.Evaluator-private record-level evidence contracts are derived from known injected causes and hidden from the system.
  • Evaluation design: The benchmark evaluates outcome correctness, evidence access, evidence sufficiency, and semantic support through an evidence-ladder and retrieval-versus-reasoning attribution protocol.The first three levels are measured automatically, while semantic support is defined but left unadjudicated.
  • Main result: 77.70% macro required-record recall and 72.44% exact 16-class accuracy were achieved versus 0.83% and 2.05% when only retrieval changed under fixed reasoning-model settings.The paired accuracy difference was 70.39 percentage points, with a 95% bootstrap CI of 66.06–74.72.
  • Interpretation: 95 structural retrieval failures versus 15 reasoning failures with sufficient retrieval, plus 254 correct predictions despite incomplete retrieval, show that correct labels do not ensure grounded diagnoses.Strict returned-evidence contract accuracy was 5.72%, while classical ML reached 95.44% held-out exact accuracy versus 72.44% for TPGR + LLM.

2. Related Work

FinRCA-Bench differs from prior financial and enterprise benchmarks by evaluating retrieval against ground-truth individual source records across an operational transaction lifecycle. It also separates correct root-cause naming from auditable attribution through record-level evidence contracts over persisted transaction relationships.

  • Evidence supervision: FinRCA-Bench’s distinguishing property is ground-truth supervision over individual source records, enabling retrieval scoring independently of answer correctness.The comparison is property-based and does not claim benchmark priority; several benchmark families are complementary.
  • Financial reasoning benchmarks: FinQA, TAT-QA, and FinanceBench evaluate financial reasoning or open-book QA with prepared contexts, semantically addressable evidence, or reference evidence strings.These benchmarks focus on numerical programs or document QA rather than transaction provenance.
  • Back-office finance: FinAuditing, FinRule-Bench, and FinBalance move toward back-office finance but address disclosures, accounting principles, or source-document reconciliation rather than multi-table operational causal RCA.FinBalance additionally shows that plausible numerical outputs can be linked to incorrect supporting documents.
  • Relational retrieval: Unlike prose-induced or document-oriented graph retrieval, FinRCA-Bench uses operational-record nodes and frozen transaction relationships already persisted in source systems.This makes provenance a first-class retrieval primitive and separates persisted relationships from hypothesized entity matches.
  • Auditable diagnosis: Following attribution-aware QA and OpenRCA’s warning about ungrounded diagnoses, FinRCA-Bench evaluates auditable attribution through record-level evidence contracts over a transaction graph.The benchmark treats naming the correct cause and recovering verified supporting evidence as distinct concerns.

3. Problem Formulation and the Evidence Ladder

FinRCA-Bench frames reconciliation RCA as both diagnosis and evidence recovery, with required records defined independently of the predicted label. Its four-level evidence ladder separates outcome correctness, evidence access, evidence sufficiency, and semantic support, revealing that correct labels need not establish auditable evidence.

  • 3.1 Reconciliation root-cause analysis: The task predicts one of 16 classes: 15 failure categories plus NO_FAILURE, from a routed case anchor and model-visible operational records.The corpus includes entity types such as vendors, purchase orders, invoices, approvals, payments, allocations, GL entries, bank transactions, and audit events.
  • 3.1 Reconciliation root-cause analysis: RCA must recover investigative records, separate causal failures from symptoms, and support diagnoses with evidence, including meaningful absences that content similarity cannot retrieve.Meaningful absence, such as a missing bank counterpart, is treated as a structural stress test for content-based retrieval.
  • 3.2 Evidence contracts: Per-case record recall is defined as |G_c ∩ R_c| / |G_c|, while full-record coverage requires G_c ⊆ R_c.Because required records derive from the known injected cause and remain hidden at inference, retrieval can be evaluated independently of the predicted label.
  • 3.3 The evidence ladder: The evidence ladder independently scores outcome correctness, evidence access, evidence sufficiency, and semantic support; its levels use distinct constructs and are not interchangeable.The framework distinguishes whether the class is correct, whether required records were exposed, whether contracts were satisfied, and whether records support causal statements.
  • 3.3 The evidence ladder: 72.44% outcome correctness collapses to 5.72% strict returned-evidence sufficiency in held-out TPGR results.The figure presents these as different-denominator measures; Level 4 semantic support was defined but not adjudicated and is shown unfilled.
  • 3.4 Research questions: RQ1–RQ3 test schema-aligned diagnosis, dense semantic recovery under relational evidence definitions, and typed multi-hop traversal beyond deterministic single-hop expansion.These questions compare structured deterministic and supervised diagnosis with retrieval strategies suited to transactional relationships.
  • 3.4 Research questions: RQ4–RQ5 test whether improved evidence access improves RCA and whether correct root-cause labels imply sufficient evidence for justification.The questions also distinguish retrieval errors from reasoning errors while holding the downstream reasoning model fixed.

4. FinRCA-Bench

FinRCA-Bench is a deterministic synthetic accounts-payable-to-bank reconciliation benchmark with private causal labels and record-level evidence annotations. Its balanced failure taxonomy, hard negatives, relationally reachable evidence, and invariant-preserving generation support separate evaluation of evidence retrieval and root-cause reasoning.

  • Benchmark composition: Determinism makes each injected cause known, enabling enumerable required-evidence sets while keeping labels, mutation provenance, and oracle evidence private during inference.The operational corpus remains fully visible to models, separating evidence access from hidden ground truth.
  • Benchmark composition: 2,250 cases comprise 1,500 injected failures and 750 NO_FAILURE / hard-negative cases across 15 causal categories with 100 cases each.This creates a 16-class prediction target while retaining legitimate and adversarial non-failure scenarios.
  • Data generation: The generator builds a clean lifecycle that passes structural, temporal, accounting, and bank-statement checks before deterministic injectors mutate eligible groups and record private RCA ground truth.Fixed-point decimal arithmetic, balanced double-entry journals, and recomputed bank statements preserve financial invariants despite economically wrong failures.
  • Relational evidence: 156 of 416 TPGR validation cases contained required evidence reachable only at graph depth two or three, demonstrating that evidence assembly depends on multi-hop transaction relationships.The rule registry classifies 9 of 15 rules as Tier-3 relational reasoning, with registered minimum paths extending to four hops.

5. Methods

The methods evaluate structured diagnosis and evidence retrieval on a frozen operational snapshot while withholding labels, expected evidence, provenance, and failure manifests. Retrieval comparisons isolate evidence access by holding the downstream model configuration fixed, contrasting dense semantic search with deterministic relational methods and TPGR.

  • Structured baselines: 15 deterministic rules cover the failure categories using exact relational and numerical checks, returning ANOMALY, MATCH, or INSUFFICIENT_EVIDENCE before applying a frozen collision policy.The only preregistered validation search selected a 7-day F01 duplicate-invoice window.
  • Structured baselines: 147 structured features represent each case for preregistered logistic regression, random forest, and histogram gradient boosting configurations.Train-fitted preprocessing produces 265 transformed features, with selection based on validation macro F1 and a frozen tie-break hierarchy.
  • Retrieval comparators: 155,391 operational rows are serialized into one-document-per-row Dense RAG, embedded with text-embedding-3-small at 1,536 dimensions and searched exhaustively by cosine similarity.Dense RAG is explicitly treated as a frozen semantic baseline rather than a representation of every modern RAG design.
  • Retrieval comparators: TPGR uses 155,391 source-record nodes and 184,223 edges with 22 executable relation types and a grammar permitting 30 directed transitions across 15 relation types.Its default-deny traversal permits only explicitly specified node, relation, direction, and next-node combinations, excluding semantic similarity, fuzzy identifiers, and inferred Payment-to-Bank-Transaction edges.
  • Controlled comparison: Dense RAG and TPGR share the same downstream model, taxonomy, prompt, output schema, token limit, verbosity, and retry policy, so retrieval is the only varied component in the held-out comparison.The structural comparator separately uses deterministic exact relational expansion to distinguish general relational access from TPGR’s typed multi-hop traversal.

6. Experimental Protocol and Metrics

The experimental protocol freezes benchmark, retrieval, modeling, and prompting choices before held-out evaluation, while separating retrieval-only validation from the canonical held-out LLM comparison. Evaluation combines exact classification metrics with record-level evidence coverage and an independent partition of retrieval sufficiency versus reasoning outcomes.

  • Evaluation design: Protocols were frozen before held-out evaluation; Graph-vs-Relational uses all 416 validation cases, while TPGR-vs-Dense-RAG uses the canonical 439-case held-out set.The Graph-vs-Relational experiment is retrieval-only.
  • Classification metrics: Exact 16-class accuracy is primary, supplemented by macro F1, binary failure-detection F1, and counts of INSUFFICIENT_EVIDENCE and technical failures.Technical failures remain in the canonical classification denominator; paired accuracy uses McNemar testing and paired nonparametric bootstrap uncertainty estimates.
  • Evidence metrics: Evidence evaluation reports macro and micro required-record recall, full-record coverage, strict full-contract coverage, and citation-ID validity.Strict full-contract coverage includes stricter evidence semantics, including applicable absence-based conditions.
  • Failure attribution: TPGR cases are independently partitioned by evidence sufficiency and label correctness into structural retrieval failure, reasoning failure, success, incomplete-retrieval success, or technical failure.A correct class does not retroactively establish retrieval success, and an incorrect class is not automatically a reasoning failure.

7. Results

Results show that structured and relational access makes the benchmark learnable, while frozen dense retrieval largely fails to recover required evidence. TPGR substantially improves retrieval and downstream classification, but retrieval completeness and auditable evidence-contract satisfaction remain far below label accuracy.

  • Structured baselines: 95.44% held-out exact accuracy is achieved by classical ML, versus 84.97% for Rules/SQL, a 10.48 percentage-point improvement.In the paired comparison, ML is uniquely correct on 58 cases and Rules/SQL on 12; exact McNemar p = 2.25 × 10⁻⁸.
  • Dense retrieval: 0.83% macro record recall and 2.05% exact 16-class accuracy show Dense RAG rarely retrieves benchmark-required evidence.Dense RAG achieves 0/437 full-record coverage, returns INSUFFICIENT_EVIDENCE on 428 of 439 cases, and averages 39.93 irrelevant records out of 40.
  • Retrieval comparison: 78.70% macro record recall for TPGR exceeds Relational Retrieval’s 68.88% on validation, while full-record coverage rises to 36.30% from 15.14%.TPGR’s absolute recall improvement is 9.82 percentage points, and 88 validation cases are fully covered through multi-hop retrieval.
  • Controlled held-out comparison: 77.70% held-out macro record recall and 72.44% exact classification are achieved by TPGR + LLM, compared with 0.83% and 2.05% for Dense RAG + LLM.The controlled comparison changes only retrieval; TPGR uses 19.56 records and approximately 2,370 source tokens per case versus 40 records and approximately 5,531 tokens.
  • Error analysis: 95 structural retrieval failures outnumber 15 reasoning failures with sufficient retrieval, while 254 cases are correct despite incomplete retrieval.The 437 TPGR model-context cases also include 64 successes with sufficient retrieval and 9 model-context technical failures.
  • Evidence sufficiency: 72.44% label accuracy, 36.38% complete required-record retrieval, and 5.72% strict evidence-contract accuracy measure materially different levels of system performance.TPGR + LLM satisfies the strict full evidence contract for 80/437 cases, whereas complete required records are provided for 159/437 cases.

8. Discussion

The discussion argues that retrieval architecture can dominate observed LLM performance in relational financial RCA, while outcome correctness, evidence access, and evidence sufficiency must be evaluated separately. It also frames relationship-aware, absence-sensitive provenance and auditable evidence as requirements for broader enterprise AI systems.

  • Retrieval and reasoning: 72.44% exact RCA accuracy was achieved when the same model changed evidence sources, showing retrieval can dominate observed LLM performance.The controlled experiment changed only the retrieval layer, making operational facts and relationships available to downstream reasoning.
  • Baselines and scope: 95.44% is an upper reference for construction-aligned classical ML, which was the study’s strongest classifier on stable structured features.The discussion cautions that schema privilege limits transferability and that the baseline does not provide generative explanations or evidence-grounded diagnoses.
  • Evidence sufficiency: 254 correct predictions despite incomplete retrieval show that evidence access and label correctness are dissociable and must be measured separately.These cases may reflect redundant diagnostic signals, stricter evidence contracts, or model shortcuts from partial patterns.
  • Future retrieval design: Zero-recall classes motivate scoped, time-bounded provenance for entity history and anti-joins or expected-counterpart checks for absence evidence.Similarity-inferred links may be useful, but their confidence and derivation should remain distinguishable from authoritative provenance.
  • Evaluation recommendations: Benchmarks should report outcome correctness, evidence access, and evidence sufficiency separately, with semantic citation support as a possible fourth layer.The study defines semantic citation support but does not complete its adjudication.

9. Limitations and Threats to Validity

FinRCA-Bench’s synthetic scope, aligned generation and features, incomplete comparisons, and untested evidence assumptions limit how broadly its findings generalize. Evaluation also remains constrained by missing human support adjudication, single-model configurations, and technical failures.

  • Dataset and Benchmark Construction: The synthetic accounts-payable-to-bank lifecycle omits production schema drift, inconsistent identifiers, undocumented conventions, delayed ingestion, manual adjustments, additional systems, imbalance, and simultaneous causes.The rule engine and ML feature registry are aligned with the generative process, potentially simplifying evaluation.
  • Retrieval Baselines: Dense RAG is only a semantic baseline, omitting BM25, metadata filters, decomposition, reranking, entity linking, and hybrid structured retrieval.The study identifies a hybrid baseline as its most important missing comparison, with possible gains especially for one-hop identifier co-occurrences.
  • Retrieval Baselines: TPGR assumes a clean schema, omits some non-authoritative links, and requires entity resolution or probabilistic linkage plus dedicated negative-evidence operators in real deployments.These limitations materially affect coverage for F11, F13, and F15.
  • Attribution and Comparisons: The 70.39-point TPGR–Dense RAG difference cannot isolate graph structure because TPGR also reaches relevant records; relational expansion recovers 68.88% macro recall in validation-only retrieval.The comparison separates retrieval reach only partially and has no downstream reasoning arm.
  • Attribution and Comparisons: The 254 correct labels with incomplete retrieval show that evidence contracts may exceed minimum classifier needs or permit shortcuts from partial patterns.Attribution counts remain provisional until a minimal-sufficient-evidence ablation tests this assumption.
  • Evaluation Scope: The Direct-LLM arm lacked held-out evaluation, the challenge split remains unevaluated, citation support lacks human adjudication, and experiments use one LLM configuration and embedding model.The 5.72% strict returned-evidence contract score cannot substitute for human support evaluation, while retained TPGR technical failures further constrain interpretation.

10. Conclusion

FinRCA-Bench shows that retrieval architecture strongly shapes observed financial RCA performance when evidence must be reconstructed across relational records. Its conclusion is that auditable diagnosis requires provenance-preserving evidence access and evaluation beyond outcome accuracy.

  • Retrieval and reasoning: Changing only retrieval raised macro required-record recall from 0.83% to 77.70% and exact accuracy from 2.05% to 72.44% with fewer records and source tokens.Structural retrieval failures outnumbered reasoning failures with sufficient retrieval by 95 to 15.
  • Benchmark findings: Rules/SQL reached 84.97% held-out exact accuracy, while classical ML reached 95.44%, showing that structured classification is tractable.Classical ML remained the best classifier in the study.
  • Benchmark findings: Dense RAG achieved 0.83% macro record recall and 2.05% LLM accuracy in the evaluated configuration, which was misaligned with relational evidence requirements.The conclusion limits this result to the evaluated configuration rather than semantic retrieval in general.
  • Auditability: A correct root-cause label is a weak proxy for an auditable diagnosis, so remediation and audit systems need provenance-preserving evidence access and independent support evaluation.The paper also identifies explicit negative-evidence operators and uncertainty-aware record linkage as requirements.
  • Auditability: The long-term target is a diagnosis reconstructable, challengeable, and verifiable from underlying financial records, alongside evaluation that distinguishes such diagnoses from label prediction.FinRCA-Bench is designed to separate evidence access from reasoning quality through hidden root-cause labels and record-level evidence contracts.

Reproducibility and Data Statement · Appendix A. Operational Tables and Row Counts · Appendix B. Preregistered Direct-LLM Arm

The benchmark and experimental artifacts are frozen and reproducible, while the appendices document operational row-count reporting and limit the preregistered Direct-LLM arm to incomplete validation evidence without a completed held-out run.

  • Reproducibility and Data Statement: The benchmark is deterministic and synthetic, with frozen dataset identity SHA-256 c73ad3e98575cb4093b1b3898f759d69c57a97840b4d39661b983e91692761d8.
  • Reproducibility and Data Statement: The repository retains frozen method specifications, manifests, predictions, hashes, paired tests, and evaluation artifacts.
  • Reproducibility and Data Statement: The code, benchmark-generation pipeline, experimental specifications, evaluation scripts, predictions, manifests, and reproducibility artifacts are publicly available.The supplied passage provides the repository URL: https://github.com/PratikGhawate/FinRCA-AI-.
  • Appendix A. Operational Tables and Row Counts: Table A1 reports final post-injection operational row counts.
  • Appendix A. Operational Tables and Row Counts: Appendix D note D1 relates the Table A1 counts to the 155,391-row serialized corpus.
  • Appendix B. Preregistered Direct-LLM Arm: The preregistered Direct-LLM baseline used label-blind packets from 14 operational tables without semantic retrieval or graph traversal.
  • Appendix B. Preregistered Direct-LLM Arm: The frozen Direct-LLM configuration used gpt-5.6-sol, medium reasoning effort, and structured JSON output.
  • Appendix B. Preregistered Direct-LLM Arm: Only 63 of 96 planned validation calls completed, 21 cases completed all three repetitions, and no completed held-out run exists.

Appendix C. Review-Sensitive Claim Boundaries

The appendix narrows how FinRCA-Bench results should be interpreted: the completed held-out comparison isolates retrieval architecture, while several validation, measurement, baseline, and evidence-sufficiency boundaries limit broader claims. Correct labels do not establish complete retrieval or fully grounded reconciliation.

  • Comparison scope: TPGR versus Dense RAG is the completed held-out downstream comparison with the LLM configuration fixed, isolating retrieval architecture rather than graph structure alone.TPGR versus Relational Retrieval was validation-only, with no held-out downstream LLM comparison.
  • Comparison scope: Dense RAG is one frozen semantic baseline, so its failure is not generalized to every semantic, hybrid, or reranked RAG design.The comparison supports a bounded claim about this baseline rather than semantic retrieval generally.
  • Evidence validity: Citation-ID validity confirms that IDs reference supplied context records, but semantic citation support and fully grounded reconciliation were not manually adjudicated.Level 4 of the evidence ladder is defined but unmeasured.
  • Evidence sufficiency: A correct RCA label does not imply complete retrieval or sufficient returned evidence, and retrieval-versus-reasoning attribution is relative to benchmark evidence contracts.Those contracts encode one notion of sufficiency rather than an exhaustive standard.
  • Baseline interpretation: Structured baselines are construction-aligned and schema-privileged, making their accuracy an upper reference rather than a production or transfer estimate.Their benchmark performance should not be read as evidence of deployment performance or generalization.
Loading 2608.18534v1…