Source-linked AI summary
Auditable by Construction: An Ontology-Driven Framework for Trustworthy LLM Analytics in Enterprise Finance
Sergiy Lunyakin
TL;DR
In regulated finance workflows, LLM outputs need grounding that makes claims traceable, relationship-typed, and auditable. The paper introduces CARP for ontology-driven evidence retrieval and finds that retrieval is necessary, while benchmark correctness remains low across conditions.
Problem
Finance workflows require model inputs and outputs to be traceable and independently understandable, but generated claims lack a principled grounding method for auditability.
Method
CARP retrieves evidence using ontology-defined relationship weights and dynamic thresholds while emitting seed, traversal, decision, and provenance records.
Results
4.1% automatic correctness for zero-context inference falls below every retrieval-augmented condition, while benchmark correctness remains low overall.
Takeaways & Limitations
Retrieval is necessary for FinanceBench-style questions because parametric knowledge alone is insufficient and produces no citations for auditability.
Takeaways & Limitations
The evaluation uses a small candidate pool and graph, excludes dense-vector and published graph-RAG baselines, and does not establish enterprise-scale behavior.
Abstract
from arXiv · showhide
Enterprise adoption of large language models in finance is constrained less by fluency than by trust: in Financial Planning and Analysis (FP&A) and other regulated workflows, an answer is usable only if it is traceable to authoritative sources and auditable after the fact. This paper argues that retrieval-augmented generation for enterprise finance should be evaluated on auditability alongside accuracy, and presents the Knowledge-Driven Analytics Framework (KDAF), which builds ontology-driven knowledge systems through six iterative stages and retrieves evidence via Context-Aware Relevance Propagation (CARP), so that every retrieved fact carries its relationship type, confidence, and source lineage. An evaluation on FinanceBench (145 questions) compares KDAF against zero-context inference, BM25, concept-weighted lexical retrieval, and ungrounded graph traversal. First, retrieval is necessary: zero-context inference reaches 4.1% correctness against 10-12% for retrieval-augmented conditions. Second, on answer correctness the retrieval conditions are statistically indistinguishable (KDAF vs BM25: -0.007, 95% CI [-0.021, 0.000]), so accuracy alone does not justify structured retrieval here -- a negative result we report explicitly. Third, on auditability the ordering reverses: KDAF attains the highest citation traceability F1 (0.515), exceeding ungrounded traversal by +0.027 (CI [0.006, 0.050]) and BM25 by +0.052 (CI [0.024, 0.083]), intervals excluding zero. Graph-structured retrieval also admits no evidence from outside the question subject entity (0 of 426 items, against 16.8% and 20.2% for lexical baselines), and every selected item resolves to a complete provenance chain. We argue that auditability, not accuracy, is the axis on which ontology-grounded retrieval earns its cost.
1. Introduction
Finance LLM adoption depends on outputs that are traceable, explainable, and auditable, while ontology-grounded retrieval remains underexplored for FP&A reasoning. KDAF addresses this gap by combining an ontology-driven construction methodology with retrieval that exposes typed paths and provenance.
- Motivation: Regulated financial analysis requires documented, traceable, and independently reviewable model behavior.Trustworthy outputs must support effective challenge and confidence calibration.
- Research gap: FP&A research has largely overlooked knowledge-graph support for planning, variance analysis, and narrative explanation.Prior financial KG applications concentrate mainly on fraud, credit risk, anti-money laundering, and compliance.
- Research gap: Flat retrieval exposes similarity scores, whereas ontology engineering can expose structured reasoning and inspectable source provenance.This distinction motivates evaluating retrieval architectures on auditability as well as answer quality.
- Contributions: KDAF proposes a six-stage methodology for constructing ontology-driven knowledge graphs aligned with FP&A analytical requirements.The methodology addresses cold-start construction through a minimum viable graph approach.
- Contributions: CARP combines ontology-defined relationship weights with dynamic relevance thresholding and emits a complete retrieval trace for LLM grounding.The trace includes seed entities, typed traversal paths, threshold decisions, and end-to-end source provenance.
- Evaluation claim: On FinanceBench, ontology-grounded retrieval improves citation traceability while matching lexical retrieval on answer correctness.The paper therefore treats auditability, rather than accuracy, as the axis on which structured retrieval earns its cost in this setting.
2. The Knowledge-Driven Analytics Framework (KDAF)
KDAF constructs business-aligned ontology-driven knowledge systems through iterative stages, then uses CARP to retrieve bounded, provenance-bearing evidence. Its evaluated retrieval is deliberately hybrid: graph structure governs eligibility and accountability, while lexical evidence dominates final ordering.
- Framework overview: KDAF uses six sequential yet iterative stages to construct ontology-driven knowledge systems for enterprise finance.The framework addresses cold-start construction through specialized algorithms and validation techniques.
- Stages 1–2: Competency questions are decomposed into entities, metrics, and relationships that form a Minimum Viable Graph.The MVG provides the scaffold for the knowledge graph and is validated before broader population.
- Stage 3: Schema-guided LLM extraction populates the graph from heterogeneous financial sources using predefined ontology classes.Newly discovered entities can trigger iterative ontology evolution.
- Stage 4: Contextual relations carry relevance type, confidence, validation status, and source provenance as first-class structure.CARP uses relevance types as traversal weights, distinguishing causal drivers from supporting or merely correlational facts.
- Stage 4: OWL/RDF specifies portable meaning, while property-graph materialization makes traversal-time weighting inexpensive.The RDF relation is reified as a ContextualRelation resource; the graph edge exposes equivalent information natively.
- Stage 6: CARP identifies query seeds, propagates weighted relevance, detects contextual boundaries, and selects bounded evidence with provenance chains.The resulting trace is treated as the audit record rather than as a debugging artifact.
- Stage 6: CARP is a hybrid in which graph structure governs eligibility, reachability, and provenance, while lexical evidence dominates final ordering.The normalized lexical score carries the largest single weight, 0.45, in the final composite.
- Evaluation scope: The public-benchmark evaluation instantiates Stages 3, 4, and 6, not the expert- and organizational-context-dependent stages.Stages 1, 2, and 5 are not exercised because the benchmark does not supply those conditions.
3. Design-Level Evaluation Scenario: Variance Analysis in FP&A
The design-level scenario demonstrates KDAF in a synthetic but realistic FP&A setting focused on quarterly variance analysis. CARP retrieves a focused explanatory subgraph and returns a narrative linked to financial artifacts and source documentation.
- Evaluation scope: The scenario demonstrates system behavior and scope without making quantitative performance claims.Empirical results are reported separately in Section 4.
- Scenario: The scenario models five business units, approximately 50 cost centers, a Q3 horizon, OPEX, and contribution margin.Leadership asks which cost drivers explain the variance versus plan in Q3.
- Knowledge representation: The instantiated ontology covers business units, cost centers, expense categories, forecasts, actuals, variances, and periods.Relationships represent allocation, aggregation, and attribution, while commentary is linked through contextual annotations.
- Retrieval behavior: For a Q3 variance question, CARP seeds relevant entities and retrieves cost centers, expense categories, business units, and explanatory narrative annotations.Entities not directly relevant to Q3 variance are excluded from the retrieved subgraph.
- Output: The system produces a concise narrative identifying primary cost drivers with traceable references to business units, cost centers, and source documentation.The output emphasizes explanatory structure rather than raw tables or aggregate totals and includes the retrieval trace.
4. Empirical Evaluation
The evaluation finds that retrieval is necessary for FinanceBench, but structured graph retrieval does not improve answer correctness over lexical retrieval. Its advantage appears instead in citation traceability, entity-constrained selection, and provenance, while imposing larger contexts and higher latency.
- Traceability F1 is computed from exact evidence-identifier overlap between generated citations and gold sources, with setwise precision, recall, and F1.The metric evaluates answer-attributed citations rather than retrieval selections directly, and multi-source labels receive proportional recall credit.
- 4.1% automatic correctness for zero-context inference falls below every retrieval-augmented condition, and it produces no citations.Retrieval is therefore necessary for these FinanceBench questions and auditability is impossible without citations.
- KDAF and BM25 are statistically indistinguishable on automatic correctness: -0.007 with 95% CI [-0.021, 0.000].The four retrieval conditions span 0.103 to 0.117 automatic correctness, with heavily overlapping intervals.
- 0.515 traceability F1 for KDAF exceeds ungrounded traversal by +0.027 and BM25 by +0.052, with both 95% confidence intervals excluding zero.The comparison supports an attribution advantage, but no condition produces citations that can be accepted without inspection.
- 0 of 426 KDAF-selected evidence items came from another company, compared with 16.8% for BM25 and 20.2% for the lexical hybrid.Graph retrieval enforces the company boundary and resolves every selected item to a complete provenance chain.
- KDAF retrieves the largest mean context, 1,581 tokens versus 1,402 for the lexical hybrid, and has the highest median end-to-end latency, 8.3 seconds versus 6.6 seconds.The evaluation does not support a minimality or efficiency claim for the assembled context.
5. Discussion
The discussion positions auditability as the central value of ontology-grounded retrieval while explicitly bounding the evaluation’s claims. It identifies benchmark, data-quality, measurement, and audit-trace limitations, then proposes metrics and FP&A-specific benchmarks as next steps.
- Scientific Implications: Accuracy alone does not justify graph-structured retrieval, while its architectural value lies in auditability and reconstructable evidence paths.The discussion reports equal correctness against sparse retrieval but argues that auditable retrieval is a deployment precondition in high-stakes domains.
- Scientific Implications: Ontology-grounded retrieval may transfer beyond FP&A to other enterprise domains requiring high-stakes reasoning over heterogeneous, semi-structured data.Examples include supply chain analytics, healthcare operations, and regulatory reporting.
- Limitations: The framework requires upfront ontology bootstrapping by domain and technical experts, but the paper does not quantify this investment.Minimum viable graphs mitigate cold start, yet fragmented definitions or limited semantic-modeling expertise may remain barriers.
- Limitations: Poor source data and inconsistent planning assumptions can propagate through the graph because the framework improves traceability without correcting source inaccuracies.Reliability therefore remains contingent on underlying data quality and governance.
- Limitations: Correctness comparisons are directional, whereas traceability differences are statistically supported within this benchmark setting.Traceability confidence intervals exclude zero; retrieval-discipline and provenance findings are exhaustive counts over the evaluation.
- Limitations: Citation-overlap F1 does not measure whether auditable traces reduce verification effort, improve reviewer confidence, or catch otherwise missed errors.The paper calls for path faithfulness, explanation consistency, and practitioner measures of reviewer verification effort.
- Limitations: The evaluation is limited to public-filings question answering, so claims about FP&A planning, variance structures, and narrative commentary remain argumentative rather than measured.The Section 3 FP&A scenario is a design illustration, not an executed system.
- Future Work: Future work should construct a realistic FP&A variance-attribution benchmark with causal ground truth, linked commentary, and deliberately conflicting explanations.Planned question types include variance attribution, trend analysis, allocation tracing, and conflict resolution.