Source-linked AI summary
FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs
Yan Wang, Keyi Wang, Shanshan Yang, Jaisal Patel, Jeff Zhao, Fengran Mo, Xueqing Peng, Lingfei Qian, Yankai Chen, Víctor Gutiérrez-Basulto, Jimin Huang, Guojun Xiong, Xiao-Yang Liu, Xue Liu, Jian-Yun Nie
TL;DR
It remains unclear whether LLMs can support professional-grade auditing across hierarchical, taxonomy-defined, cross-document financial information, even though disclosures demand inconsistency detection at scale. FINAUDITING is a taxonomy-aligned benchmark built from real-world XBRL filings with three complementary tasks; evaluations show that leading models still struggle across key auditing capabilities, including low FinMR accuracy.
Problem
It remains unclear whether LLMs can support professional-grade auditing that requires reasoning across hierarchical structures, taxonomy-defined relations, and cross-document dependencies.
Source: §1 · p. 2Method
FINAUDITING evaluates structured financial reasoning on real-world US-GAAP XBRL filings through FinSM, FinRE, and FinMR.
Source: §1 · p. 2Results
Leading LLMs still struggle with cross-document reasoning, structured interpretation, and taxonomy alignment; Fin-o1-14B achieves only 13.86% accuracy on FinMR.
Source: §4.3 · p. 6, §5 · p. 7Takeaways & Limitations
FINAUDITING provides a high-fidelity testbed for taxonomy-grounded reasoning and structured retrieval in complex, multi-document financial environments.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Going beyond simple text processing, financial auditing requires detecting semantic, structural, and numerical inconsistencies across large-scale disclosures. As financial reports are filed in XBRL, a structured XML format governed by accounting standards, auditing becomes a structured information extraction and reasoning problem involving concept alignment, taxonomy-defined relations, and cross-document consistency. Although large language models (LLMs) show promise on isolated financial tasks, their capability in professional-grade auditing remains unclear. We introduce FinAuditing, a taxonomy-aligned, structure-aware benchmark built from real XBRL filings. It contains 1,102 annotated instances averaging over 33k tokens and defines three tasks: Financial Semantic Matching (FinSM), Financial Relationship Extraction (FinRE), and Financial Mathematical Reasoning (FinMR). Evaluations of 13 state-of-the-art LLMs reveal substantial gaps in concept retrieval, taxonomy-aware relation modeling, and consistent cross-document reasoning. These findings highlight the need for realistic, structure-aware benchmarks. We release the evaluation code at https://github.com/The-FinAI/FinAuditing and the dataset at https://huggingface.co/collections/TheFinAI/finauditing. The task currently serves as the official benchmark of an ongoing public evaluation contest at https://open-finance-lab.github.io/SecureFinAI_Contest_2026/.
1 Introduction
Financial auditing must detect inconsistencies across large, structurally complex disclosures, while existing evidence leaves unclear whether LLMs can support professional-grade auditing. FINAUDITING addresses this with a taxonomy-aligned benchmark built from real-world XBRL filings and complementary reasoning tasks.
1 Introduction
- Financial disclosures span thousands of pages, making manual inconsistency checks nearly impossible and detection challenging even for advanced AI systems.The challenge involves the scale and structural complexity of the disclosures.Source: §1 · p. 2
- Existing financial NLP benchmarks often omit cross-document dependencies, multi-step structural reasoning, and evaluation against official GAAP taxonomy constraints.The paper also identifies uncertainty about LLMs' ability to support professional-grade auditing.Source: §1 · p. 2
- FINAUDITING reframes financial auditing as structured information reasoning over real-world US-GAAP-compliant XBRL filings through FinSM, FinRE, and FinMR.The tasks assess concept-level semantic alignment, structural and relational consistency, and taxonomy-grounded numerical verification.Source: §1 · p. 2
- The benchmark integrates multi-document structures and GAAP taxonomies, evaluates 13 state-of-the-art LLMs, and is designed to support RAG and multi-agent systems.Its evaluation spans structured retrieval, hierarchical information extraction, multi-step numerical reasoning, taxonomy-driven supervision, and cross-document reasoning.Source: §1 · p. 2
2 Related work
Prior financial NLP research covers several tasks, but existing benchmarks generally do not evaluate the structured, hierarchical, taxonomy-driven reasoning across multiple XBRL documents that FINAUDITING targets.
2 Related work
- Prior financial NLP research has advanced information extraction, semantic understanding, and numerical reasoning.The passage cites work on each of these areas.Source: §2 · p. 2
- Most financial NLP benchmarks use unstructured or semi-structured data from individual documents and omit XBRL's structured, hierarchical, interconnected nature.Early XBRL studies instead address localized subtasks, while related tabular and textual reasoning benchmarks lack taxonomy-driven supervision.Source: §2 · p. 2
- FINAUDITING is designed to evaluate structured, hierarchical, and cross-document financial reasoning grounded in realistic XBRLs.It targets taxonomy-level dependencies not captured by the narrow components evaluated in earlier XBRL efforts.Source: §2 · p. 2
3 FinAuditing
FINAUDITING benchmarks detection of semantic, structural, and numerical inconsistencies across multi-document XBRL filings. It combines three complementary tasks with DQC-grounded data construction, structure-preserving processing, and taxonomy-informed annotation.
3.1 Task Formulation
- FINAUDITING evaluates XBRL inconsistency detection through FinSM, FinRE, and FinMR, covering semantic mismatches, relationship errors, and numerical violations.The benchmark frames auditing as reasoning over filing structure and the reference US-GAAP taxonomy.Source: §3 · p. 2, §3.1 · p. 3
3.2.1 Authoritative Error Signals from DQC Messages.
- Researchers collected 4,545 DQC error messages for 372 companies, selected nine error types covering 60.33% of cases, and retrieved 218 SEC filings.The messages were collected from 2020 to 2024 and provide deterministic rule labels for semantic, relational, or numerical inconsistencies.Source: §3.2.1 · p. 4
3.2.2 XBRL Filing Segmentation.
- Statement-level segmentation uses schema roleType identifiers and presentation linkbases to create manageable sub-filings while preserving dependencies among XBRL components.Each filing combines instance values, schema element types, and linkbases specifying hierarchy, arithmetic relationships, and semantic metadata.Source: §3.2.2 · p. 4
3.2.3 US-GAAP Taxonomy Digitalization.
- The US-GAAP taxonomy is organized into concept-centered core and relation chunks that retain concept attributes and structural or semantic links for indexing and reasoning.Relation chunks encode presentation paths, calculation parent-child relationships, definitional arcs, and external references.Source: §3.2.3 · p. 4
3.3 Data Annotation and Quality Control
- The benchmark contains 1,102 annotated instances averaging over 33k tokens, capturing the long-context characteristics of practical financial auditing.The reported resource statistics are measured in tokens via cl100k_base.Source: §3.3 · p. 5, Table 4 · p. 5
3.3.1 Annotation via LLM-Assisted Parsing.
- GPT-4o-mini parses DQC messages into labels, task elements, and evidence pointers; FinSM, FinRE, and FinMR instances use those signals to define answers and filing inputs.FinRE maps DQC rule IDs to relation labels and represents participating elements as (head, relation, tail) triples; FinMR uses reported and calculation-implied values.Source: §3.3.1 · p. 5
3.3.2 Quality Control.
- Quality control manually checked 50 filings, double-validated 50% of instances against human review, corrected discrepancies, and verified taxonomy relation references.Structural checks assessed roleType matching and preservation of reporting hierarchy.Source: §3.3.2 · p. 5
3.4 Evaluated LLMs
- The evaluation includes 13 LLMs—one closed-source general model, ten open-source general models, and two open-source financial models—with inputs capped at 81,920 tokens.Evaluations use the LM Evaluation Harness; open-source models run locally on a 4×H200 GPU cluster, with generation limited to 512 tokens.Source: §3.4 · p. 5
4 Benchmarking Results and Empirical Analysis
Across FinSM, FinRE, and FinMR, leading models still struggle with taxonomy-constrained concept retrieval, structural relationship reasoning, and numerical verification. The analyses identify difficulty both in validating accounting structures and in carrying out or expressing required calculations.
4.1 FinSM
- Hit Rate@20 stays below 13% even for state-of-the-art general-purpose LLMs, with no consistent advantage from larger models or financial pretraining.Errors include missing inappropriate concepts and overlooking when reporting requires granular components rather than aggregate tags.Source: §4.1 · p. 5
4.2 FinRE
- GPT-4o has the strongest and most stable FinRE performance, with high accuracy and balanced macro precision, recall, and F1; DeepSeek-V3 is the strongest open-source model but trails noticeably.FinRE is harder than standard relation classification because it requires hierarchical, compositional, and cross-document constraints.Source: §4.2 · pp. 5–6, Table 6 · p. 6
- Most models reach near-zero F1 on CombinationErr, the hardest FinRE relation type, which requires validating axis–member combinations across interrelated linkbases.Top models handle Reversal and Inappropriateness more reasonably; even competitive models drop sharply on CombinationErr.Source: §4.2 · p. 6, Figure 3 · p. 6
4.3 FinMR
- Fin-o1-14B achieves the best FinMR accuracy at 13.86%, while overall zero-shot accuracy remains low across representative models.The task jointly tests hierarchical calculation structures, cross-fact dependencies, and arithmetic constraints in XBRL filings.Source: §4.3 · p. 6, Figure 4 · p. 6
- Calculation errors account for 70–83% of failures across most models, while extraction errors are approximately 10–27%.Models often locate candidate facts but fail to execute or validate multi-step arithmetic relations specified by taxonomy-grounded calculation rules.Source: §4.3 · p. 6, Figure 4 · p. 6
- Fin-o1-14B combines a low CER of 9% with a high SER of 71%, showing that strong numerical reasoning can coexist with frequent output-format failures.Qwen3-32B and FinR1 also exhibit extremely high structural-error values.Source: §4.3 · p. 6, Figure 4 · p. 6
5 Conclusion
The conclusion presents FINAUDITING as a benchmark for structured, hierarchical, taxonomy-driven financial reasoning across multi-document filings. Its evaluation finds that leading models still struggle across these demands.
5 Conclusion
- The zero-shot evaluation finds that leading LLMs still struggle with cross-document reasoning, structured interpretation, and taxonomy alignment.The paper emphasizes stronger structural grounding and explicit reasoning awareness as priorities for future models.Source: §5 · p. 7
A LLM Judgment for Data Annotation
The appendix gives task-specific instructions for using LLMs to annotate DQC validation messages. The prompts specify which financial concepts, relations, values, and explanations to extract and how to structure the outputs.
A.1 Prompt for the FinSM task
- FinSM prompts extract erroneous concepts and dimensions from DQC messages, with separate formats and criteria for tag, statement, and dimension information.The instructions distinguish explicitly erroneous tags from valid references and request contextual reasons for extracted dimensions or tags.Source: §A.1 · p. 8
A.2 Prompt for the FinRE task
- FinRE prompts identify erroneous tag relationships, including inappropriate axis–member pairs and head–tail presentation relations, with brief explanations.The instructions require that a pair have a stated structural relationship and that the message indicate inconsistency or potential misclassification.Source: §A.2 · p. 8
A.3 prompt for the FinMR task
- FinMR prompts extract statement, concept, period, reported and correct values, and error reasons for calculation or sign violations.For total-element errors, the expected value is the sum of child components defined by the calculation linkbase; for sign errors, the correct value is the reported value without its negative sign.Source: §A.3 · p. 9
B US-GAAP Taxonomy Chunking
The taxonomy chunking process converts official US-GAAP spreadsheets into retrievable concept-centered units. It separates intrinsic concept attributes from relation information while retaining taxonomy structure and provenance.
B US-GAAP Taxonomy Chunking
- Each GAAP concept is represented by a core chunk of intrinsic attributes and relation chunks encoding semantic and structural links.The resulting units are intended to support structured retrieval and reasoning while preserving hierarchical and relational organization.Source: B US-GAAP Taxonomy Chunking · pp. 9–10
- The pipeline normalizes workbook sheets and headers, identifies relevant taxonomy sheets, and records file, sheet, and row provenance for extracted elements.It ignores suspicious all-unnamed reference sheets and infers the taxonomy version from the filename.Source: B US-GAAP Taxonomy Chunking · pp. 9–10
- Core chunks extract each concept’s identifier, label, type, balance, period type, abstract flag, documentation, and deprecation status into a canonical text form.Each core chunk receives a stable concept::core identifier and is designed as a self-contained summary for retrieval or embedding.Source: B US-GAAP Taxonomy Chunking · pp. 9–10
- Relation chunks cover presentation, calculation, definition, reference, and enumeration links, while accommodating heterogeneous layouts and missing columns.Chunks record roles, arcroles, and provenance; integrity checks verify that relation targets refer to existing core concepts, and malformed rows are skipped.Source: B US-GAAP Taxonomy Chunking · pp. 9–10
C.1 The metrics for FinSM
FinSM retrieval is evaluated at several top-k cutoffs using hit rate, recall, and query-level Macro-F1. The metrics capture whether relevant concepts are found, how completely they are retrieved, and the balance of precision and recall.
C.1 The metrics for FinSM
- Hit Rate@k measures the proportion of queries with at least one relevant element among the top-k results.It does not account for how many relevant elements are retrieved.Source: §C.1 · p. 10
- Recall@k measures the fraction of all relevant elements retrieved within the top-k predictions.The metric reflects how completely the relevant set is covered under the cutoff.Source: §C.1 · p. 10
- Macro-F1@k averages query-level F1 scores computed from top-k predictions, balancing precision and recall across queries.Each query receives equal weight in the average.Source: §C.1 · p. 10
- The retrieval metrics are evaluated at k = 1, 5, 10, and 20.Source: §C.1 · p. 10
C.2 The metrics for FinRE
FinRE is evaluated with accuracy and class-averaged precision, recall, and F1 across three relation categories. These measures capture overall correctness and per-class identification performance.
C.2 The metrics for FinRE
- Accuracy measures the overall proportion of correctly classified relations among all predictions.Source: §C.2 · p. 11
- Macro Precision averages the proportion of correctly predicted relations within each of the three relation classes.Source: §C.2 · p. 11
- Macro Recall averages the proportion of true relations identified for each of the three classes.Source: §C.2 · p. 11
- Macro F1 averages the per-class harmonic mean of precision and recall across the three relation categories.Source: §C.2 · p. 11
C.3 The metrics for FinMR
FinMR evaluates numerical-answer correctness with an LLM-as-a-judge framework and separates structural, extraction, and calculation errors. Its checks proceed from answer format to extracted value and then calculated value.
C.3 The metrics for FinMR
- FinMR uses an LLM-as-a-judge framework to assess whether generated answers give the correct numerical result under the auditing rule.Accuracy is reported alongside Structural Error Rate, Extraction Error Rate, and Calculation Error Rate.Source: §C.3 · p. 11
- Structural errors reflect invalid JSON structure, extraction errors reflect an incorrect reported value, and calculation errors reflect an incorrect computed value after the first two checks pass.Together, the indicators distinguish formatting, value-identification, and arithmetic failures.Source: §C.3 · p. 11
- The judge first verifies that pred_answer has the same structure as true_answer, requiring a JSON object with exactly two keys; if invalid, it outputs label S; otherwise, it compares extracted_value fields.Source: §C.3 · p. 11
- After structure and extraction pass, the judge requires the predicted calculated value to match the ground truth exactly, with zero tolerance.The judge outputs one label: S, E, C, or A.Source: §C.3 · p. 11
D Evaluation Models
The evaluation spans closed- and open-source, general-purpose and finance-specific models. The paper also outlines how FinAuditing’s tasks could fit into existing auditing workflows and scale through taxonomy substitution or tiered deployment.
D Evaluation Models
- The model evaluation spans closed- and open-source systems, general-purpose and financial-specific models, and varied architectures and scales.GPT-4o serves as a proprietary upper-bound reference; the set also includes open-source and finance-specialized models.Source: D Evaluation Models · pp. 11–12, Table 7 · p. 12
E Practical Deployment and Scalability Considerations
- In a proposed workflow, FinAuditing operates downstream of deterministic validators on suspicious items, with FinSM supporting taxonomy disambiguation, FinRE categorizing errors, and FinMR verifying targeted numerical cases.The workflow retains expert review for high-stakes decisions.Source: E Practical Deployment and Scalability… · p. 12
- Although instantiated with US-GAAP, the benchmark can be adapted to other regulatory environments by substituting taxonomies and rule sources while preserving task interfaces.Source: E Practical Deployment and Scalability… · p. 12
- The proposed tiered deployment uses lightweight retrieval or classifiers for FinSM and FinRE, reserving larger models or human review for selected high-risk FinMR cases.The paper presents this as a way to balance performance and computational cost.Source: E Practical Deployment and Scalability… · p. 12