Source-linked AI summary
AUDITFLOW: Executable Symbolic Environments for Structured Financial Reporting Verification
Yan Wang, Xuguang Ai, Jaisal Patel, Xueqing Peng, Fengran Mo, Yupeng Cao, Haohang Li, Mingyu Cao, Lingfei Qian, Víctor Gutiérrez-Basulto
TL;DR
Language-model agents struggle with financial audit verification when correctness depends on structured evidence and numerical relationships rather than text alone. AuditFlow separates adaptive search from deterministic verification in a graph-grounded multi-agent framework, achieving 82.09% joint audit accuracy and outperforming the strongest baseline by 14.93 points.
Problem
Financial audit verification requires linking dispersed facts, taxonomy relations, recomputed values, and audit rules across structured evidence rather than text alone.
Method
AuditFlow combines dual taxonomy and filing graphs with typed deterministic tools and a three-agent protocol that separates evidence search from computation.
Results
82.09% joint audit accuracy under GPT-5.5 exceeded the strongest baseline by 14.93 points, while removing deterministic checks reduced accuracy to 17.91%.
Takeaways & Limitations
The results support building reliable audit agents around executable structured environments, with LLMs guiding search and deterministic operations determining verification outcomes.
Takeaways & Limitations
Evaluation covered only 67 FinMR instances and three DQC rule families, not the broader range of XBRL audit rules, filings, companies, or years.
Abstract
from arXiv · showhide
Structured financial audit verification is difficult for language-model agents because correctness depends on structured evidence rather than text alone. A model must link reported facts to taxonomy concepts, traverse calculation or dimensional relations, and recompute expected values before applying an audit rule. We propose AuditFlow, a graph-grounded multi-agent framework that separates adaptive search from deterministic verification. AuditFlow builds a symbolic environment from a static US-GAAP taxonomy graph and a dynamic XBRL filing graph, and exposes it through typed tools for fact retrieval, taxonomy traversal, numerical checking, and rule evaluation. Two junior auditors inspect each case from regulatory and evidentiary views, while a senior auditor resolves disagreements and can request further investigation. The final reports are fused through evidential aggregation to produce an audit verdict, expected value, evidence trail, and trustworthiness score. On a FinAuditing-derived FinMR sample, AuditFlow reaches 82.09% joint audit accuracy under GPT-5.5, outperforming the strongest baseline by 14.93 points. Removing deterministic checks drops accuracy to 17.91%, showing that the symbolic environment performs the verification step that the model cannot reliably replace.
1 Introduction
XBRL audit verification requires structured operations over filing facts and taxonomy relations, not text retrieval alone. AUDITFLOW separates adaptive search from deterministic computation through a dual-graph symbolic environment, achieving 82.09% joint audit accuracy under GPT-5.5 while showing that deterministic checks are essential.
- Motivation: XBRL verification links reported facts to accounting concepts, periods, units, and contexts, then checks their consistency with taxonomy constraints and related facts.Correctness depends on interactions among structured filing evidence rather than a single sentence.
- Verification challenge: A single audit rule may require linking dispersed documents, traversing taxonomy relations, recomputing expected values, and applying rules over structured evidence.These operations define the symbolic environment used for verification.
- AUDITFLOW: AUDITFLOW separates search from computation by letting LLM agents guide retrieval while typed deterministic tools perform fact retrieval, taxonomy traversal, numerical checks, and rule evaluation.Its environment connects a static US-GAAP taxonomy graph with a dynamic filing evidence graph, with junior and senior auditors handling complementary inspection and disagreement resolution.
- Evaluation: 82.09% joint audit accuracy under GPT-5.5 surpassed the strongest baseline, Single Agent, by 14.93 points.On ablation, removing deterministic checks reduced accuracy to 17.91% and increased invalid outputs to 35.82%.
2 Related Work
Prior work improves LLM agents through tools, retrieval, graphs, grounding, and multi-agent interaction, including applications to financial auditing. AuditFlow differs by making verification steps executable within a structured environment.
- Agents, grounding, and verification: LLM agents use tools, retrieval, graphs, and multi-agent interaction to improve beyond one-step generation.ReAct, Toolformer, and Gorilla study tool use, while graph-based retrieval and neuro-symbolic methods ground or validate reasoning with external structure.
- Agents, grounding, and verification: AuditFlow instead makes these verification steps executable in the structured environment.
- LLM agents for financial auditing: Financial LLM systems use retrieval, tools, graphs, and agents to analyze filings and disclosures.Related audit studies use knowledge graphs and graph neural networks to model accounting structure and detect suspicious entries.
3 Methodology
AUDITFLOW separates adaptive search from deterministic verification through a dual-graph environment, typed tools, and a three-agent audit protocol. The system produces inspectable, evidence-backed verdicts by combining structured observations, rule checks, and evidential aggregation.
- Architecture: The four-part architecture combines dual graphs, deterministic typed tools, two junior auditors with complementary views, and senior review.The static graph represents US-GAAP taxonomy constraints, while the dynamic graph represents XBRL filing evidence.
- Dual-graph environment: The taxonomy graph encodes concepts and presentation, calculation, and dimensional relations, while the filing graph stores facts, contexts, periods, units, and dimensional assignments.Bridge edges connect reported facts to governing taxonomy concepts for structured verification.
- Executable tools: Typed tools let agents retrieve evidence, traverse relations, perform forensic comparisons, and execute deterministic sign, calculation-tree, and dimensional checks.Checker outputs are treated as authoritative for final verification rather than being overridden by LLM self-reflection.
- Three-agent protocol: The Compliance Auditor follows a specification-first path, whereas the Forensic Auditor follows an evidence-first path over the same environment.A required-tool gate prevents either junior report from being finalized before mandatory checks return non-error observations.
- Three-agent protocol: The Senior Auditor accepts agreement or sends targeted feedback for further investigation when junior reports conflict, after which evidential aggregation combines the final reports.The output includes the decision, expected value, action path, supporting evidence, and trustworthiness score, making the verdict inspectable.
4 Experimental Setup
AuditFlow is evaluated on a 67-instance FinMR subset using independently retrieved XBRL filings and taxonomy releases. Experiments span multiple model families, baselines, audit-artifact metrics, and a controlled executable pipeline.
- Dataset: 67 instances from FinAuditing’s FinMR task provide tickers, audit queries, target concepts, reporting periods, and ground-truth answers.The benchmark labels remain aligned with FinAuditing while cases are verified from raw structured filings and taxonomy evidence.
- Backbones: The evaluation uses proprietary and open-weight backbones spanning GPT-5.5, GPT-4o, Claude Sonnet 4.6, Qwen3.5-397B-A17B, Qwen3.6-27B, and Fino1-14B.This tests generalization across model families and scales rather than dependence on one proprietary model.
- Baselines: Seven baselines test manually curated evidence, benchmark workflows, full-context prompting, retrieval, structured context, and single-agent tool use against AuditFlow.FinAuditing and Herculean use released benchmark frameworks, while Direct LLM uses complete XBRL filings and another baseline replaces the three-agent protocol with one ReAct-style agent.
- Metrics: Joint audit accuracy requires correct verdict, reported value, and applicable expected value; SER, EER, CER, and VAcc provide complementary evaluation measures.The metrics separate structural, evidence-extraction, computation, and verdict-only performance.
- Implementation: AuditFlow runs as an executable Python pipeline with shared backbones across auditor roles, temperature-0 greedy decoding, and bounded token and ReAct-step budgets.Junior auditors receive 4096-token budgets and 20 tool steps, while the senior auditor receives 8192 tokens to read both junior reports.
5 Results
AUDITFLOW consistently achieves the strongest joint audit accuracy across backbones, with deterministic verification providing the largest performance gain. Results also show that role-specialized search, evidential aggregation, and backbone-dependent tool-use behavior affect audit-artifact completeness and stability.
- 82.09% Joint ACC on GPT-5.5 is AUDITFLOW’s best result, and it remains strongest across all six backbones.AUDITFLOW also achieves 80.60% on GPT-4o, Claude Sonnet 4.6, and Qwen-397B; 73.13% on Qwen-27B; and 31.34% on Fino1-14B.
- 94%+ VAcc alongside roughly 80% Joint ACC shows that binary verdict accuracy exceeds full audit-artifact correctness.Joint ACC additionally requires the reported value and expected value to be correct.
- 73.13% Joint ACC and 85.07% VAcc for Qwen-27B contrast with 31.34% Joint ACC and 59.70% VAcc for Fino1-14B.The larger Fino1-14B gap indicates plausible binary decisions often fail to become complete audit artifacts; expected-value reconstruction remains the main residual difficulty.
- 17.91% Joint ACC after removing deterministic checks, compared with 82.09% originally, demonstrates that rule-specific checkers perform verification models cannot reliably replace.Invalid outputs rise from 1.49% to 35.82%, while accuracy on DQC.US.0117 and DQC.US.0126 falls to 0.00%.
- 74.63% Joint ACC without the required-tool gate and 77.61% without ER fusion show that both components improve output stability.Without the gate, invalid outputs rise to 17.91%, suggesting agents may finalize before gathering sufficient evidence.
- A1 exclusively uses static taxonomy tools, whereas A2 mixes static and dynamic tools and shifts from dynamic evidence gathering toward taxonomy tools and deterministic checks.GPT-5.5 is the most exploratory; Qwen models use fewer tools, and Fino1-14B produces no well-formed tool calls. On Qwen-27B, disagreement lowers final accuracy from 80.4% to 36.4%.
6 Conclusion
AUDITFLOW is a graph-grounded multi-agent framework for XBRL audit verification that separates evidence exploration from deterministic computation. Its results support building reliable audit agents around executable structured environments rather than text-only reasoning.
- AUDITFLOW combines LLM-guided evidence exploration with symbolic fact retrieval, taxonomy traversal, numerical checking, and rule evaluation.The framework separates search from computation by assigning evidence exploration to LLM agents and verification operations to a symbolic environment.
- Experiments on a FinAuditing-derived FinMR sample show that AUDITFLOW outperforms LLM-only, retrieval-based, and agentic baselines.
- The conclusion argues that reliable audit agents should use executable structured environments rather than text-only reasoning.
Limitations
AuditFlow is evaluated on a narrow sample and rule scope, assumes reliable structured XBRL and US-GAAP graphs, and remains dependent on LLM protocol adherence and judge-based evaluation. These limitations constrain coverage across filings and rules and motivate broader, more robust validation.
- Evaluation scope: The evaluation covers only 67 FinMR instances and three DQC rule families, not the full range of XBRL rules, filings, companies, or reporting years.Future work should test larger datasets and broader rule sets.
- Structured-data assumptions: AuditFlow assumes XBRL filings and US-GAAP taxonomy releases can be parsed into reliable structured graphs, limiting direct handling of scanned, incompletely tagged, or less structured disclosures.Supporting those settings requires additional extraction and normalization steps.
- Model and evaluation dependence: Despite deterministic verification, AuditFlow depends on the LLM backbone to follow the tool protocol and produce usable reports, while LLM-as-a-judge evaluation may introduce judge errors.Weaker models can fail before reaching deterministic checks, and the evaluation follows the FinAuditing LLM-as-a-judge protocol.
Ethical Considerations
AuditFlow is a research system for XBRL audit verification, not a replacement for professional audit judgment. Its deployment requires evidence-based safeguards, additional validation, human review, and integration with existing audit and compliance procedures.
- System role and risks: AuditFlow is not intended to replace professional audit judgment, and incorrect outputs could create risks in financial-filing verification.False negatives may miss inconsistencies, while false positives may raise unnecessary concern about valid filings.
- Risk mitigation: Evidence trails, deterministic rule checks, and explicit reported and expected values are required to reduce risks from incorrect audit outputs.
- Deployment conditions: Real-world use requires additional validation, human review, and integration with existing audit and compliance procedures.The system uses public-company filing data and FinAuditing-derived benchmark labels, but does not trade, execute financial actions, or make regulatory decisions.
Artifact Use and Licensing
The study evaluates AuditFlow using public SEC filings, FinAuditing metadata, and US-GAAP taxonomy releases under applicable source terms. The released materials exclude raw filing packages and proprietary model outputs, while experimental fields contain no private personal information.
- Artifact Use: Experiments use SEC EDGAR filings, FinAuditing benchmark metadata, and public US-GAAP taxonomy releases for research evaluation.Original sources are cited for these resources.
- Licensing: Released code or derived data will reference original resources and follow their applicable terms of use.Raw SEC filing packages and proprietary model outputs are not redistributed.
- Data Content: Experimental fields include identifiers, filing metadata, audit queries, taxonomy concepts, reporting periods, and numerical ground-truth answers, but no private personal information.Publicly disclosed officer or signatory names may appear in SEC filings but are not prediction targets or evaluation labels.
Use of AI Assistants · A Tool Access and Tool Inventory
The authors used AI assistants for limited writing support while retaining author verification of all substantive work. AUDITFLOW’s tool environment assigns role-specific access, enforces deterministic checks, and connects taxonomy and filing evidence through structured verification outputs.
- Use of AI Assistants: AI assistants supported language polishing, brainstorming, and LaTeX editing, while authors reviewed and verified all technical claims, results, analysis, and final writing decisions.
- A Tool Access and Tool Inventory: AUDITFLOW documents agent-specific tool access and a complete inventory through a typed action-observation layer.
- A Tool Access and Tool Inventory: Compliance Auditor A1 receives static taxonomy tools for metadata, calculation trees, dimensional axes, DQC rules, and deterministic checking.
- A Tool Access and Tool Inventory: Forensic Auditor A2 receives static and dynamic tools to inspect filing-specific history, dimensions, related concepts, arithmetic, units, and magnitude patterns.
- A Tool Access and Tool Inventory: The Senior Auditor As reads junior reports without production tool access; a ReAct senior with the A2 catalog is used only for ablation.
- A Tool Access and Tool Inventory: Junior auditors may choose search trajectories but must complete required deterministic checks before finalizing or requesting arbitration.A1 must use check_sign, check_calc_tree, and check_dim_consistency; A2 additionally must use get_fact_history.
- A Tool Access and Tool Inventory: Static tools expose taxonomy constraints, dynamic tools recover filing evidence, and deterministic checkers apply DQC rules across both graph views.The inventory covers taxonomy, rule-specific checking, and filing-specific evidence tools.
- A Tool Access and Tool Inventory: The three check_-prefixed tools are terminal verification tools whose structured outcomes provide authoritative audit values, including reported and expected values, supporting facts, and rule findings.
B Dataset Details … F.3 Inter-Agent Disagreement
The evaluation uses a 67-instance FinMR subset with structured XBRL and US-GAAP evidence, tests multiple model families, and measures both audit correctness and grounded interaction. Across backbones, AUDITFLOW leads Joint ACC, while junior-auditor disagreement marks substantially harder cases.
- B Dataset Details: 67 FinMR instances cover three DQC rule families, annual and quarterly filings, and 47 unique tickers.The subset contains 22 DQC.US.0015, 24 DQC.US.0117, and 21 DQC.US.0126 cases, including 29 10-K and 38 10-Q filings.
- C Model Details: AUDITFLOW tests proprietary, open-weight, and hosted backbone models rather than relying on a single model family.Proprietary parameter scales are undisclosed, while released scales are reported for open-weight models.
- D Evaluation Metric Details: The evaluation separates Joint ACC, verdict-only accuracy, diagnostic error rates, grounded interaction metrics, and inter-agent conflict mass K.Joint ACC requires the full audit artifact to be correct; SER, EER, and CER distinguish structural, extraction, and calculation failures.
- E Implementation Details: Junior auditors may take at most 20 ReAct steps, the senior-feedback loop is capped at 3 rounds, and main evaluations run asynchronously with up to 8 concurrent cases.The production senior is a single-shot arbitrator; a 10-step ReAct senior is implemented only for ablation.
- F Full Experimental Results: The appendix reports complete method-by-backbone Joint ACC results and corresponding diagnostic error rates for all baselines and backbones.These results provide the full Joint ACC matrix and diagnostic error profiles beyond the primary comparison.
- F.1 Full Joint ACC: 82.09% Joint ACC under GPT-5.5 makes AUDITFLOW the strongest method, ahead of Single Agent at 67.16% and Herculean at 65.67%.AUDITFLOW achieves the best result under every backbone, across proprietary and open-weight models.
- F.2 Full Diagnostic Error Rates: AUDITFLOW’s remaining errors are mainly calculation errors, while EER stays near zero across most backbones.Retrieval-based methods reduce some structural failures but retain substantial calculation errors; Single Agent has higher CER than AUDITFLOW under the strongest backbone.
- F.3 Inter-Agent Disagreement: 80.4% final ACC when juniors agree falls to 36.4% when they disagree, while mean conflict mass K rises to 0.330.On Qwen3.6-27B, disagreement identifies cases where evidence is harder to reconcile before final aggregation.
G Bootstrap Analysis for Backbone Comparisons
Case-level paired bootstrap analysis tests whether backbone differences in Joint ACC are robust over the 67-case evaluation set. GPT-5.5 is only modestly above GPT-4o and not statistically distinguishable, whereas Qwen3.6-27B is significantly lower, preserving a clear performance tiering.
- Bootstrap methodology: B = 104 bootstrap resamples recompute Joint ACC and paired comparisons report ΔACC, 95% confidence intervals, and two-sided recentred p-values.The analysis resamples 67 cases with replacement and paired correctness indicators.
- Per-backbone results: 82.09% Joint ACC for GPT-5.5 leads GPT-4o, Claude-Sonnet-4.6, and Qwen3.5-397B-A17B at 80.60%.GPT-5.5 records 55/67 correct with a 95% CI of [75.8, 90.2], while each 80.60% backbone records 54/67 with [70.1, 89.6].
- Paired comparisons: +7.5 points, p = 0.032 for GPT-4o versus Qwen3.6-27B is significant, unlike GPT-5.5’s 1.5-point gap with GPT-4o.The GPT-5.5 comparison has a paired interval of [−8.8, +5.8], which includes zero; identical case-level correctness yields Δ = 0 and p = 1 for three other backbones versus GPT-4o.