Source-linked AI summary
Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks
Antoine Bigeard, Langston Nashold, Rayan Krishnan, Shirley Wu
TL;DR
Finance Agent Benchmark addresses limited evidence about how reliably LLM agents perform real-world financial research and analysis. The paper constructs and validates an expert-authored benchmark and evaluates tool-equipped agents, finding that current systems remain unreliable for autonomous high-stakes use despite substantial efficiency advantages.
Problem
Existing benchmarks provide limited evidence about interactive, fresh, tool-augmented financial analysis, where plausible misinformation could affect high-stakes decisions.
Method
The paper builds a nine-category benchmark of 537 expert-authored questions validated through review and evaluates LLMs with tools including Google Search and EDGAR access.
Results
Across tested systems, performance ranged from below 3% to 46.8% accuracy, with the best-performing model achieving 46.8%.
Takeaways & Limitations
Models completed tasks faster than human experts, suggesting potential productivity benefits for supporting routine financial work while autonomous high-stakes deployment still requires advancement.
Takeaways & Limitations
Human-comparison calculations include only direct question-answering time and exclude question creation and answer review.
Abstract
from arXiv · showhide
Artificial Intelligence (AI) technology has emerged as a transformative force in financial analysis and the finance industry, though significant questions remain about the full capabilities of Large Language Model (LLM) agents in this domain. We present the Finance Agent Benchmark, featuring challenging and diverse real-world finance research problems that require LLMs to perform complex analysis using recent SEC filings. We construct the benchmark using a taxonomy of nine financial task categories, developed in consultation with experts from banks, hedge funds, and private equity firms. The dataset includes 537 expert-authored questions covering tasks from information retrieval to complex financial modeling, each validated through a rigorous review process to ensure accuracy and relevance. Moreover, we implement an agentic harness that equips LLMs with tools sufficient to produce accurate responses, including Google Search and EDGAR database access. Overall, the Finance Agent Benchmark provides a comprehensive testbed for measuring the progress of LLM-driven finance agents. Our evaluation reveals significant limitations in current AI capabilities - even the best-performing model (OpenAI o3) achieved only 46.8% accuracy at an average cost of $3.79 per query. This underscores the need for further advancements before reliable deployment in high-stakes finance settings.
1 Introduction
Finance Agent Benchmark targets the gap between promising finance automation and reliable, domain-specific evaluation. It combines expert-designed real-world tasks with tool-equipped agents and finds current models remain below autonomous-deployment reliability.
- Up to 40% of entry-level finance professionals’ workweek can be spent gathering data rather than analyzing it.
- Previous benchmarks often omit interactive environments, nuanced reasoning, fresh questions, and in-depth financial analysis grounded in live information.
- Finance Agent Benchmark is a standardized framework for rigorously evaluating AI agents on real-world financial analysis tasks.
- Seven financial-industry experts contributed realistic queries spanning common analytical tasks involving public financial documents.
- Questions include expert answers and reasoning trajectories, are verifiable through SEC EDGAR filings, and undergo peer review.
- 3.1 minutes and $3.78 were o3’s average task time and cost, versus 16.8 minutes and $25.66 for human experts.The authors frame this efficiency-to-performance ratio as promising for supporting human analysts while accuracy improves.
2 Finance Agent Benchmark: High-quality Financial Benchmark
The benchmark is built from expert-defined finance tasks and validated annotations, then evaluated with structured rubrics and contradiction checks. Its 537 questions are split across public, private, and fully private evaluation sets.
- Experts from banks, hedge funds, and startups developed a taxonomy spanning finance tasks from basic retrieval to complex market analysis.
- 537 questions cover nine task categories and require multiple financial documents, with ground-truth answers, source documents, and step-by-step solutions.Questions focus on documents published no earlier than 2024 and are answerable using public filings or the open internet.
- Each question underwent peer review, correction or removal when errors were found, and a final author review for consistency.
- Rubric-based evaluation checks distinct key points from expert answers rather than judging responses holistically.This structure is intended to cover critical aspects while maintaining factual accuracy and completeness.
- Reported human-comparison calculations include only direct question-answering time, excluding question creation and answer review.
- A dedicated contradiction rubric checks whether generated answers conflict with expert answers.The approach emphasizes detecting contradictions because this is described as more reliable than confirming complete agreement.
- The dataset contains 50 public validation samples, 150 private validation samples, and 337 fully private test samples.Reported metrics use all 537 samples; the private test set is kept from future overfitting and contamination.
3 Financial Agent Harness
The Financial Agent Harness evaluates LLMs in a tool-augmented environment for researching financial information. It provides specialized retrieval tools, context management, execution feedback, and error handling.
- The harness uses a ReAct-informed agentic infrastructure that gives models tools sufficient to produce accurate responses.
- Specialized tools provide general web search, EDGAR filing search, HTML parsing, and retrieval from stored document content.
- ParseHTML and RetrieveInformation jointly let models manage their own context window.
- 3.2 Environment Feedback: After each tool call, the model receives success or failure feedback and the successful result.
- 3.2 Environment Feedback: Rate-limit failures are retried with exponential backoff, while token-limit or incorrect-argument failures are returned as model-generated agent errors.
- The full evaluation harness has been released on GitHub.
4 Experiments and Results
Experiments evaluate models under consistent settings using class-balanced and naive accuracy, then analyze cost-accuracy trade-offs and tool-use behavior. Results link stronger performance to deeper, more persistent, and better-balanced exploration, while tool misuse produces errors.
- Experimental Setup: Models were evaluated with identical prompts, controlled temperature where supported, high token limits, and a shared compute environment.The default maximum token limit was 16,384 tokens, and experiments ran on an AWS t2.2xlarge instance.
- Evaluation Metrics: Class-balanced accuracy averages per-category scores equally, whereas naive accuracy measures the overall percentage of correctly answered questions.Class-balanced accuracy is used as the representative measure of general agentic financial capabilities because retrieval categories are overrepresented.
- Accuracy-Cost Results: The accuracy-cost Pareto curve includes o3, o4-mini, and GPT 4.1 mini above 20% accuracy, with a logarithmic cost-accuracy trend and notable outliers.OpenAI’s o3-mini and Cohere Command A fall significantly below the curve.
- Tool Calling Analysis: Claude 3.7 Sonnet and o3 support the pattern that more extensive tool exploration accompanies stronger performance, while GPT-4o Mini combines many calls with the highest error rate.GPT-4o Mini often repeats failed calls without changing strategy, whereas Claude 3.7 Sonnet maintains a low error rate while iterating across tools.
- Tool Calling Analysis: Higher-performing models use retrieval, search, and parsing tools more evenly, while LLaMA 4 Maverick disproportionately retrieves and hallucinates nonexistent documents.The benchmark distinguishes extensive exploration from precise tool selection: high call volume alone does not ensure effective tool use.
- Case Study Examples: Claude 3.7 Sonnet verifies iteratively, Gemini 2.5 Pro Preview uses a minimal sufficient sequence, and LLaMA 4 Maverick fails to parse retrieved results.Within the same 200-second timeframe, Claude 3.7 makes more calls than o1, while o1 fails despite using the right tool sequence.
5 Related Work
Prior agent benchmarks cover interactive and tool-augmented tasks but rarely test fresh, time-sensitive financial analysis with live evidence. FinanceAgent Benchmark addresses this gap through expert-authored questions, live EDGAR retrieval, and evidence-grounded evaluation.
- General Autonomous-Agent Evaluation: General-purpose benchmarks evaluate interactive simulations, web navigation, games, tool use, browsing, and real-world coding tasks.Examples include AgentBench, GTA, SWE-Lancer, GAIA, and WebVoyager.
- Benchmark Gap: Existing benchmarks rarely use fresh, time-sensitive questions, making retrieval during execution difficult to distinguish from memorized knowledge.They also lack benchmarks targeting in-depth financial analysis requiring real-time, tool-augmented reasoning.
- Finance-Based NLP and QA Datasets: Earlier finance QA datasets such as FinQA and TAT-QA support numerical and hybrid reasoning over static financial reports but not autonomous planning.Recent efforts add verification or noisy EDGAR queries, yet still assume static inputs or focus outside finance.
- FinanceAgent Benchmark: FinanceAgent Benchmark evaluates autonomous agents on 537 expert-authored, validated questions across nine categories using live SEC filings and tools including EDGAR search, calculators, and web access.The benchmark emphasizes multi-step reasoning and evidence-backed answers in realistic financial workflows.
6 Conclusion
The benchmark finds large capability gaps across current financial-analysis agents, alongside substantial speed and cost advantages over human experts. It therefore supports progress tracking while indicating that autonomous high-stakes deployment still requires major improvement.
- Conclusion: Performance ranges from below 3% to 46.8% accuracy, with the best-performing model reaching only 46.8%.The benchmark reports significant limitations across all tested systems and a substantial gap between AI and human expert performance.
- Conclusion: Models complete tasks significantly faster than human experts, while o3, Claude 3.7 Sonnet, and o4 Mini define the efficiency frontier.Performance follows a clear logarithmic cost-accuracy relationship.
- Future Work: The benchmark provides a measure for tracking progress toward real-world financial analysis requirements and motivates future study of structured tabular data and more complex agents.The authors state that experiments can be extended through deeper investigation of these settings.
A Dataset Creation
The dataset is built from expert-designed, multi-document financial questions and converted into structured, contradiction-aware evaluation checks. Its questions, code, documentation, and validation resources are released for reproducible use.
- Rubric Construction: GPT-4o was used to generate rubrics from each question, reasoning trace, and answer before final review.The rubric-generation prompt converts expert answers into structured evaluation checks.
- Rubric Construction: Evaluation checks use the EDGAR research operator to verify complete factual concepts such as figures, names, dates, and relationships.Checks are intended to be specific, measurable, objective, and independent of answer style.
- Rubric Construction: The rubric instructs evaluators to assess answers rather than the contextual question or reasoning and to avoid splitting related facts into unnecessary checks.Distinct separable components receive separate checks, while logical connections remain together.
- Illustrative Example: The example decomposes a CVNA answer into five checks covering two values, a CAGR, and two stated drivers of change.The checks separately capture gross profit per unit in 2019 and 2024, the 20.3% CAGR, operating and technology efficiencies, and origination fees.
- Quality Control: A contradiction check compares generated answers against the full expert answer to detect conflicting information.The methodology treats contradiction detection as more reliable than confirming complete agreement on every point.
- Release and Reproducibility: The dataset, agent code, simulation environment, evaluation scripts, and documentation are released under permissive licenses and hosted through public repositories and archives.The dataset uses CC BY 4.0, the code uses the MIT License, and resources are available through GitHub, Zenodo, and HuggingFace.
B.2 Dataset Split Correlation Plots
The benchmark preserves a consistent distribution of question categories across training, validation, and test sets. Validation and test accuracies show a strong linear relationship across models.
- Question categories were distributed consistently across the training, validation, and test splits.
- Validation accuracies were plotted against corresponding test accuracies for each model.
- 0.98 Pearson correlation and 0.97 R2 indicate a strong linear relationship between validation and test accuracies.
- Experiments covered all 537 samples, while the public leaderboard reports test-set performance only.
C.3.2 Token Limit Errors
The harness handles oversized documents and token-limit failures through partial retrieval and conversation truncation. Its simpler context-management strategy removes older messages when conversations exceed the allowed limit.
- Oversized document errors are returned to the agent, which can retry retrieval using partial document chunks.
- When conversations exceed the token limit, older messages are removed until the exchange fits within the allowed limits.
- The authors did not implement a more sophisticated long-short term memory mechanism because context exhaustion was rare in typical conversations.
- The harness uses detailed error messages to help agents correct input-formatting problems.
D.1 Dataset
The dataset emphasizes short-passage questions and evaluates agents using a simple, publicly grounded harness. Results show substantial tool-use variation and faster model execution than human experts, alongside important scope limitations.
- Dataset scope: The dataset emphasizes questions answerable from relatively short passages or focused on the final output of complex reasoning.
- Dataset scope: Citation or source accuracy was requested from models but not evaluated.
- Agentic harness: The harness uses a relatively simple ReAct-inspired architecture and gives models access to publicly available SEC filings rather than private knowledge bases.
- Agent behavior: Tool usage varies substantially across models, including differences in EDGAR Search and RetrieveInformation calls.
- Efficiency: Models are two to more than ten times faster than human experts across the reported analyses.