Source-linked AI summary
FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answering
Chanyeol Choi, Jihoon Kwon, Alejandro Lopez-Lira, Chaewoon Kim, Minjae Kim, Juneha Hwang, Jaeseon Ha, Hojun Choi, Suyeol Yun, Yongjin Kim, Yongjae Lee
TL;DR
Financial retrieval requires precise search across long, complex documents, yet existing approaches and benchmarks do not capture the multi-step reasoning involved. FinAgentBench evaluates document selection and passage ranking with expert-annotated examples, finding strong document-type performance but persistent chunk-level difficulty; domain-specific fine-tuning improves both stages.
Problem
Financial documents require fine-grained, multi-step retrieval, but no benchmark evaluates whether LLMs can select the right document type and locate evidence within it.
Source: §1 · pp. 1–2Method
FinAgentBench provides expert-annotated financial queries for a two-stage evaluation of document-type ranking and chunk-level passage selection.
Source: §1 · p. 2, §3.3 · p. 4Results
LLMs perform strongly on document-type ranking but remain limited at chunk selection; reinforcement fine-tuning raises chunk-ranking MRR from 0.526 to 0.587.
Source: §5 · p. 5, §4.2 · p. 5Takeaways & Limitations
FinAgentBench provides a foundation for studying end-to-end retrieval behavior in complex domains.
Source: §5 · p. 5
Abstract
from arXiv · showhide
Accurate information retrieval (IR) is critical in the financial domain, where investors must identify relevant information from large collections of documents. Traditional IR methods -- whether sparse or dense -- often fall short in retrieval accuracy, as it requires not only capturing semantic similarity but also performing fine-grained reasoning over document structure and domain-specific knowledge. Recent advances in large language models (LLMs) have opened up new opportunities for retrieval with multi-step reasoning, where the model ranks passages through iterative reasoning about which information is most relevant to a given query. However, there exists no benchmark to evaluate such capabilities in the financial domain. To address this gap, we introduce FinAgentBench, the first large-scale benchmark for evaluating retrieval with multi-step reasoning in finance -- a setting we term agentic retrieval. The benchmark consists of 26K expert-annotated examples on S&P-500 listed firms and assesses whether LLM agents can (1) identify the most relevant document type among candidates, and (2) pinpoint the key passage within the selected document. Our evaluation framework explicitly separates these two reasoning steps to address context limitations. This design enables to provide a quantitative basis for understanding retrieval-centric LLM behavior in finance. We evaluate a suite of state-of-the-art models and further demonstrated how targeted fine-tuning can significantly improve agentic retrieval performance. Our benchmark provides a foundation for studying retrieval-centric LLM behavior in complex, domain-specific tasks for finance.
1 Introduction
Sparse and dense retrieval methods face accuracy ceilings on complex financial documents, where investors need precise, multi-step access to information. FinAgentBench addresses the missing benchmark for this setting with expert-annotated document- and chunk-ranking tasks and evaluates LLMs and targeted fine-tuning.
1 Introduction
- Sparse and dense retrieval methods can hit accuracy ceilings when tasks require fine-grained understanding and structured reasoning over complex documents.LLMs have motivated generative retrieval approaches that incorporate deeper reasoning capabilities.Source: §1 · pp. 1–2
- Financial retrieval often requires first selecting the relevant document type and then locating evidence within long, dense filings, but no benchmark tested this multi-step setting.The gap leaves open whether LLMs can serve as retrieval agents in high-stakes finance, where precision, explainability, and structured navigation matter.Source: §1 · p. 2
- FinAgentBench evaluates agentic retrieval through document-level ranking followed by chunk-level ranking, using about 26K expert-annotated samples from real-world financial documents.The benchmark is presented as the first large-scale evaluation of this setting in finance.Source: §1 · p. 2
- The paper evaluates state-of-the-art LLMs and reports that targeted fine-tuning can substantially improve document selection and chunk-level reasoning.The benchmark is intended to support systematic analysis of LLM-based retrieval strengths and limitations in high-stakes finance.Source: §1 · p. 2
2 Related Work
Prior retrieval methods and finance benchmarks leave agentic reasoning about what to retrieve and where to look largely unevaluated. FinAgentBench addresses this gap by testing document identification and within-document chunk ranking.
Retrieval Systems
- Sparse and dense retrievers can struggle with multi-hop reasoning and long-context queries because their retrieval pipelines are fixed.Generative retrieval reframes retrieval as sequence generation, while agentic frameworks such as ReAct incorporate tool use and step-by-step reasoning [26].Source: §2 · p. 2
Finance-Retrieval Benchmarks
- Existing financial benchmarks cover specialized reasoning and retrieval-augmented tasks but generally treat retrieval as a fixed search subroutine.They do not test whether LLMs can reason about what to retrieve or where to look within long documents.Source: §2 · p. 2
- FinAgentBench evaluates whether an LLM can identify the relevant document and rank relevant chunks within it, testing end-to-end reasoning-driven retrieval in finance.The two-stage pipeline first selects a document type and then ranks passages from that document.Source: §2 · p. 2, Figure 2 · p. 3
3 FinAgentBench Dataset
FinAgentBench models financial retrieval as sequential document-type and passage selection, reflecting the need to filter long disclosures before searching their contents. It combines a corporate-document corpus with expert-written queries and relevance annotations.
3.1 Problem Setup
- Long, redundant financial filings make brute-force reasoning over all available text costly and inefficient, motivating document selection before passage selection.Because disclosures follow regular conventions, document type can help indicate where particular information is organized.Source: §3.1 · p. 3
- The benchmark uses a two-stage pipeline: rank five filing types for a query, then rank non-overlapping passages from the selected document.Evaluation uses MRR, MAP, and nDCG at both stages; success requires accurate document-type and chunk-level retrieval.Source: §3.1 · p. 3
3.2 Data Collection
- The corpus draws on SEC disclosures from 2023–2024 across about 3,000 companies, with over 15,000 documents before analysis focuses on approximately 500 S&P 500 documents.The five document types are 10-K, 10-Q, 8-K, earnings call transcripts, and DEF14A proxy statements; tables are kept intact within paragraph-level chunks.Source: §3.2 · pp. 3–4
- Two experienced experts wrote investor queries across ten categories, with cross-validation used to select ten queries per firm for coverage, diversity, and clarity.Categories include analyst Q&A, management commentary, guidance, risks, and operating metrics.Source: §3.2 · p. 4
3.3 Data Annotation
- Finance professionals annotated document-type rankings and graded each passage 0, 1, or 2 for relevance to the query.Each query was independently labeled by two annotators, with disagreements resolved through adjudication.Source: §3.1 · p. 3, §3.3 · pp. 4–5
4 Experiments
The experiments evaluate LLMs on document-type ranking and chunk-level retrieval, then test domain-specific fine-tuning. Results show stronger performance on document-type ranking than chunk-level retrieval, with fine-tuning gains across both stages.
4.1 Experimental Setup
- The evaluation benchmarks reasoning-capable LLMs on document-type ranking and chunk selection, using zero-shot commercial models and a separate GPT-o4-mini fine-tuning comparison.Models rank five SEC filing categories and paragraph-level chunks; tasks use 80/20 training/evaluation splits, with ranking metrics measured at top five and no external tools or retrieval augmentation.Source: §4 · p. 5, §4.1 · p. 5
4.2 Results
- Claude-Sonnet-4 led document-type ranking with nDCG 0.783, MAP 0.849, and MRR 0.892, and all three models performed above random.The reported margins indicate that subtle document-type cues benefit from architectural or scale differences.Source: §4.2 · p. 5
- Chunk retrieval was weaker: Sonnet led nDCG at 0.419, while Opus led MAP at 0.307 and MRR at 0.568.The paper attributes the difficulty to knowledge-intensive reasoning over lengthy financial disclosures.Source: §4.2 · p. 5
- Document-type ranking nDCG improved from 0.758 to 0.808 and MRR from 0.872 to 0.933; chunk-ranking MRR rose from 0.526 to 0.587 after fine-tuning GPT-o4-mini.The evaluation compares GPT-o4-mini before and after reinforcement fine-tuning.Source: §4.2 · p. 5
5 Conclusion
FinAgentBench evaluates agentic retrieval over financial filings, separating document-type selection from locating relevant content. Experiments find strong document-type ranking but limited chunk-level retrieval, while domain-specific fine-tuning improves both stages.
5 Conclusion
- FinAgentBench evaluates realistic investor queries over diverse corporate filings, requiring reasoning about both document types and intradocument content.The benchmark provides a foundation for studying end-to-end retrieval behavior in complex domains.Source: §5 · p. 5
- State-of-the-art LLMs show strong document-type ranking but remain limited at identifying relevant chunks in long, information-dense documents.The paper relates this limitation to the challenges of fine-grained reasoning.Source: §5 · p. 5
- Reinforcement-style fine-tuning with domain-specific supervision substantially improves performance on both retrieval stages.The paper highlights alignment with domain experts’ annotations as valuable.Source: §5 · p. 5
- The authors identify improving agentic retrieval and jointly modeling retrieval with generation for investment decision support as future directions.FinAgentBench is presented as a foundation for studying end-to-end retrieval in complex domains.Source: §5 · p. 5