Source-linked AI summary
FinanceBench: A New Benchmark for Financial Question Answering
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, Bertie Vidgen
TL;DR
Financial QA requires reliable retrieval and reasoning over complex, current information, but rigorous finance-specific evaluation has been limited. The paper introduces FinanceBench and evaluates 16 model configurations on a manually reviewed sample. Models show substantial weaknesses, including only 79% correctness for the best non-oracle configuration, motivating robust evaluation and result verification.
Problem
FinanceBench addresses the limited availability of rigorous, systematic, and measurable evaluation for finance-specific LLM tasks involving document retrieval and reasoning.
Method
The paper constructs a 10,231-question open-book benchmark and manually evaluates 2,400 responses from 16 model configurations on 150 cases.
Results
79% correctness was achieved by the best tested model outside the oracle setting, while qualitative review identified refusals, irrelevant responses, and hallucination risks.
Takeaways & Limitations
The authors encourage robust predeployment evaluation, additional evidence sources, and cross-checking financial QA results with multiple sources.
Takeaways & Limitations
FinanceBench contains only publicly listed companies, biasing its coverage toward larger companies and properly audited, well-written documents.
Abstract
from arXiv · showhide
FinanceBench is a first-of-its-kind test suite for evaluating the performance of LLMs on open book financial question answering (QA). It comprises 10,231 questions about publicly traded companies, with corresponding answers and evidence strings. The questions in FinanceBench are ecologically valid and cover a diverse set of scenarios. They are intended to be clear-cut and straightforward to answer to serve as a minimum performance standard. We test 16 state of the art model configurations (including GPT-4-Turbo, Llama2 and Claude2, with vector stores and long context prompts) on a sample of 150 cases from FinanceBench, and manually review their answers (n=2,400). The cases are available open-source. We show that existing LLMs have clear limitations for financial QA. Notably, GPT-4-Turbo used with a retrieval system incorrectly answered or refused to answer 81% of questions. While augmentation techniques such as using longer context window to feed in relevant evidence improve performance, they are unrealistic for enterprise settings due to increased latency and cannot support larger financial documents. We find that all models examined exhibit weaknesses, such as hallucinations, that limit their suitability for use by enterprises.
1 Introduction
Financial analysts perform difficult, time-intensive information retrieval, summarization, and reasoning, yet rigorous evaluation of finance-specific LLM capabilities remains limited. FinanceBench addresses this gap with an open-book benchmark and systematic model evaluation.
- Financial specialists retrieve, summarize, analyze, and reason about company and industry information for investment decisions, financial strategies, and due diligence.
- Rigorous finance-specific evaluation is needed to identify model strengths and weaknesses, assess readiness for high-stakes use, and track capability changes over time.
- Finance QA challenges include domain knowledge, outdated model information, numerical reasoning, mixed structured and unstructured inputs, and long multi-document contexts.
- FinanceBench provides an open-book benchmark and evaluates 16 model configurations across closed-book, oracle, vector-store, and long-context settings.The study manually checked 2,400 answers from a diverse 150-case sample, which is available open-source.
2 Prior work
Existing QA benchmarks often lack financial coverage and do not reflect analysts’ document-retrieval and reasoning workflows. FinanceBench is motivated by the need for ecologically valid open-book financial QA evaluation.
- Finance-specific models such as BloombergGPT, FinGPT, and ConFIRM demonstrate growing attention to financial language modeling and information retrieval.
- Generic QA benchmarks typically contain few or no financial questions, and strong open-domain performance does not necessarily generalize to financial QA.
- Existing finance benchmarks cover specialized areas such as sentiment, numerical reasoning, conversational QA, or exam questions rather than the full day-to-day analyst workflow.
- Financial QA evaluation should include retrieval from relevant documents and reasoning over them, rather than supplying only the information needed for the answer.
3 FINANCEBENCH Dataset
FinanceBench is a large benchmark of question-answer-evidence triplets drawn from public-company filings and designed around realistic financial analysis tasks. Its questions span domain knowledge, company-specific reasoning, and metric computation.
- 10,231 question-answer-evidence triplets cover 40 publicly traded U.S. companies and 361 filings released from 2015 to 2023.The filings include 10-Ks, 10-Qs, 8-Ks, and earnings reports.
- Domain-relevant questions ask broadly applicable financial-analysis questions and contributed 925 entries across 37 companies.
- Novel generated questions are company-, report-, and industry-specific, and annotators were instructed to make them realistic, varied, and reasoning-oriented.They were designed to reflect questions a financial analyst would ask in real-world settings.
- Metrics-generated questions target analyst computation and reasoning by extracting base metrics from financial statements across eight years.Templates introduced phrasing variation, and these questions contributed 7,983 entries.
- The taxonomy distinguishes information extraction, numerical reasoning, and logical reasoning across 8,908 domain-relevant and metrics-generated questions.The distribution is 2,493 extraction questions, 5,897 numerical-reasoning questions, and 518 logical-reasoning questions.
- Human evaluation used 150 cases sampled evenly across the three question types to support diverse, fine-grained analysis of model capabilities.
4 Experimental Setup
The experiments compare four LLMs across 16 configurations spanning closed-book, oracle, vector-store, and long-context approaches. Responses were manually categorized to assess correctness and failure behavior.
- Four LLMs were tested across five setups and two prompt orders, producing 16 distinct configurations.The models included GPT-4, GPT-4-Turbo, Claude2, and Llama2.
- The closed-book setup provides no additional context, while the oracle setup supplies the evidence page and removes the retrieval challenge.The oracle condition is explicitly described as unrealistic and substantially easier.
- Single and shared vector stores test retrieval-based implementations, with the shared store indexing all 360 FinanceBench documents.The shared setup uses Chroma, LangChain, and OpenAI embeddings.
- Long-context prompting feeds the relevant public filing directly to GPT-4-Turbo or Claude2, although some filings exceed available context windows and require truncation.
- Context-First and Context-Last schemes swap the relative order of evidence and question, enabling tests of prompt-position effects.The study evaluates both schemes in selected oracle and long-context configurations.
- Researchers manually labeled each response as correct, incorrect, or failure to answer using explicit criteria tied to the gold-standard evidence.Contradictory reasoning is labeled incorrect even when the final answer is right.
5 Results on FINANCEBENCH
FINANCEBENCH results show that performance depends strongly on access to relevant information, retrieval and prompt order, while models retain substantial error and hallucination risks. Performance also varies by question type, and qualitative review identifies both useful and problematic response behaviors.
- Overall performance: 9% of prompts received correct answers from GPT-4-Turbo in the closed-book configuration.Across augmented configurations, success rates ranged from 20% to 78%, while the Oracle configuration reached 85%.
- Overall performance: 50% versus 19% success was observed for GPT-4-Turbo with one vector store per document versus one vector store for all documents.Llama2 showed the same direction, with 41% versus 19% success.
- Overall performance: 79% was the long-context success rate for GPT-4-Turbo, compared with 76% for Claude-2.Both models had more incorrect answers than refusals in these settings, with GPT-4-Turbo at 17% incorrect versus 4% refusal and Claude-2 at 21% versus 3%.
- Performance by question type: Metrics-generated questions were typically the hardest, while extraction-oriented novel questions generally produced better performance.Metrics-generated questions often require complex numeric reasoning and multiple document passages.
- Performance by prompt order: 78% versus 25% success for GPT-4-Turbo and 76% versus 37% for Claude-2 show that Context-First substantially outperformed Context-Last in the LongContext setting.The reversed order slightly outperformed Context-First in the shorter Oracle setting, 89% versus 85%.
- Qualitative analysis of responses: Qualitative review found high-quality correct answers, valid alternatives, hallucinations, helpful refusals, and irrelevant comments.Hallucinations were superficially coherent but unfaithful to the source, while helpful refusals explained how to answer but remained technically failures.
6 Limitations of FINANCEBENCH
FINANCEBENCH has limitations in conversational coverage, document scope, cross-company comparison, and dataset difficulty or ambiguity. Despite these constraints, the authors judge its gold labels high quality.
- Single-turn conversations: FINANCEBENCH contains only single questions and answers, although analysts often ask streams of dynamically adapting questions in conversation.The authors identify single-turn coverage as a limitation but prioritize high-quality answers to individual questions.
- Public filings: FINANCEBENCH includes only publicly listed companies, biasing it toward larger companies and properly audited, well-written documents.This scope follows the decision to avoid private documents with commercial-sensitivity and privacy concerns.
- Lack of cross-company comparisons: FINANCEBENCH was designed for questions about single companies rather than comparisons between two companies.The authors note that cross-company questions require handling two separate documents, which is harder than handling two strings from one document.
- Dataset integrity: Some questions are ecologically valid but simplistic, allowing models to answer correctly without additional information and producing higher benchmark performance.The authors describe these questions as suitable for a first line of evaluation.
- Dataset integrity: Some correct answers are ambiguous because they depend on the analyst’s context, assumptions, or priorities, making certain gold labels contestable.The authors nevertheless believe the gold labels are high quality overall.
7 Conclusion
FINANCEBENCH finds critical weaknesses in state-of-the-art models for financial QA, including risks that may remain hidden without detailed inspection. The authors recommend robust evaluation, supplementary evidence, and result checking before high-stakes use.
- 7 Conclusion: 79%: GPT-4-Turbo with the long-context window was correct outside the unrealistic Oracle setting.The authors state that even this best-performing tested model could not be used with confidence in a live industry setting.
- 7 Conclusion: Models may produce hallucinated, outdated, logically incorrect, or wrongly unitized answers that appear reasonable.The authors characterize these as serious risks to effective financial analysis that may require detailed inspection to detect.
- 7 Conclusion: Analysts should robustly evaluate financial-QA models before high-stakes live deployment.The authors also encourage additional information sources, double-checking, and triangulation across multiple evidence sources.
- 7 Conclusion: The authors encourage using vector stores and long-context content, while also double-checking results against multiple evidence sources.They present these practices as ways to improve performance and verify findings.
A Phrasing variations for the metrics-based questions
Metrics-based questions use controlled phrasing and reference variation to make questions realistic and diverse. The construction varies introductions, endings, financial-statement references, ordering, and units.
- A Phrasing variations for the metrics-based questions: 22 introductory clauses and 14 ending clauses provide vanilla and creative phrasing variations for metrics-based questions.The templates include 11 vanilla and 11 creative introductory clauses, plus 7 vanilla and 7 creative ending clauses.
- A Phrasing variations for the metrics-based questions: Each financial statement has 2-3 unique ways of being referenced.This creates additional wording variation for the same underlying metric question.
- A Phrasing variations for the metrics-based questions: Randomness changes the ordering of referenced statements and the units used in questions.Examples include asking for operating income in USD millions or D&A as a percentage of total revenue.
B LLM implementation
The study evaluates 16 model configurations, including Oracle and Closed book settings, using fixed implementation choices. Long-context inputs were truncated when filings exceeded the available context.
- B LLM implementation: 16 model configurations were tested, including the Oracle and Closed book settings.All configurations were tested in November 2023.
- B LLM implementation: Llama2 was accessed through Replicate, while OpenAI and Anthropic models were accessed through their respective APIs.The study used default system prompts for all calls.
- B LLM implementation: Temperature was set to 0.01 and maximum token length to 2,048 for all calls.The employed prompts are shown in Table 3.
- B LLM implementation: 95,000 tokens: long-context filing inputs were truncated when the relevant filing exceeded the available context.This applied when the filing did not fit into the possible context.
C Data Documentation
FinanceBench documents each question with identifiers, company and filing metadata, question labels, answers, evidence, page references, and optional justifications. The dataset spans 40 companies and 360 documents across multiple question types and document categories.
- Dataset composition: The dataset includes 10,231 cases, with 150 used for expert evaluation and made available in the open-source sample.
- Dataset fields: Each FinanceBench entry includes a unique ID, evaluation/open-source status, company name, GICS sector, filing name and link, document type, and fiscal year.
- Dataset fields: Each entry records the question type, reasoning type, question text, gold-standard answer, evidence text, evidence page number, full page text, and any answer justification.Evidence for domain-relevant and novel-generated questions is extracted by annotators; metrics-generated evidence is constructed from source pages.
- Dataset composition: 40 companies appear in FinanceBench, with 32 represented in metrics-generated questions, 37 in domain-generic and novel-generated questions, and 29 in all three types.
- Dataset composition: The corpus contains 360 documents, including 270 10Ks, 5 annual reports, 29 8Ks, and 29 earnings reports.The 270 10Ks account for 75% of all documents.
D Comparison of GPT-4 and GPT-4-Turbo
The paper compares GPT-4 and GPT-4-Turbo across evaluation configurations, reporting GPT-4-Turbo in the main text because it performed better. LongContext performance cannot be compared because GPT-4 does not support that setting.
- GPT-4-Turbo performed better than GPT-4 and is therefore the model shown in the main text.
- LongContext performance is not compared because GPT-4 does not support a long-context setting.
- Figure 5 compares GPT-4 and GPT-4-Turbo across the different evaluation configurations.