Source-linked AI summary
FinQA: A Dataset of Numerical Reasoning over Financial Data
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, William Yang Wang
TL;DR
Financial analysis requires heterogeneous-data understanding and complex numerical reasoning, while expanding financial documents make deep analysis difficult to perform manually. FinQA addresses this challenge with an expert-annotated dataset and reasoning programs; baseline results show current large pretrained models remain far behind human experts.
Problem
Expanding financial documents require heterogeneous-data understanding and complex numerical reasoning, creating uncertainty about whether humans can analyze them sufficiently for current decision making.
Method
The paper constructs FINQA from S&P 500 earnings reports, containing 8,281 expert-written financial QA pairs with annotated numerical reasoning processes and supporting facts.
Results
Baseline experiments show that current large pretrained models still fall far behind human expert performance on financial knowledge and complex numerical reasoning.
Takeaways & Limitations
FINQA provides a resource for developing pretraining tasks and further research on realistic, complex application domains.
Takeaways & Limitations
Execution accuracy can overestimate performance through chance, whereas program accuracy can produce false negatives when multiple correct programs exist.
Abstract
from arXiv · showhide
The sheer volume of financial statements makes it difficult for humans to access and analyze a business's financials. Robust numerical reasoning likewise faces unique challenges in this domain. In this work, we focus on answering deep questions over financial data, aiming to automate the analysis of a large corpus of financial documents. In contrast to existing tasks on general domain, the finance domain includes complex numerical reasoning and understanding of heterogeneous representations. To facilitate analytical progress, we propose a new large-scale dataset, FinQA, with Question-Answering pairs over Financial reports, written by financial experts. We also annotate the gold reasoning programs to ensure full explainability. We further introduce baselines and conduct comprehensive experiments in our dataset. The results demonstrate that popular, large, pre-trained models fall far short of expert humans in acquiring finance knowledge and in complex multi-step numerical reasoning on that knowledge. Our dataset -- the first of its kind -- should therefore enable significant, new community research into complex application domains. The dataset and code are publicly available\url{https://github.com/czyssrs/FinQA}.
1 Introduction
FINQA frames financial analysis as a difficult numerical-reasoning problem over heterogeneous reports and introduces an expert-annotated dataset with executable reasoning programs and baseline systems. Models remain substantially below expert performance despite the proposed retriever-generator approach.
- Financial analysis requires complex numerical reasoning across structured tables and unstructured text, while poor analysis can cost billions of dollars.
- FINQA addresses whether deep analysis of expanding financial documents can be automated.
- FINQA questions require selecting information from both tables and unstructured text and applying calculations such as addition, comparison, and table aggregation.
- The retriever-generator framework achieves 65.05% execution accuracy, compared with 50.68% for non-experts and 91.16% for human experts.
- The paper proposes financial-data QA, constructs FINQA, and finds baseline models remain far behind expert performance.
- The dataset contains 8,281 financial QA examples written by financial experts with fully annotated numerical reasoning programs.
2 Related Work
Prior numerical QA datasets largely target general-domain data, whereas FINQA focuses on complex numerical reasoning over real-world financial reports combining tables and text. The paper identifies this as an unaddressed financial QA setting.
- Existing numerical QA datasets such as DROP generally use general-domain sources and involve less calculation than financial questions.
- HybridQA combines tables and text but does not focus on numerical reasoning, while FINQA centers on that combination in finance.
- Financial NLP has included fraud detection, sentiment analysis, opinionated QA, and finance-specific pre-trained models, mostly for sentiment classification.
- The authors state that no previous work or dataset built QA systems for numerical reasoning on financial reports.
3 Task Definition
FINQA formulates financial QA as generating an executable reasoning program from report text, tables, and a question, then executing it to obtain an answer. Its DSL supports mathematical and table operations, while evaluation distinguishes answer execution from program generation.
- Problem Formulation: Given textual contents E, structured table T, and question Q, the task generates a DSL reasoning program G and executes it to produce answer A.
- Problem Formulation: The paper focuses on regular financial-table layouts and excludes more complicated nested structures for simplicity.
- Domain Specific Language: The DSL contains mathematical operations and table aggregations, including add, subtract, multiply, divide, greater, exp, table-max, table-min, table-sum, and table-average.
- Domain Specific Language: Programs are operation sequences whose arguments can be report values, table row names, or results from earlier steps.
- Evaluations: The evaluation provides gold programs alongside execution accuracy because financial applications require explainability and transparency.
- Evaluations: Execution accuracy can overestimate performance through chance matches, whereas program accuracy can produce false negatives when multiple correct programs exist.
4 The FINQA Dataset
FINQA is built from S&P 500 earnings reports and expert-authored questions, reasoning programs, and supporting-fact annotations. The dataset emphasizes numerical reasoning over financial tables and text while restricting complex table layouts.
- Data Preparation: FINQA uses publicly available S&P 500 earnings reports from 1999 to 2019, containing financial tables and textual content.The reports were collected from FinTabNet, which provides table annotations.
- Data Preparation: 12,719 report pages were selected after filtering out oversized, nested, multi-header, and otherwise unsuitable tables.The filtering retained pages with at most one table and simplified tables with two description headers.
- Annotation Procedure: Eleven US-based finance professionals created the dataset after training on the annotation task.Annotators included professionals with backgrounds such as CPAs and MBAs.
- Annotation Procedure: Each annotation records a meaningful financial question, a reasoning program of up to five operations, and the supporting sentences or table rows.The interface validated operation inputs, and annotators marked all information needed to answer each question.
- Dataset Statistics: FINQA contains 8,281 examples split into 6,251 training, 883 validation, and 1,147 test examples without overlapping input reports.The split follows a 75%/10%/15% allocation.
- Dataset Statistics: 62.43% of questions require only table information, 23.42% only text, and 14.15% both text and table information.Reasoning programs contain one step in 59.10% of examples, two steps in 32.71%, and three or more steps in 8.19%.
5 Baseline Systems
FinQANet combines retrieval of relevant report facts with generation of executable reasoning programs. Its generator uses pretrained language models, DSL tokens, and step memory, while baselines test simpler or alternative architectures.
- The FinQANet Framework: FinQANet first retrieves supporting facts from financial reports and then generates an executable program to answer each question.Retrieved facts include text sentences and table rows.
- The FinQANet Framework: The retriever converts table rows into sentences, pairs each fact with the question, and uses pretrained language models to select the top n facts.This addresses reports that can exceed 2,000 tokens.
- The FinQANet Framework: The program generator can emit input numbers or row names, DSL special tokens, and step-memory tokens representing previous results.An LSTM decoder combines attention over the input and decoding history with contextual information.
- The FinQANet Framework: Step-memory embeddings are updated after each operation step to represent the program’s reasoning path.The update occurs when the decoder generates the ending parenthesis of an operation-argument unit.
- The FinQANet Framework: Grammar masks enforce structural correctness during program decoding, and the generator is trained on retrieved facts, supplemented with gold facts for retrieval errors.At inference, the retriever supplies the top n results to the generator.
- Baseline Systems: Baselines include TF-IDF with a single operation, direct answer generation, Seq2seq generation, NeRd, and a Longformer without retrieval.These variants test simplified reasoning, alternative program generators, and direct encoding of the full report.
6 Experimental Results
Experiments show that FinQANet benefits from pretrained encoders and the retriever-generator design, but performance declines on heterogeneous, multi-step, and constant-dependent questions. Error analysis attributes failures to retrieval, financial knowledge, and numerical reasoning.
- Experiment Setup: 89.66% recall for the top 3 retrieved facts supports using a compact retriever input for program generation.The setup uses the top 3 ranked facts because longer inputs lower generator performance.
- Performance Breakdown: 45.81% table-only and 15.80% text-only accuracy fall below the 61.24% full-result accuracy, showing value in combining both sources.The breakdown evaluates inferences using facts from only one source versus the complete input.
- Performance Breakdown: 67.38% accuracy on table-only questions exceeds the 43.80% accuracy on questions requiring both tables and text.Questions combining tables and text are the most challenging, while tables have more unified structures.
- Performance Breakdown: 22.78% accuracy on questions requiring three or more steps shows that multi-step reasoning is especially challenging.Questions requiring only one step are reported as the easiest.
- Performance Breakdown: 43.88% accuracy on programs with constants is lower than the 61.24% whole-set accuracy, reflecting conversion and implicit-number challenges.Constants support unit conversion and can represent unstated denominators, such as 3 for averaging three years.
- Error Analysis: 15% of sampled errors come from retrieval, while remaining errors mainly involve financial knowledge and numerical reasoning.Numerical errors include multi-step programs, unit conversions, and matching numbers with years.
7 Conclusion and Future Work
FINQA targets numerical reasoning over real-world financial data through expert annotation, reasoning programs, baseline frameworks, and comprehensive analysis. Results show that large pretrained models remain far behind human experts, motivating further work on complex financial applications.
- Conclusion: FINQA is an expert-annotated dataset for numerical reasoning over real-world financial data.The paper frames the dataset around domain-specific knowledge and complex numerical reasoning abilities.
- Future Work: Baseline frameworks and comprehensive experiments show that current large pretrained models remain far behind human expert performance.The conclusion identifies this gap as motivation for future pretraining research in realistic, complex application domains.
8 Ethical Considerations
FINQA uses publicly available S&P 500 earnings reports and paid annotation work, with IRB approval governing the project. The trained systems are intended to augment, not replace, financial-analysis experts.
- Data Access and Licensing: FINQA is built from publicly available S&P 500 earnings reports from 1999 to 2019 under a permissive data license.The license permits additional annotations and publication of those annotations.
- Dataset Collection Process and Conditions: Annotators were introduced to the task through paid interviews and received around $2.0 per question for large-scale annotation.Workers first tried example questions before agreeing to continue with the larger job.
- IRB Approval: IRB approval covers the project, and systems trained on FINQA are intended to augment human decision-making rather than replace experts.The stated use boundary concerns financial analysis.
Appendix B: Experiment Details
The appendix reports validation procedures, optimization settings, and implementation resources for the baseline experiments. Models were trained on TITAN RTX GPUs using Hugging Face Transformers and Adam.
- Implementation: All baseline validation results are reported from models trained on TITAN RTX GPUs using Hugging Face Transformers and Adam.The appendix also specifies optimizer and model-training settings for the experiments.
- Parameter Settings: The retriever uses a 3e-5 learning rate and batch size 16, while FinQANet uses a 1e-5 learning rate.The appendix lists separate parameter settings for the retriever and FinQANet.
- Implementation: Additional modeling details are provided in the released code.The appendix directs readers to the code for details beyond the listed settings.
Appendix C: Case Studies
The appendix presents additional case studies using complete input reports. Gold evidence is highlighted in blue for every example.
- Additional case studies are presented with the full input reports.
- Each example identifies its gold evidence through blue highlighting.
- The case studies are designed to show complete report inputs rather than isolated excerpts.
Appendix D: Annotation Interface
The annotation interface uses a local Django-based web application and provides automated input validation. The appendix also presents operation definitions, validation-set results, full report inputs, and error case studies through tables and figures.
- Annotation platform: Turkle7 provides the Django-based annotation platform, which runs on a local server.
- Annotation platform: Annotators use a validation check button to automatically verify completed inputs.
- Annotation interface: Figure 7 and Figure 8 show the annotation interface and its annotator input fields.
- Supporting materials: Table 4 defines all operations used in the annotation process.
- Supporting materials: Table 5 reports results on the validation set.
- Case studies: The appendix includes full input reports and case studies of retrieval, table-understanding, and complex numerical-reasoning errors.