PaperlayerView exact revision on arXiv

Source-linked AI summary

ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering

Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, William Yang Wang

arXiv:2210.03849v1cs.CL

TL;DR

Complex numerical reasoning in realistic financial conversations remains challenging beyond language-pattern matching and simple prompted tasks. The paper introduces ConvFinQA, constructs conversational financial-report questions, and evaluates neural-symbolic and prompting-based methods, finding substantial gaps from human performance and limitations in coverage and prompting evaluation.

  • Problem

    Current reasoning research has limited evidence on complex numerical reasoning in realistic financial conversations with long-range dependencies.

  • Method

    The paper constructs ConvFinQA from decomposed and concatenated FinQA questions, adds expert-written conversational turns, and evaluates neural-symbolic and prompting-based methods.

  • Results

    Both approaches achieve execution accuracy below 70.0%, versus 89.4% for human performance; GPT-3 175B performs below 50.0%.

  • Takeaways & Limitations

    ConvFinQA provides a resource for studying chains of numerical reasoning in real-world conversational question answering.

  • Takeaways & Limitations

    The conversation construction covers only two mechanisms and does not represent all possible real-world conversations.

Abstract

from arXiv · show

With the recent advance in large pre-trained language models, researchers have achieved record performances in NLP tasks that mostly focus on language pattern matching. The community is experiencing the shift of the challenge from how to model language to the imitation of complex reasoning abilities like human beings. In this work, we investigate the application domain of finance that involves real-world, complex numerical reasoning. We propose a new large-scale dataset, ConvFinQA, aiming to study the chain of numerical reasoning in conversational question answering. Our dataset poses great challenge in modeling long-range, complex numerical reasoning paths in real-world conversations. We conduct comprehensive experiments and analyses with both the neural symbolic methods and the prompting-based methods, to provide insights into the reasoning mechanisms of these two divisions. We believe our new dataset should serve as a valuable resource to push forward the exploration of real-world, complex reasoning tasks as the next research focus. Our dataset and code is publicly available at https://github.com/czyssrs/ConvFinQA.

1 Introduction

ConvFinQA targets complex numerical reasoning in realistic financial-report conversations, where questions may require calculations and depend on earlier turns. It introduces a dataset and evaluates neural-symbolic and prompting-based approaches against human performance.

  • Complex reasoning tasks remain less satisfactory for current research than language-pattern tasks.
  • Financial conversational QA requires extensive calculations and cross-dependent reasoning chains over financial reports.
  • CONVFINQA contains 3,892 conversations and 14,115 questions constructed from decomposed and concatenated FinQA questions with expert-written turns.
  • The dataset evaluates neural-symbolic and prompting-based approaches, which must learn when to use conversation history and assemble reasoning paths.
  • GPT-3 175B achieves less than 50.0% on the task despite stronger performance on simple general reasoning tasks.

2 Related Work

Related work spans conversational QA, numerical reasoning, and financial NLP. ConvFinQA extends prior financial-report QA toward question sequences designed for more natural conversational use.

  • Conversational Question Answering: Conversational QA datasets support follow-up questions, reducing the need to pose one complex multi-hop question at once.
  • Numerical Reasoning: Prior numerical-reasoning datasets mainly cover simple text calculations, math word problems, or prompted explanations in general domains.
  • Financial NLP: Financial NLP has largely focused on sentiment analysis, fraud detection, and opinionated QA, alongside financial-report numerical QA.
  • Financial NLP: CONVFINQA constructs question sequences in a conversational setting for more natural real-world usage and compares them with existing datasets.

3 Task Formulation

The task models conversational QA over financial reports containing text and tables, where later questions may depend on earlier questions. Systems generate executable reasoning programs for the final answer and are evaluated by execution and program accuracy.

  • Inputs comprise financial-report text T, structured table B, and a sequence of questions {Q_i} where later questions may depend on earlier turns.
  • The target is to generate a reasoning program G that executes to produce the answer A for the last question.
  • Possible programs G_i are evaluated to identify the program yielding the correct answer.
  • Reasoning programs use FinQA’s domain-specific language as sequences of operation-argument clauses, such as op1[args1], op2[args2], ..., opn[argsn].
  • Execution accuracy measures the final execution result, while program accuracy measures program equivalence.

4 The CONVFINQA Dataset

CONVFINQA constructs realistic financial-report conversations by simulating reasoning flows and having finance experts realize them as textual questions. Its statistics indicate long-range, cross-aspect numerical dependencies and substantial expertise requirements.

  • Dataset Construction: The two-stage construction framework simulates conversational QA flows, then composes textual questions from turn-level reasoning semantics.The framework uses decomposition and concatenation of FinQA multi-hop reasoning programs before expert question composition.
  • Conversation Flow Simulation: Type I conversations decompose one multi-hop question into single-step turns, while Type II conversations integrate decompositions of two questions from the same report.Additional number-selection turns are inserted when new numbers appear, and hybrid conversations concatenate reasoning flows across related report aspects.
  • Question Composition: Expert annotators can skip redundant turns and use references to make simulated conversation flows more natural.Annotators are given reports, conversation skeletons, and turn-level reasoning semantics when composing questions.
  • Dataset Statistics: The dataset contains 3,892 conversations and 14,115 questions, split into 3,037/421/434 train/dev/test conversations.
  • Dataset Analysis: 65.0% of 200 sampled hybrid turns depended on the first question set, while 34.73% were number-selection questions and 35.10%, 25.41%, and 4.75% used 1-, 2-, and over-3-step programs.These statistics characterize cross-question dependencies and the distribution of reasoning forms in CONVFINQA.
  • Data Quality Assessment: Expert annotators achieved 89.44% execution accuracy and 86.34% program accuracy, compared with 46.90% and 45.52% for lay annotators.Expert agreement exceeded 85.0% for both metrics, whereas lay annotator agreement was below 60.0%.

5 Experiments on Neural Symbolic Approaches

The experiments evaluate neural-symbolic models for conversational financial numerical reasoning, combining retrieval of report evidence with program generation. Analyses show that question type and conversation position strongly affect performance, while longer dependencies and missing financial knowledge remain major challenges.

  • Methods and Main Results: FinQANet retrieves supporting facts before generating reasoning programs, using conversation context to retrieve evidence from the financial report.The model follows a retriever-generator pipeline and incorporates questions up to the current turn.
  • Methods and Main Results: 86.38% recall is achieved for the top 3 retrieved facts in the adopted retrieval process.
  • Performance Breakdown: Number selection questions are easiest, whereas hybrid conversations are harder than simple conversations, especially in their second part.The second hybrid segment combines irrelevant questions with questions that depend on the first segment.
  • Performance Breakdown: Later conversation turns are harder because they involve longer reasoning dependencies on previous context.The model faces greater difficulty finding the correct reasoning chains as dependencies lengthen.
  • Analyses and Findings: The model performs well on number selection and usually transitions correctly when previous context should be discarded.
  • Analyses and Findings: Missing financial knowledge produces errors in retrieving facts, selecting values, and generating mathematical reasoning.The analysis argues that stronger domain knowledge is still needed for high-level, complex financial reasoning.

6 Experiments on Prompting-Based Approaches

The prompting experiments use GPT-3 in retrieval-generation settings and compare answer-only, program-generation, and Chain-of-Thought prompts. Results show weaknesses in retrieval, conversational reference, complex arithmetic, unfamiliar program formats, and new task paradigms.

  • 6.1 Methods and Main Results: GPT-3 retrieval concatenates report sentences or table rows with conversation context and predicts relevance using 16 exemplars.The experiments use a retriever-generator setup because full-report prompts exceed the model's length constraint.
  • 6.1 Methods and Main Results: 74.25% average recall on 300 test examples was achieved across three exemplar sets, below the full-training retriever's recall.
  • 6.1 Methods and Main Results: The program-generation comparison includes Answer-only, Program-original, Program-normal, and Chain-of-Thought prompting with expert-written explanations.Program-normal converts reasoning programs into common mathematical notation, while CoT includes natural-language reasoning steps.
  • 6.1 Methods and Main Results: Even with gold retrieval, GPT-3 underperforms neural-symbolic approaches trained on the full dataset.Table 5 reports averages and standard deviations across exemplar sets or annotators.
  • 6.2 Performance Breakdown: GPT-3 performs worse on number-selection turns when answers require references to previous conversational context.It often selects a value from the previous year instead of the value requested in the current turn, despite explicit instructions.
  • 6.3 Analyses and Findings: GPT-3 often produces correct numerical results without the requested reasoning program, but struggles with long-digit calculations and divisions.This creates a larger gap between execution accuracy and program accuracy for program-generating methods.
  • 6.3 Analyses and Findings: Program-normal outperforms Program-original because its common calculation format is more familiar to GPT-3.Program-original also incurs many grammar errors.
  • 6.3 Analyses and Findings: GPT-3 struggles with the new complex task paradigm, often copying exemplar reasoning while ignoring the actual conversational context.CoT performs even worse than generating the program alone in these cases.

7 Conclusion and Discussion

The conclusion positions CONVFINQA as a benchmark for complex reasoning in realistic settings. The authors compare neural-symbolic and prompting-based approaches and identify specialized modeling and task boundaries as open issues.

  • CONVFINQA targets the study of human-like reasoning abilities in complex real-world settings.
  • Both neural-symbolic and prompting-based approaches remain below human expert performance on this task.
  • Neural-symbolic models learn domain-specific numerical reasoning from crafted architectures and large-scale training data, whereas prompting recalls reasoning patterns from pre-training.
  • For highly specialized domains or tasks, designing specific models may be more realistic and effective than relying only on language-model prompting.
  • The authors identify the boundary between reasoning tasks that benefit from language modeling and those that do not as a key question for large-model reasoning.

8 Limitations

The dataset construction covers only two conversation-generation mechanisms and the prompting study is limited to GPT-3 with restricted prompt engineering.

  • The conversation construction uses decomposition of one multi-hop question or decomposition and concatenation of two questions about the same report.The authors state that these mechanisms do not cover all possible real-world conversations.
  • The prompting experiments use only GPT-3 and do not extensively explore complex prompt engineering because of cost constraints.The authors leave larger models and advanced prompt engineering for future work.

9 Ethical Considerations

The dataset annotation process used paid Upwork workers, while the resulting systems are intended to augment rather than replace human financial-analysis experts.

  • Upwork workers received approximately $4.0 per simple conversation and $7.0 per complex conversation after paid introductory interviews.Evaluation and prompt-writing tasks followed a similar procedure and rate structure.
  • The institutional review board classified the dataset annotation as exempt.
  • Systems trained on CONVFINQA are intended to augment human decision-making in financial analysis, not replace human experts.

Appendix A: Operation Definitions

The appendix documents the operation definitions and annotation interface used to turn decomposed program steps into questions. It also identifies the validation-results table.

  • Annotators view financial reports and decomposed program lists, then rewrite each program step as a question.The annotation platform is a locally runnable Django-based web application.
  • Table 7 provides definitions for all operations used in the dataset.
  • Figure 7 presents the annotation interface used for this rewriting process.
  • Table 8 reports validation results.

Appendix C: Experiment Details

The appendix details neural-symbolic training settings and several GPT-3 prompt formats for conversational numerical reasoning. It also describes turn-indexed evaluation and reports associated tables, figures, and error cases.

  • Neural-symbolic experiments: Neural-symbolic models use TITAN RTX GPUs, Hugging Face Transformers, Adam, batch size 16, and learning rates of 1e-5 or 1e-4 for T-5.Table 8 contains the validation-set results.
  • GPT-3 prompts: GPT-3 prompts include retrieval, answer-only, program, and chain-of-thought formats using the question sequence up to the current turn.The retrieval prompt predicts whether a supporting fact candidate is needed, while answer-only prompts request the execution result.
  • GPT-3 prompts: Program prompts request multi-step mathematical solutions and use symbols such as #0 and #1 for intermediate results.The program output is provided in a solution field.
  • GPT-3 prompts: Chain-of-thought prompts request step-by-step explanations together with the program.
  • Evaluation details: Question indices such as Q1 and Q2 are added before questions in each sequence.The appendix also references turn-wise performance, exemplar-count results, and GPT-3 error cases.
Loading 2210.03849v1…