PaperlayerView exact revision on arXiv

Source-linked AI summary

Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks

Xianzhi Li, Samuel Chan, Xiaodan Zhu, Yulong Pei, Zhiqiang Ma, Xiaomo Liu, Sameena Shah

arXiv:2305.05862v2cs.CLcs.AI

TL;DR

The effectiveness of general-purpose LLMs on financial text analytics remains insufficiently understood. This study compares ChatGPT and GPT-4 with fine-tuned and domain-specific models across benchmark tasks, finding they outperform alternatives on some tasks but fall short on others, particularly those needing deeper semantics or structural analysis.

  • Problem

    The effectiveness of broadly trained LLMs on financial text analytics remains insufficiently understood, despite their strong performance on generic NLP tasks.

    Source: §1 · p. 1

  • Method

    The study empirically evaluates ChatGPT and GPT-4 across eight financial benchmark datasets and compares them with fine-tuned and domain-specific models.

    Source: §1 · p. 1

  • Results

    ChatGPT and GPT-4 outperform domain-specific and fine-tuned models on many tasks but fall short on others, particularly those requiring deeper semantics and structural analysis.

    Source: §1 · p. 1, §6 · p. 6

  • Takeaways & Limitations

    General-purpose LLMs may suit relatively simple financial NLP tasks, while pretraining plus fine-tuning remains a leading option for more complicated structured prediction.

    Source: §5 · p. 6

    The study is an initial investigation, and existing LLMs still fall short particularly on tasks requiring deeper semantics and structural analysis.

    Source: §1 · p. 1, §6 · p. 6

Abstract

from arXiv · show

The most recent large language models(LLMs) such as ChatGPT and GPT-4 have shown exceptional capabilities of generalist models, achieving state-of-the-art performance on a wide range of NLP tasks with little or no adaptation. How effective are such models in the financial domain? Understanding this basic question would have a significant impact on many downstream financial analytical tasks. In this paper, we conduct an empirical study and provide experimental evidences of their performance on a wide variety of financial text analytical problems, using eight benchmark datasets from five categories of tasks. We report both the strengths and limitations of the current models by comparing them to the state-of-the-art fine-tuned approaches and the recently released domain-specific pretrained models. We hope our study can help understand the capability of the existing models in the financial domain and facilitate further improvements.

1 Introduction

Although general-purpose LLMs perform impressively on generic NLP tasks, their effectiveness in finance remains insufficiently understood. This study compares ChatGPT and GPT-4 with domain-specific and fine-tuned models across financial benchmarks, finding strengths and limitations.

1 Introduction

  • Despite impressive performance on generic NLP tasks, the applicability and effectiveness of broadly trained LLMs in finance remain insufficiently understood.The paper notes that understanding their effectiveness can influence financial applications.Source: §1 · p. 1
  • The study evaluates recent LLMs on eight benchmark datasets spanning five typical financial text analytics tasks and compares them with fine-tuned and domain-specific pretrained models.Table 1 gives statistics for the five tasks and eight datasets.Source: §1 · p. 1, Table 1 · p. 2
  • The authors characterize the study as among the first comprehensive explorations of generically trained LLMs on financial text analytics.They report detailed analysis and recommendations.Source: §1 · p. 1
  • ChatGPT and GPT-4 outperform domain-specific pretrained and fine-tuned models on many tasks, while GPT-4 is significantly better than ChatGPT on nearly all tested financial benchmarks.The study also analyzes limitations of existing LLMs on these benchmark datasets.Source: §1 · p. 1

2 Related Works

Prior work reports broad capabilities and some failures for general-purpose LLMs, while their effectiveness on financial text analytics remains an open question. Finance-specific modeling includes BloombergGPT and adaptations of existing language models.

ChatGPT and Related Models

  • Prior studies report that ChatGPT and related models outperform fine-tuned systems on some tasks but still fail on others, leaving their effectiveness in financial text analytics open.The cited studies examined multitasking, multilingual and multimodal tasks, and zero-shot NLP capabilities.Source: §2 · p. 2

Domain-specific Models

  • Only a handful of LLMs are trained specifically for finance; BloombergGPT uses mixed training and is reported to perform strongly on financial benchmarks and competitively on some general NLP benchmarks.BloombergGPT is described as a 50-billion-parameter model; other work adapted T5 and used parameter-efficient tuning for finance tasks.Source: §2 · p. 2

3 Experiment Setup

The setup tests general-purpose LLMs on a range of financial NLP tasks, using prompting strategies and comparisons with earlier, domain-specific, and fine-tuned models. It evaluates classification, entity extraction, and other tasks with task-appropriate metrics.

Tasks and Datasets

  • The testbed spans sentiment analysis and text classification, information extraction, and question answering, covering different modeling complexity and financial knowledge demands.The section characterizes sentiment and classification as more straightforward than information extraction, and QA as often requiring internal logic or numerical reasoning.Source: §3 · p. 2

Models

  • ChatGPT and GPT-4 are evaluated with zero-shot, few-shot, and chain-of-thought prompting, and compared with previous LLMs, BloombergGPT, and state-of-the-art fine-tuned models.Most experiments use gpt-3.5-turbo and GPT-4 (8k); FinQA few-shot uses gpt-3.5-turbo-16k because of longer inputs, and the reported model versions are current as of July 7, 2023.Source: §3 · pp. 2–3

Evaluation Metrics

  • The evaluation uses accuracy, macro-F1, and weighted F1, with entity-level F1 for named entity recognition.The setup refers readers to Table 1 for experiment details.Source: §3 · p. 3

4 Results and Analysis

Across five financial text-analysis tasks, ChatGPT and GPT-4 perform unevenly: GPT-4 leads on question answering and several classification tasks, while fine-tuned models retain advantages on structured extraction. Performance also depends on task difficulty and prompting setup.

4.1 Sentiment Analysis

  • On PhraseBank, few-shot GPT-4 is comparable to fine-tuned FinBert; on FiQA, it beats the other models, while TweetFinSent performance is about 72% accuracy and F1, similar to fine-tuned RoBERTa-Twitter.Removing emojis lowers ChatGPT and GPT-4 performance by 2–3 points on TweetFinSent.Source: §4.1 · pp. 3–4, Table 2 · p. 3, Table 3 · p. 3

4.2 Headline Classification

  • Fine-tuned BERT reaches 95% F1 on headline classification, 9% higher than 5-shot GPT-4, although GPT-4 outperforms ChatGPT, Prior LLMs, and BloombergGPT.The task is framed as six separate binary classification problems for the six commodity-headline categories.Source: §4.2 · p. 4, Table 5 · p. 4

4.3 Named Entity Recognition

  • GPT-4 is weaker than BloombergGPT and comparable to or worse than Prior LLMs on NER; CRF trained on similar FIN5 data performs best, while CRF trained on CoNLL performs poorly.The authors characterize LLM performance on this structured-prediction task as not ideal.Source: §4.3 · p. 4, Table 6 · p. 5

4.4 Relation Extraction

  • Fine-tuned Luke-base beats ChatGPT and GPT-4 by a notable margin on REFinD, while GPT-4 performs considerably better than ChatGPT.Source: §4.4 · p. 5, Table 7 · p. 5

4.5 Question Answering

  • GPT-4 leads the other LLMs on both QA datasets; on FinQA, zero-shot accuracy is 68.79% for GPT-4 versus 48.56% for ChatGPT.Few-shot and Chain-of-Thought prompting raise FinQA accuracy, and the best GPT-4 result exceeds fine-tuned FinQANet; model accuracy still falls below professionals’ ~90%.Source: §4.5 · pp. 5–6, Table 8 · p. 5

5 Discussions

The discussion finds that ChatGPT and GPT-4 lead most tested LLMs on financial tasks, while performance varies by task and prompting strategy. It recommends generalist LLMs for relatively simple tasks but favors fine-tuning for structured prediction and notes that financial QA remains below industry requirements.

Comparison over LLMs

  • GPT-4 beats ChatGPT by 10+% on straightforward tasks and brings 20–100% performance growth on challenging RE and QA tasks; both outperform other LLMs on almost all datasets except NER.Both also perform better on financial NLP tasks than BloombergGPT; the paper suggests their larger model size might explain this.Source: §5 · p. 6

Prompt Engineering Strategies

  • Chain-of-thought prompting yields 20–30% accuracy improvement over zero-shot and few-shot, while few-shot gives 1% to 4% performance boost over zero-shot across datasets.The authors argue both strategies should be considered first when applying LLMs to financial tasks.Source: §5 · p. 6

LLMs vs. Fine-tuning

  • Fine-tuned models remain strong on most tasks, outperforming LLMs on NER and RE, while LLMs have an advantage on the tested QA tasks.The tested QA reasoning is still basic for financial analysis, and simple mistakes persist despite some multi-step and numerical reasoning ability.Source: §5 · p. 6

Using LLMs in Financial Services

  • Generalist LLMs may suit relatively simple financial NLP tasks, but pretraining plus fine-tuning remains a leading option for structured prediction.Although ChatGPT and GPT-4 outperform other models on QA, they remain far from industry requirements; the authors call for significant improvement before use as trustworthy financial analyst agents.Source: §5 · p. 6

6 Conclusion

The paper presents an initial, broad study of generically trained LLMs on financial text analytics, finding that they outperform domain-specific fine-tuned models on some tasks but fall short on others. It calls for further investigation of financial applications, human decision-making integration, and robustness in high-stakes settings.

6 Conclusion

  • The study examines generically trained LLMs, including ChatGPT and GPT-4, across eight datasets from five task categories, but characterizes its effort as an initial study.The authors describe it as among the first studies of these models across a wide range of financial text analytics tasks.Source: §6 · p. 6
  • Generically trained LLMs outperform models fine-tuned with domain-specific data on some tasks but fall short on others, particularly when deeper semantics and structural analysis are needed.Source: §6 · p. 6
  • The authors call for more financial application tasks, integration of LLMs into human decision making, and investigation of model robustness in high-stakes financial tasks.Source: §6 · p. 6

Disclaimer

The disclaimer limits the document to informational purposes and disclaims responsibility for the information's completeness, accuracy, or reliability. It also states that the document is not investment research or advice, nor a solicitation or recommendation concerning securities or financial products.

Disclaimer

  • The document is prepared for informational purposes in part by JPMorgan Chase & Co.'s AI Research group and is not a product of its Research Department.Source: Disclaimer · p. 7
  • JPMorgan disclaims all liability and makes no warranty regarding the information's completeness, accuracy, or reliability.Source: Disclaimer · p. 7
  • The document is not investment research or investment advice, and is not a recommendation, offer, or solicitation to buy or sell a security or financial product or service.Source: Disclaimer · p. 7

A Details of the Related Work

The related work describes ChatGPT and its model lineage, prior evaluations of generalist LLMs, and domain-specific or adapted models. It notes that prior work reports both strong task performance and continuing failures, while BloombergGPT was trained for financial tasks using mixed financial and public data.

ChatGPT and Related Models

Domain-specific Models

  • BloombergGPT, a 50-billion-parameter model trained on over 700 billion tokens from proprietary financial archives and public datasets, is reported to perform strongly on financial tasks and competitively on some general NLP benchmarks.The related work also notes that only a handful of finance-specific LLMs were available.Source: A Details of the Related Work · p. 9
  • Other work adapts general-purpose models to domain tasks, including GPT-3 for legal work and T5 for finance; parameter-efficient tuning has also been studied for Banking77 intent detection.Source: A Details of the Related Work · p. 9

B Dataset Details

The appendix describes datasets and prompting choices spanning financial sentiment, information extraction, and question answering. It also reports where ChatGPT struggles and how performance changes with reasoning complexity and prompt format.

B Dataset Details

  • The sentiment datasets cover financial-news sentences, aspect sentiment in news and microblogs, and tweets labeled for sentiment and anticipated or realized stock gains or losses.Financial PhraseBank labels are positive, negative, or neutral; FiQA is converted from a continuous scale to three classes.Source: B Dataset Details · pp. 9–10
  • The Headlines dataset classifies 11,412 news headlines across six available tags, whereas the BloombergGPT evaluation used a nine-category version.Source: B Dataset Details · p. 10
  • The financial information-extraction datasets include NER over eight annotated SEC agreement documents and REFinD, with about 29,000 instances and 22 relations.NER uses FIN5 for training and FIN3 for testing; REFinD draws from publicly traded companies’ SEC filings.Source: B Dataset Details · p. 10
  • ConvFinQA extends FinQA to 3,892 earnings-report conversations with 14,115 questions, including follow-ups that depend on previous conversational context.The turn-based prompt appends each generated answer to the next question’s context.Source: B Dataset Details · pp. 10–11
  • On ConvFinQA, ChatGPT sometimes miscalculated final sums or failed to resolve conversational coreference, while GPT-4 faced the coreference issue much less.The example sum was $753 million + $785 million + $1,134 million, incorrectly returned as $3,672 million instead of $2,672 million.Source: B Dataset Details · p. 11
  • FinQA contains 8,281 expert-annotated question-answer pairs from S&P 500 earnings reports, requiring numerical reasoning over tables and unstructured text.Prompts combine pre-text, markdown-formatted tables, and post-text; a function-call format requests both reasoning steps and a final numerical answer.Source: B Dataset Details · p. 11
  • 32.14% versus FinQANet’s 22.78%: ChatGPT had higher accuracy on FinQA problems requiring more than two calculation steps.In a separate preliminary test, the models handled single-question prompts better than multi-tag binary classification, with drops on past information, future information, and asset comparison.Source: B Dataset Details · pp. 10–11

C Few Shots Experiments

Few-shot prompting improved ChatGPT on six datasets, but gains varied by task and shot count. Performance also fluctuated across repeated runs, indicating sensitivity to the selected examples and prompts.

C Few Shots Experiments

Loading 2305.05862v2…