PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Large Language Models in Finance (FinLLMs)

Jean Lee, Nicholas Stevens, Soyeon Caren Han, Minseok Song

arXiv:2402.02315v1cs.CLq-fin.GN

TL;DR

FinLLM research remains limited despite the broad capabilities of general-domain LLMs and their potential in finance. This survey synthesizes FinLLM evolution, techniques, benchmarks, advanced datasets, and challenges, finding differing strengths across tasks and identifying evaluation and implementation boundaries.

  • Problem

    FinLLM research remains limited compared with extensive general-domain LLM research, despite growing financial applications and potential.

  • Method

    The survey traces model evolution, compares five financial LM techniques, reviews six benchmark tasks, presents eight advanced tasks and datasets, and discusses opportunities and challenges.

  • Results

    Mixed-domain FinPLMs perform notably on sentiment analysis, text classification, and named entity recognition, while task-specific SOTA models outperform LLMs on question answering, stock movement prediction, and summarization.

  • Takeaways & Limitations

    The survey provides a broad research foundation through conventional and advanced benchmarks, datasets, and a synthesis of directions for developing FinLLMs.

  • Takeaways & Limitations

    Financial evaluation remains constrained by reliance on common NLP metrics and the need for expert assessment and finance-specific metrics for knowledge-driven tasks.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown remarkable capabilities across a wide variety of Natural Language Processing (NLP) tasks and have attracted attention from multiple domains, including financial services. Despite the extensive research into general-domain LLMs, and their immense potential in finance, Financial LLM (FinLLM) research remains limited. This survey provides a comprehensive overview of FinLLMs, including their history, techniques, performance, and opportunities and challenges. Firstly, we present a chronological overview of general-domain Pre-trained Language Models (PLMs) through to current FinLLMs, including the GPT-series, selected open-source LLMs, and financial LMs. Secondly, we compare five techniques used across financial PLMs and FinLLMs, including training methods, training data, and fine-tuning methods. Thirdly, we summarize the performance evaluations of six benchmark tasks and datasets. In addition, we provide eight advanced financial NLP tasks and datasets for developing more sophisticated FinLLMs. Finally, we discuss the opportunities and the challenges facing FinLLMs, such as hallucination, privacy, and efficiency. To support AI research in finance, we compile a collection of accessible datasets and evaluation benchmarks on GitHub.

1 Introduction

FinLLM research is still at an early stage despite growing financial NLP applications and the broader success of general-domain LLMs. This survey addresses the gap by tracing FinLLM evolution, comparing techniques, reviewing benchmarks, and discussing challenges.

  • FinLLM research remains at an early stage despite growing interest in applying NLP to financial tasks.
  • The survey presents a comprehensive evolution from general-domain language models to financial-domain models.
  • It compares five techniques across four financial PLMs and four financial LLMs, including training data and instruction fine-tuning.
  • It summarizes evaluations across six benchmark tasks and datasets while providing eight advanced financial NLP tasks and datasets.
  • The survey discusses FinLLM opportunities and challenges involving datasets, techniques, evaluation, implementation, and real-world applications.

2 Evolution Trends: from General to Finance

Language-model development progressed from Transformer-based general-domain models and scaling-driven capabilities toward open-source and finance-specific systems. Financial models now include four FinPLMs and four FinLLMs built from general-domain foundations, with techniques compared across the transition.

  • Transformer-based language models use discriminative or generative pre-training, and Figure 1 traces their evolution toward financial-domain models.
  • GPT scaling progressed from GPT-1 with 110M parameters to GPT-3 with 175B parameters, alongside the emergence of in-context learning.
  • ChatGPT combined GPT-3, Codex, and InstructGPT, while GPT-4 expanded capabilities toward multimodal data and large-scale models.
  • Open-source efforts included LLaMA, which encouraged diverse variants using instruction fine-tuning and Chain-of-Thought prompting, and BLOOM, trained across 46 natural languages and 13 programming languages.
  • The financial landscape contains four FinPLMs and four FinLLMs, with FinPLMs based on BERT or ELECTRA and FinLLMs based mainly on LLaMA, other open-source models, or BLOOM.

3 Techniques: from FinPLMs to FinLLMs

The survey compares three FinPLM techniques and two FinLLM techniques, covering pre-training data, training strategies, and fine-tuning. It also describes representative financial models built through continual, domain-specific, mixed-domain, and instruction-tuning approaches.

  • The survey reviews three techniques for four FinPLMs and two techniques for four FinLLMs, comparing pre-training, fine-tuning, and evaluation.These comparisons are summarized across financial language models in the survey’s model overview.
  • Continual Pre-training: Continual pre-training initializes a general-domain language model, trains it on financial-domain data, and fine-tunes it for financial NLP tasks.FinBERT-19 applies this sequence after initialization with a general-domain BERT model trained on 3.3B tokens.
  • Domain-Specific Pre-training: Domain-specific pre-training trains a model exclusively on unlabeled financial-domain text while preserving the original architecture and training objective.FinBERT-20 was pre-trained on a financial communication corpus containing 4.9B tokens and evaluated through sentiment-analysis fine-tuning.
  • Mixed-Domain Pre-training: Mixed-domain pre-training combines general-domain and financial-domain corpora so general language remains relevant while financial data adds domain adaptation.FinBERT-21 simultaneously used general and financial corpora and applied six self-supervised pre-training tasks.
  • Mixed-Domain Pre-training: FLANG uses financial keywords and phrases for masking while following ELECTRA’s training strategy, and introduces five financial NLP benchmark tasks.The benchmarks cover sentiment analysis, headline text classification, named entity recognition, structure boundary detection, and question answering.
  • Mixed-Domain LLM with Prompt Engineering: Mixed-domain LLMs combine general and financial corpora, while prompt engineering uses a frozen model for downstream tasks without weight updates.BloombergGPT used 345B general-domain tokens and 363B financial-domain tokens, with financial, internal, and general-purpose evaluations.
  • Instruction Tuning: Instruction tuning trains LLMs with explicit text instructions; FinMA, InvestLM, and FinGPT use financial instruction datasets or parameter-efficient adaptation.FinMA uses FIT with 136k samples, InvestLM uses a manually curated financial instruction dataset, and FinGPT uses LoRA with six open-source LLMs.

4 Evaluation: Benchmark Tasks and Datasets

The survey evaluates FinPLMs, FinLLMs, ChatGPT, GPT-4, and task-specific models across six financial NLP benchmark tasks, while identifying datasets for further research. Results vary by task: mixed-domain FinPLMs are competitive on SA, TC, and NER, whereas task-specific models remain stronger on QA, SMP, and summarization.

  • Evaluation scope: The evaluation reviews six financial NLP benchmark tasks and datasets, comparing FinPLMs, FinLLMs, ChatGPT, GPT-4, and task-specific SOTA models.The reviewed results come from original research or analysis research, with SOTA results from task-specific models.
  • Sentiment Analysis: Sentiment analysis uses Financial PhraseBank and FiQA-SA, with additional evaluation datasets including SemEval-2017 Task 5 and StockEmotions.Financial PhraseBank contains 4,845 annotated English financial news articles, while SemEval-2017 contains 4,157 sentences and StockEmotions contains 10,000 microblog sentences with binary sentiment and 12 emotion classes.
  • Text Classification: Text classification includes the gold news Headline dataset, which contains 11,412 headlines labeled across nine binary categories such as price up and price down.The survey also identifies FedNLP, FOMC, and Banking77 as additional financial text-classification datasets.
  • Question Answering: GPT-4 with zero-shot prompting achieves 69%-76% EM Accuracy on FinQA and ConvFinQA, below human experts at Avg. 90% EM Accuracy, while BloombergGPT reaches 43% EM Accuracy.FinQA and ConvFinQA assess numerical reasoning in financial question answering.
  • Stock Movement Prediction: GPT-4 with zero-shot prompting averages 54% Accuracy across StockNet, CIKM18, and BigData22, exceeding FinMA’s 52% but trailing the SOTA model’s 58%.The survey notes that financial evaluation should also consider metrics such as the Sharpe ratio and backtesting results.
  • Cross-task findings: Across the six benchmarks, mixed-domain FinPLMs are noteworthy on SA, TC, and NER, while task-specific SOTA models outperform all LLMs on QA, SMP, and summarization.GPT-4 performs impressively across the benchmarks except summarization, suggesting that scaling alone may not yield optimal financial performance.

5 Advanced Financial NLP Tasks and Datasets

The survey expands evaluation beyond six conventional benchmarks by presenting eight advanced financial NLP tasks and associated datasets. These tasks cover structured information extraction, causality, numerical reasoning, document structure, multimodal understanding, translation, and market forecasting.

  • Motivation: The eight advanced benchmarks address complex financial NLP capabilities not covered by the current six benchmark tasks.The survey presents associated datasets for each advanced task.
  • Information Extraction: Relation Extraction uses FinRED, a finance-domain dataset from financial news and earnings call transcripts with 29 relation tags.Example relations include “owned by.”
  • Event and Causality Analysis: Event Detection uses EDT, containing 9,721 token-labeled news articles and 303,893 additional articles with minute-level timestamps and stock-price labels.The dataset covers 11 types of corporate events.
  • Event and Causality Analysis: Causality Detection uses FinCausal20 to detect causal schemes and identify cause-and-effect sentences in financial text.The dataset comes from the Workshop on Financial Narrative Processing.
  • Numerical and Document Understanding: Numerical Reasoning draws on FiNER-139, FinQA, and ConvFinQA to identify numbers and operators, perform calculations, and interpret financial context.FiNER-139, FinQA, and ConvFinQA were also introduced for NER or QA.
  • Numerical and Document Understanding: Structure Recognition uses FinTabNet to detect document boundaries and table structures in unstructured PDF earnings reports from S&P 500 companies.The task also concerns logical relationships between tables, surrounding content, and table cells.
  • Multimodal, Translation, and Forecasting: Multimodal understanding, machine translation, and market forecasting are supported by datasets including MAEC, MONOPOLY, MINDS-14, MultiFin, StockEmotions, and EDT.These datasets combine modalities or extend financial prediction beyond stock-movement classification.

6 Opportunities and Challenges

The survey identifies future FinLLM directions across datasets, techniques, evaluation, implementation, and applications. It emphasizes higher-quality and multimodal data, privacy-aware methods, expert-centered evaluation, cost-performance trade-offs, and non-technical deployment barriers.

  • Datasets: Developing sophisticated FinLLMs requires high-quality financial data in diverse formats and instruction-finetuned datasets for specific financial NLP tasks.The survey also highlights financial multimodal datasets as an important direction.
  • Techniques: Retrieval Augmented Generation can retrieve private external knowledge, improve answer reliability through cross-referencing, and use internal nontrainable data without retraining the model.The technique is presented as a response to privacy, security, and trust challenges.
  • Evaluation: Evaluation remains constrained by the need for financial-expert validation, financial metrics beyond F1-score or Accuracy, and feedback for model alignment.The survey proposes advanced benchmarks to assess whether FinLLMs can serve as general financial problem-solvers while considering cost and performance.
  • Implementation: Selecting FinLLMs and techniques involves a cost-performance trade-off, and general-domain prompting or task-specific models may be more practical than building FinLLMs for some tasks.Implementation also requires LLMOps skills such as PEFT and CI/CD monitoring systems.
  • Applications: Real-world financial applications face business, industry, privacy, accountability, ethics, and communication barriers between financial and AI experts.The survey encourages sharing use cases and pursuing applications such as report generation and document understanding.

7 Conclusion

The survey synthesizes FinLLM evolution, techniques, conventional and advanced benchmarks, datasets, and future opportunities and challenges. It presents this broad view and dataset collection as resources for computer science and finance research.

  • Conclusion: The survey investigates FinLLM evolution from general-domain language models, compares FinPLM and FinLLM techniques, and presents six conventional and eight advanced benchmarks with datasets.It also discusses opportunities and challenges for developing advanced FinLLMs.
  • Conclusion: The survey’s broad perspective and extensive dataset collection are intended to benefit both computer science and finance research communities.The collection is positioned as a resource for more advanced evaluation and future FinLLM directions.
Loading 2402.02315v1…