PaperlayerView exact revision on arXiv

Source-linked AI summary

Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models

Boyu Zhang, Hongyang Yang, Xiao-Yang Liu

arXiv:2306.12659v1cs.CLcs.LGq-fin.STq-fin.TR

TL;DR

Financial sentiment analysis is important for understanding and forecasting market movements, but models can struggle with numerical values and sentiment when context is absent or ambiguous. The method instruction-tunes a pre-trained LLM by transforming classification-based financial sentiment data into a generation task. Instruct-FinGPT surpasses general-purpose LLMs and state-of-the-art supervised models on benchmarks despite using little instruction data and training resources.

  • Problem

    Financial sentiment analysis is important for understanding and forecasting market movements, but models can struggle with numerical values and sentiment when context is absent or ambiguous.

    Source: §1 · p. 1

  • Method

    The method instruction-tunes a pre-trained LLM by transforming classification-based financial sentiment data into a generation task.

    Source: §1 · p. 1

  • Results

    Instruct-FinGPT surpasses general-purpose LLMs and state-of-the-art supervised models on benchmarks despite using little instruction data and training resources.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The authors conclude that instruction tuning can yield superior financial sentiment performance with little task-specific data and motivate further research on LLMs across financial tasks.

    Source: §5 · p. 6

    Because of existing limitations, requests used the GPT-3.5 API; GPT-4.0 was unavailable for programmatic access and accessible only via a web interface.

    Source: §4.3 · p. 4

Abstract

from arXiv · show

Sentiment analysis is a vital tool for uncovering insights from financial articles, news, and social media, shaping our understanding of market movements. Despite the impressive capabilities of large language models (LLMs) in financial natural language processing (NLP), they still struggle with accurately interpreting numerical values and grasping financial context, limiting their effectiveness in predicting financial sentiment. In this paper, we introduce a simple yet effective instruction tuning approach to address these issues. By transforming a small portion of supervised financial sentiment analysis data into instruction data and fine-tuning a general-purpose LLM with this method, we achieve remarkable advancements in financial sentiment analysis. In the experiment, our approach outperforms state-of-the-art supervised sentiment analysis models, as well as widely used LLMs like ChatGPT and LLaMAs, particularly in scenarios where numerical understanding and contextual comprehension are vital.

1 Introduction

Financial sentiment analysis matters for understanding and forecasting market movements, but models can struggle with numerical values and sentiment that depends on missing or ambiguous context. The paper proposes instruction-tuning a general-purpose LLM to address these challenges and reports benchmark performance surpassing general-purpose LLMs and state-of-the-art supervised models with little instruction data and training resources.

1 Introduction

  • Financial sentiment models face difficulties with numerical values, implicit context, financial jargon, multilingual data, temporal dependency, limited labels, and noisy social media.Financial sentiment analysis is framed as important for understanding and forecasting market movements.Source: §1 · p. 1
  • The study asks how to address numerical sensitivity and what role contextual understanding plays in financial sentiment analysis.The questions respond to LLM difficulties interpreting numbers and sentiment when context is absent or ambiguous.Source: §1 · p. 1
  • Instruct-FinGPT instruction-tunes a pre-trained LLaMA and converts classification-based sentiment analysis into a generation task.The approach aims to reduce fine-tuning-data requirements while leveraging LLM contextual understanding and numerical sensitivity.Source: §1 · p. 1
  • The proposed Instruct-FinGPT surpasses general-purpose LLMs and state-of-the-art supervised models on benchmarks despite using little instruction data and training resources.It is an instruction-tuned FinGPT model for financial sentiment analysis.Source: §1 · p. 1
  • The paper highlights contextual understanding, including cases where context is missing or vague, as a focus for improving financial sentiment analysis.It also identifies numerical sensitivity as a critical issue often neglected by existing models.Source: §1 · p. 1

2 Related Work

Prior financial sentiment work spans lexicon-based, machine-learning, and deep-learning methods, with recurring challenges around numeric sensitivity, context, and labeled-data needs. The paper distinguishes its approach by leveraging general-purpose LLM knowledge and reasoning through instruction tuning.

2 Related Work

  • Prior financial sentiment methods range from lexicon-based techniques to machine learning and deep learning, including work on stock-movement prediction from financial news.The cited ensemble work does not extensively address numerical sensitivity or contextual understanding.Source: §2 · p. 2
  • BERT and its derivatives offer context understanding but typically need substantial labeled data, while FinBERT remains insensitive to numbers and can struggle when needed context is missing.FLANG presents financial assessment benchmarks across five financial NLP tasks and incorporates conventional benchmarks.Source: §2 · p. 2
  • BloombergGPT is proprietary and resource-intensive, whereas the paper reports effectiveness with a smaller corpus and training estimated at less than $300.The passage contrasts BloombergGPT’s 1.3M GPU hours and estimated $5M training cost with the paper’s approach.Source: §2 · p. 2
  • The paper’s approach centers on instruction tuning and leveraging LLMs’ general knowledge and reasoning for financial sentiment analysis.The authors distinguish this focus from prior methods and report experimental effectiveness.Source: §2 · p. 2

3 Our Method

The method adapts a general-purpose LLM to financial sentiment analysis by converting classification examples into instruction-formatted text, fine-tuning on those examples, and mapping generated text back to sentiment labels.

3 Our Method

  • Instruct-FinGPT adapts a pre-trained LLM to financial sentiment analysis by turning the task into text generation and instruction-tuning the model.Generated outputs are mapped to sentiment labels during inference.Source: §3 · p. 2

3.1 Instruction Tuning

Formatting Financial Sentiment Analysis Dataset into Instruction Tuning Dataset

  • Each classification example is reformatted by pairing the input and sentiment label with one randomly selected human-written instruction from a set of 10.The template is “Human: [instruction] + [input], Assistant: [output].”Source: Formatting Financial Sentiment… · p. 2

Instruction Tuning LLaMA-7B

  • LLaMA-7B is fine-tuned on the formatted instances with supervised sequence-to-sequence loss.The paper uses LLaMA-7B as an example to validate its instruction-tuning approach.Source: Instruction Tuning LLaMA-7B · pp. 2–3

Mapping the Generated Outputs into Sentiment Labels

  • Generated text containing “positive,” “negative,” or “neutral” is mapped to that label; any output containing none of them is mapped to neutral.The mapping addresses the possibility that the autoregressive model produces freestyle text.Source: Mapping the Generated Outputs into… · p. 3

3.2 Comparison Between LLMs and FinBERT for Sentiment Analysis

  • The comparison frames LLMs as stronger in contextual understanding and numerical sensitivity, while contrasting decoder-only generation with FinBERT’s encoder and separate classifier.The passages describe the LLM as generating the class label directly and FinBERT as requiring a task-specific classifier.Source: §3.2 · p. 3

4 Performance Evaluation

The evaluation tests Instruct-FinGPT against sentiment-analysis models and general-purpose LLMs on general sentiment, numerical sensitivity, contextual understanding, and zero-shot generalization. The reported results favor instruction-tuned LLaMA-7B across the three benchmark datasets and on the full Financial PhraseBank comparison.

Training Datasets

  • The training corpus combines 9,540 English Twitter financial-news samples with 961 FiQA samples, for 10,501 samples total.Both datasets use three sentiment labels, with Twitter labels Bearish, Bullish, and Neutral, and FiQA labels positive, neutral, and negative.Source: §4.1 · p. 3, Training Datasets · p. 3

Testing Datasets

  • The test material includes 2,390 Twitter validation samples, a 117-sample numerical subset, a 20-sample contextual subset, and 4,840 Financial PhraseBank samples.The numerical subset excludes strong indication words and requires at least two financial-indicator values; the contextual subset lacks essential context for sentiment prediction.Source: Testing Datasets · p. 4

4.2 Model Training

  • Instruct-FinGPT-7B is initialized from LLaMA-7B and instruction-tuned for 10 epochs with AdamW, batch size 32, and a 1e−5 initial learning rate.Training uses 8 A100 (40GB) GPUs and takes 58 minutes.Source: §4.2 · p. 4

4.3 Baseline Models

  • On a benchmark financial sentiment-analysis dataset, the model is evaluated against FinBERT; primary metrics are accuracy, the proportion of correct predictions, and F1-score, the harmonic mean of precision and recall.For API requests, GPT-3.5 was used because GPT-4.0 was unavailable programmatically and could only be accessed through a web interface.Source: §4.4 · p. 4, §4.3 · p. 4

4.4 Evaluation and Analysis

  • Instruct-FinGPT-7B consistently outperforms FinBERT and LLaMA-7B in accuracy and F1 across all three datasets.The datasets are Twitter financial news sentiment validation, numerical, and contextual.Source: Table 1 · p. 4, §4.4 · p. 5
  • In the loss-decrease example, Instruct-FinGPT and ChatGPT 4.0 identify a decrease from 2.2 million to 0.3 million as positive, while FinBERT fails.Other examples report that all models except FinBERT identify increased EPS and consumer credit exceeding expectations as positive; the Estee Lauder and FactSet consensus statement is neutral.Source: §4.4 · p. 5
  • In the contextual examples, Instruct-FinGPT identifies negative, neutral, and positive sentiment in three cases, outperforming FinBERT and ChatGPT overall.All models fail on an example where an opening price above an IPO price does not establish that the stock is rising from its current market price.Source: §4.4 · p. 5
  • Instruction-tuned LLaMA-7B achieves the highest accuracy and F1 on the full Financial PhraseBank dataset among the three compared models.FinBERT is excluded from this comparison because it was trained on Financial PhraseBank.Source: §4.4 · p. 6

5 Conclusion and Future Work

The paper presents instruction tuning of an LLM as an approach to financial sentiment analysis, reporting superior performance with a small amount of task-specific data. It identifies LLMs’ potential for broader financial tasks as a direction for future research.

5 Conclusion and Future Work

  • Instruction tuning an LLM can yield superior financial sentiment analysis performance with a small amount of task-specific data.Source: §5 · p. 6
  • The approach harnesses the general knowledge and reasoning capabilities of LLMs for financial sentiment analysis.Source: §5 · p. 6
  • The findings point to future research on LLMs’ potential across a broad range of financial tasks.Source: §5 · p. 6
Loading 2306.12659v1…