PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models

Boyu Zhang, Hongyang Yang, Tianyu Zhou, Ali Babar, Xiao-Yang Liu

arXiv:2310.04027v2cs.CLq-fin.STq-fin.TR

TL;DR

Financial sentiment analysis is hindered by limited traditional models, mismatched LLM training objectives, and insufficient context in brief financial text. The paper combines instruction-tuned LLMs with retrieval from reliable external sources, achieving 15% to 48% gains in accuracy and F1 score against traditional models and general-purpose LLMs. The authors identify reliance on textual similarity as a limitation because it omits relevant macroeconomic timing and enterprise-level information.

  • Problem

    Financial sentiment analysis is important for market and investment decisions, but existing models face limited capacity, objective mismatch, and insufficient context in brief financial text.

  • Method

    The framework combines instruction-finetuned LLMs with retrieval augmentation that supplies relevant financial background context from external sources.

  • Results

    15% to 48% performance gain in accuracy and F1 score is reported against traditional sentiment-analysis models and general-purpose LLMs.

  • Takeaways & Limitations

    Instruction tuning aligns LLM outputs with financial sentiment labels, while retrieved context supports more informed and nuanced predictions.

  • Takeaways & Limitations

    The approach relies exclusively on textual similarity for retrieval and omits macroeconomic timing and enterprise financial or operational information.

Abstract

from arXiv · show

Financial sentiment analysis is critical for valuation and investment decision-making. Traditional NLP models, however, are limited by their parameter size and the scope of their training datasets, which hampers their generalization capabilities and effectiveness in this field. Recently, Large Language Models (LLMs) pre-trained on extensive corpora have demonstrated superior performance across various NLP tasks due to their commendable zero-shot abilities. Yet, directly applying LLMs to financial sentiment analysis presents challenges: The discrepancy between the pre-training objective of LLMs and predicting the sentiment label can compromise their predictive performance. Furthermore, the succinct nature of financial news, often devoid of sufficient context, can significantly diminish the reliability of LLMs' sentiment analysis. To address these challenges, we introduce a retrieval-augmented LLMs framework for financial sentiment analysis. This framework includes an instruction-tuned LLMs module, which ensures LLMs behave as predictors of sentiment labels, and a retrieval-augmentation module which retrieves additional context from reliable external sources. Benchmarked against traditional models and LLMs like ChatGPT and LLaMA, our approach achieves 15\% to 48\% performance gain in accuracy and F1 score.

I. INTRODUCTION

Financial sentiment analysis supports market and investment decisions, but traditional models and directly applied LLMs face important limitations. The paper addresses these challenges with instruction tuning and retrieval augmentation, reporting substantial benchmark gains.

  • Financial sentiment analysis extracts affective information from financial documents, news, and social media to support market analysis and investment decisions.It can help traders and financial institutions manage risks and identify investment opportunities.
  • Traditional NLP models often struggle with complex financial news because of limited parameters and training-corpus scale.These limitations can produce suboptimal sentiment-analysis outcomes.
  • LLMs face a mismatch between their pre-training objective and the requirement to consistently generate financial sentiment labels.This mismatch can prevent outputs from aligning with the expected labels.
  • The proposed framework combines instruction-finetuned LLMs with retrieval augmentation that adds background context to brief financial statements.Retrieved context is passed to the instruction-tuned model for sentiment prediction.
  • 15% to 48% performance gain in accuracy and F1 score is reported against traditional models and general-purpose LLMs.The evaluations use established financial sentiment-analysis benchmarks.

II. BACKGROUND AND RELATED WORK

Prior financial sentiment-analysis methods include fine-tuned deep models and general-purpose or financial-domain LLMs. Their limitations include difficulty with complex or context-poor financial text and inconsistent label generation.

  • Early approaches fine-tuned pre-trained models on financial sentiment datasets but struggled with complex news, numerical information, and missing background context.
  • LLMs provide in-context learning, chain-of-thought reasoning, and zero-shot prediction capabilities for NLP tasks.
  • Financial-domain LLMs such as BloombergGPT and FinGPT can have difficulty generating expected sentiment labels because causal-language-modeling objectives differ from sentiment classification.
  • Brief news flashes and tweets often lack sufficient background information, creating a challenge for financial sentiment analysis with LLMs.
  • Instruction tuning fine-tunes pre-trained LLMs on formatted instruction-and-response instances to guide task-specific behavior.

C. Retrieval Augmented Generation

Retrieval-augmented generation combines external-context retrieval with LLM generation. In this framework, retrieved financial information is combined with the original query before an instruction-tuned LLM predicts sentiment.

  • RAG retrieves relevant documents from external knowledge sources and combines them with the input prompt before generation.This uses both the LLM’s parametric memory and retrieved nonparametric memory.
  • The framework contains an instruction-tuned LLM module and an RAG module.
  • Instruction tuning adapts an open-source pre-trained LLM to predict financial sentiment labels from news or tweets.The adaptation uses a task-specific instruction-following dataset.
  • The RAG module retrieves pertinent background information from external sources related to the input query.
  • Retrieved context is combined with the original query, and the instruction-tuned LLM generates the sentiment prediction from the augmented query.The design supplies missing background knowledge to the LLM.

B. Instruction-tuned LLMs

Instruction tuning converts financial sentiment data into instruction-following examples, fine-tunes LLMs to produce sentiment labels, and maps outputs to predefined classes. The training uses a causal language-modeling objective to maximize expected-label generation.

  • B. Instruction-tuned LLMs: Instruction tuning can align LLM behavior with user instructions using limited instruction-following data.
  • B. Instruction-tuned LLMs: The financial sentiment instruction-tuning pipeline constructs paired instructions and responses, fine-tunes the LLM, and maps generated outputs to predefined sentiment classes.
  • 1) Formatting Financial Sentiment Instruction Following Dataset:: Existing supervised financial sentiment datasets can be converted into instruction-following datasets as a lower-cost alternative to manual expert labeling.
  • 1) Formatting Financial Sentiment Instruction Following Dataset:: Ten human-written instructions are randomly paired with original inputs and outputs in a Human–Assistant format.
  • 2) Instruction Tuning:: Instruction tuning tokenizes text with byte-pair encoding before fine-tuning the LLM.
  • 2) Instruction Tuning:: The causal language-modeling objective maximizes the likelihood of predicting each next token from preceding context.
  • 2) Instruction Tuning:: The negative log-likelihood objective trains the model to maximize the probability of generating the expected financial sentiment label given an instruction.
  • 2) Instruction Tuning:: Gradient-based optimization minimizes the objective, with gradients computed through the sequence by backpropagation through time.

3) Mapping the Generated Outputs into Sentiment Classes:

The framework maps free-form autoregressive LLM outputs to one of three financial sentiment classes for evaluation. It also uses retrieval-augmented context before the instruction-tuned model generates its prediction.

  • 3) Mapping the Generated Outputs into Sentiment Classes:: The system sequentially checks generated text for “negative”, “neutral”, or “positive” and maps the first matching term to that sentiment label.Outputs containing none of these terms are classified as neutral.
  • 3) Mapping the Generated Outputs into Sentiment Classes:: RAG injects external financial knowledge into LLMs to improve response generation accuracy.
  • 3) Mapping the Generated Outputs into Sentiment Classes:: The retrieved context is combined with the original query and passed to the instruction-tuned LLM for the final prediction.

1) Setup External Knowledge Sources:

The retrieval module prioritizes reliable, relevant financial context from multiple external sources rather than random web results. It uses multi-source querying followed by similarity filtering before combining context with the input query.

  • 1) Setup External Knowledge Sources:: The framework considers financial news outlets, research publication platforms, and social media as external knowledge sources.
  • 1) Setup External Knowledge Sources:: News outlets such as Bloomberg, Yahoo Finance, Reuters, CNBC, and Market Screener provide consistent, verified, and often early financial reporting.
  • 1) Setup External Knowledge Sources:: Research sources include institutional publishers such as Goldman Sachs and Citi, alongside crowd-based platforms such as Seeking Alpha.
  • 1) Setup External Knowledge Sources:: Social media provides real-time financial information but may contain volatile and unverified content requiring cross-referencing.
  • 1) Setup External Knowledge Sources:: The retrieval pipeline preprocesses headlines or tweets, queries source APIs, applies similarity filtering, and combines selected context with the original query.Time information in a news item can restrict searches to a specific time range.
  • 1) Setup External Knowledge Sources:: The similarity stage uses a modified overlap coefficient and retains contexts with similarity higher than 0.8 to the input query.The coefficient measures overlap between financially relevant token sets in the query and context.

IV. PERFORMANCE EVALUATION

The evaluation compares the proposed approach with traditional sentiment models and general-purpose LLMs. Its retrieval algorithm searches documents and retains documents and syntactic units exceeding overlap thresholds.

  • IV. PERFORMANCE EVALUATION: The experiments evaluate instruction fine-tuning and retrieval-augmented generation against state-of-the-art sentiment analysis models.
  • IV. PERFORMANCE EVALUATION: The retrieval algorithm searches for documents matching the query, then initializes an empty context collection.
  • IV. PERFORMANCE EVALUATION: Documents are retained when their overlap with the query exceeds 0.8.
  • IV. PERFORMANCE EVALUATION: Each retained document is split into syntactic units, and units are concatenated into the context when their overlap with the query exceeds 0.7.
  • IV. PERFORMANCE EVALUATION: With only a small amount of instruction-following data, the model consistently outperforms other baselines, with further improvement from the RAG module.

A. Datasets

The study trains on combined Twitter Financial News and FiQA data, formats datasets as instruction-following examples, and evaluates on multiple financial sentiment datasets. The sources vary in size, labels, and annotation characteristics.

  • 1) Training Datasets:: The training collection combines the Twitter Financial News and FiQA datasets into 10,501 samples.
  • 1) Training Datasets:: The Twitter financial news training dataset contains 9,540 samples labeled Bearish, Bullish, or Neutral.
  • 1) Training Datasets:: FiQA contains 961 samples labeled positive, neutral, or negative.
  • 2) Testing Datasets: •: Twitter Val is a 2,388-sample validation split used to assess financial sentiment prediction from social media.
  • 2) Testing Datasets: •: Financial PhraseBank contains 4,840 financial-news samples annotated by 16 finance and business experts.
  • 2) Testing Datasets: •: The datasets are formatted as instruction-following datasets before training and testing.

B. Model Training

The model is initialized with Llama-7B and compared with BloombergGPT, ChatGPT, Llama-7B, ChatGLM2-6B, and FinBERT as evaluation baselines.

  • Training Configuration: The model is initialized with Llama-7B and instruction-tuned for 10 epochs.Training uses AdamW with batch size 32, learning rate 1e−5, weight decay 0.1, and 512-token maximum inputs on 8×A100 GPUs.
  • Baseline Models: BloombergGPT is a closed-source 50-billion-parameter model trained on broad financial data.Its reported Financial PhraseBank performance is used directly.
  • Baseline Models: ChatGPT is evaluated through the GPT-4.0 API using prepared instructions and direct sentiment outputs.The procedure includes API setup, data preparation, requests, and response interpretation.
  • Baseline Models: Llama-7B is an open-source Meta model whose training corpus is predominantly English.The study acquires the model from Meta.
  • Baseline Models: ChatGLM2-6B is an open-source bilingual model from Tsinghua University, while FinBERT is fine-tuned from pretrained BERT for financial sentiment analysis.Both models are accessed through the Hugging Face Model Hub.

D. Evaluation and Analysis

The evaluation tests instruction tuning against established baselines on FPB and Twitter Val, using accuracy and F1-score; FinBERT is excluded from FPB because it trained on that dataset.

  • Evaluation Design: The study compares the instruction-tuned model with FinBERT and general-purpose LLMs, then evaluates models with and without RAG.This design separately assesses instruction tuning and retrieval augmentation.
  • Evaluation Metrics: Accuracy measures correct predictions, while F1-score is the harmonic mean of precision and recall.These are the reported performance metrics.
  • Assessment of Instruction Finetuning: Instruction tuning is evaluated on Financial PhraseBank and Twitter Val against all baseline models.FinBERT is excluded from FPB because it uses the exact dataset for training.
  • Assessment of Instruction Finetuning: Table I compares the proposed model and baselines across the FPB and Twitter Val datasets.The table is used to assess instruction-finetuning performance.
  • Assessment of Instruction Finetuning: The instruction-tuned Llama-7B achieves the highest accuracy and F1 score among the compared models.The reported comparison includes ChatGPT 4.0 and the original Llama-7B.

3) Performance of RAG Module:

The RAG evaluation examines retrieval augmentation on Twitter Val and illustrates how external context resolves an ambiguous financial statement, while the paper notes a textual-similarity limitation.

  • Performance of RAG Module: RAG is evaluated on Twitter Val for both the instruction-tuned model and ChatGPT 4.0.The experiment tests whether retrieved context improves LLM sentiment analysis.
  • Performance of RAG Module: Table II reports experimental results on the Twitter Val dataset.It presents the RAG-module evaluation described in the experiment.
  • Performance of RAG Module: RAG universally improves the evaluated LLMs’ performance, and the proposed model with RAG achieves the best performance.The retrieved context provides additional information for more accurate predictions.
  • Case Study: A case study shows RAG correcting a neutral misclassification to positive by using Seeking Alpha context to interpret “shakes off” as a rating upgrade.The example demonstrates improved comprehension of ambiguous financial language.
  • Conclusion: The framework enriches LLM inputs with external knowledge to support more nuanced predictions.This conclusion connects retrieval with increased information depth and context.
  • Limitations and Future Work: The approach relies exclusively on textual similarity for retrieval, omitting macroeconomic timing and microeconomic enterprise information.The authors identify combining these economic dimensions with textual data as future work.
Loading 2310.04027v2…