PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMFactor: Extracting Profitable Factors through Prompts for Explainable Stock Movement Prediction

Meiyun Wang, Kiyoshi Izumi, Hiroki Sakaji

arXiv:2406.10811v1cs.CLcs.AIcs.CE

TL;DR

Keyphrases and sentiment words have limitations as stock-movement signals: keyphrases may not directly correlate with prices, while sentiment is less clear to interpret. LLMFactor uses prompting to extract stock-related factors, and its factor-based methods outperform the other evaluated approaches across four datasets.

  • Problem

    Keyphrases may not directly correlate with stock prices, while sentiment is associated with prices but lacks clarity for human interpretation.

    Source: §1 · p. 2

  • Method

    LLMFactor uses SKGP’s fill-in-the-blank prompting for background knowledge and asks LLMs to identify stock-price factors from news.

    Source: §1 · p. 2, §3.2.2 · p. 4

  • Results

    Factor-based methods outperform the other evaluated approaches across four datasets, and LLMFactor improves MCC over SOTAs by 2.9%, 0.4%, 11%, and 4.8%, respectively.

    Source: §4.5 · p. 6

  • Takeaways & Limitations

    The extracted factors offer news, human, and stock-price explainability.

    Source: §1 · p. 2

    LLM response variability makes precise replication of the experimental results challenging.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recently, Large Language Models (LLMs) have attracted significant attention for their exceptional performance across a broad range of tasks, particularly in text analysis. However, the finance sector presents a distinct challenge due to its dependence on time-series data for complex forecasting tasks. In this study, we introduce a novel framework called LLMFactor, which employs Sequential Knowledge-Guided Prompting (SKGP) to identify factors that influence stock movements using LLMs. Unlike previous methods that relied on keyphrases or sentiment analysis, this approach focuses on extracting factors more directly related to stock market dynamics, providing clear explanations for complex temporal changes. Our framework directs the LLMs to create background knowledge through a fill-in-the-blank strategy and then discerns potential factors affecting stock prices from related news. Guided by background knowledge and identified factors, we leverage historical stock prices in textual format to predict stock movement. An extensive evaluation of the LLMFactor framework across four benchmark datasets from both the U.S. and Chinese stock markets demonstrates its superiority over existing state-of-the-art methods and its effectiveness in financial time-series forecasting.

1 Introduction

LLMFactor uses Sequential Knowledge-Guided Prompting to extract stock-related factors and use them to explain and predict stock-price trends. The introduction presents factors as a more interpretable alternative to keyphrases and sentiment words.

1 Introduction

  • Stock-trend forecasting matters because historical data can shape trading strategies and help identify buying or selling opportunities.The introduction also notes that researchers have explored news, company and industry relationships, sentiment, and keyphrases for prediction.Source: §1 · p. 1
  • Unlike keyphrases, which may not correlate directly with stock prices, and sentiment, which is less clear to interpret, factors explain news, people’s understanding, and stock-price movements.The comparison is illustrated in Figure 1.Source: Figure 1 · p. 1, §1 · p. 2
  • Across four datasets, LLMFactor yields superior predictive results and clarifies the principles behind its predictions.The reported results also describe factors as a way to present market changes dynamically in an accessible, understandable format.Source: §1 · p. 2
  • LLMFactor uses SKGP to elicit background knowledge, identify reliable factors from news, and predict stock-price trends using text-formatted time-series data.The introduction describes SKGP as beginning with a fill-in-the-blank technique.Source: §1 · p. 2

2 Related Work

Prior work uses textual data and prompting for stock prediction, but the approaches described here variously treat text as a whole, focus on granular signals, or adapt LLMs to forecasting. The paper situates SKGP among prompting strategies intended to improve financial forecasts.

2.1 Stock Movement Prediction using Textual Data

  • Stock-text prediction spans whole-text approaches and granular signals, including corporate events, media sentiment, and aggregated expert opinions.Xu and Cohen (2018) use tweets and historical prices, Luo et al. (2023) model multimodality and stock correlations, Zhou et al. (2021) identify corporate events, Wan et al. (2021) report an association between strong media sentiment and market return, and Wang et al. (2020) include aggregated expert opinions.Source: §2.1 · p. 2

2.2 Time-series Forecasting with LLMs

  • LLM forecasting work adapts numerical inputs to text prompts and uses summaries or keyphrases for financial series, while the latter prompts can yield less detailed responses.The subsection notes that LLMs’ transformer architecture is not intrinsically designed for time-series analysis.Source: §2.2 · pp. 2–3, Figure 2 · p. 3

2.3 Prompt Engineering

  • Prompting strategies include chain-of-thought, generated knowledge, and retrieval-augmented generation; the paper proposes SKGP to enhance financial forecast accuracy.The cited approaches use intermediate reasoning steps, model-generated supplementary knowledge, and retrieve-augment-generate flows, respectively.Source: §2.3 · p. 3

3 LLMFactor

LLMFactor uses Sequential Knowledge-Guided Prompting (SKGP) to derive factors from stock-related news and background knowledge, then combines them with textualized price history to predict and explain stock movements.

3.1 Task Definition

  • The task is binary classification: predict the next rise or fall from the target date, related news, and a window of prior stock movements.A movement is encoded as 1 for a rise and 0 for a fall.Source: §3.1 · p. 3

3.2.1 Matching and Acquiring News

  • SKGP first matches a stock with related news and uses a fill-in-the-blank prompt to elicit its relationship with matched companies as background knowledge.Company relationships can inform interpretation; for example, a competitor’s information could negatively affect the target stock.Source: §3.2.1 · pp. 3–4

3.2.2 Generating Factors that May Affect Stock Prices

  • The LLM identifies potentially influential factors by analyzing news, and its factors may summarize news implications rather than repeat words appearing in the article.The section’s example, “Nvidia stock gain in January,” links a past gain to a possible positive influence on future price.Source: §3.2.2 · p. 4

3.2.3 Predicting Stock Price Movements

  • For prediction, LLMFactor combines background relations and factors with a textual template of dated past rises and falls, then prompts for the next direction and rationale.The historical sequence is converted into “rose” or “fell” entries; the output is “rise” or “fall” with a rationale.Source: §3.2.3 · p. 4

3.3 Factor Analysis in the Stock Market

  • The Nvidia example shows SKGP factors explaining an upward movement through past stock gains, product announcements, and EV makers selecting Nvidia Drive Thor.The section presents these factors as concise explanations of the observed price trend.Source: §3.3 · pp. 4–5

4 Experiments

The experiments frame evaluation around prediction accuracy and explanatory insight, using four datasets, two metrics, and baselines spanning keyphrase, sentiment, and time-series approaches.

4 Experiments

  • The experiments ask whether LLMFactor improves stock-movement prediction accuracy and provides more insightful explanations.Source: §4 · p. 5

4.1 Datasets

  • The evaluation covers StockNet, CMIN-US, CMIN-CN, and EDT; the first three pair stock-related tweets with prices, while EDT centers on news articles linked to stocks and prices.Source: §4.1 · p. 5

4.2 Evaluation Metrics

  • Accuracy (ACC) and Matthews Correlation Coefficient (MCC) are the evaluation metrics, following prior studies.Source: §4.2 · p. 5

4.3 Baselines

  • Keyphrase baselines include PromptRank, KeyBERT, YAKE, TextRank, TopicRank, SingleRank, and TFIDF.Source: §4.3 · pp. 5–6
  • Sentiment baselines include event detection, financial LLMs, GPT models, RoBERTa, and FinBERT variants.Source: §4.3 · p. 6
  • Time-based baselines combine textual and time-series data, with CMIN and StockNet representing multimodal prediction approaches.Source: §4.3 · p. 6

4.4 Implementation Details

The implementation describes keyphrase scoring, dataset-specific exclusions, and the models and settings used for factor-based experiments.

4.4 Implementation Details

  • For keyphrase-based methods, examples are split by fall or rise labels before extracting negative and positive keyphrases to score each text.The score uses the presence of positive and negative keyphrases and a sigmoid function.Source: §4.4 · p. 6
  • Time-based methods are omitted for EDT, while factor-based evaluation runs SKGP without the TimeTemplate on that news-only dataset.Source: §4.4 · p. 6
  • Factor-based experiments use gpt-3.5-turbo-1106, gpt-4, and gpt-4-1106preview, with window size 5, five factors or keyphrases, and batch sizes of 64 for BERT and 5 for GPT models.Source: §4.4 · p. 6

4.5 Results

LLMFactor outperforms state-of-the-art methods across four datasets, while ablations identify factor extraction as its largest contributing layer. Case studies illustrate how it connects stock movements with relevant company and market events.

4.5 Results

  • 2.9%, 0.4%, 11%, and 4.8% MCC improvements over SOTAs across the four datasets, respectively, accompany LLMFactor’s superior performance.Table 2 reports ACC in percentages and omits time-based method comparisons for EDT, which lacks time-series data.Source: §4.5 · p. 6, Table 2 · p. 7
  • SKGP combines stock-relevant factors with relation and temporal information to filter irrelevant content and provide a more comprehensive analysis.Keyphrases mainly summarize text, while factors are presented as more readable and more directly explanatory of stock movements.Source: §4.5 · pp. 6, 8

4.6.1 Ablation Study

  • The factor layer contributes most to overall performance, improving ACC by 9% and MCC by 46% in the ablation study.The price layer accounts for approximately 86% of ACC and 32% of MCC; the relation layer adds 5% ACC and 22% MCC.Source: Table 3 · p. 7, §4.6.1 · p. 8

4.6.2 Case Study

  • In the case study, AAPL rises amid Corning investment and iPhone 11 demand, while TSLA declines amid losses, investor concerns, and competition.The authors describe integrating company affiliations with historical news and price data as supporting interpretability.Source: Figure 3 · p. 7, §4.6.2 · p. 8

5 Conclusion

The paper presents LLMFactor, using SKGP to combine background knowledge, stock-related factors, and temporal data for stock-movement forecasting. Across four benchmark datasets, it reports superiority over several classes of existing methods and highlights factor analysis for interpreting market dynamics.

5 Conclusion

  • LLMFactor uses Sequential Knowledge-Guided Prompting to integrate background knowledge, stock-related factors, and temporal data for stock-movement prediction.The conclusion describes the framework as designed to predict and elucidate stock market trends.Source: §5 · p. 8
  • Testing on four benchmark datasets shows superiority over state-of-the-art methods relying on keyphrases, sentiment analysis, and multimodal inputs.The conclusion does not specify dataset-level metrics in this passage.Source: §5 · p. 8
  • The authors present factor analysis as a means of highlighting LLMFactor’s effectiveness for financial analysis and transparent forecasting.The conclusion characterizes this as an advancement in applying LLMs to explainable financial forecasting.Source: §5 · p. 8

6 Limitations

The authors identify temporal-data conversion, response variability, and factor quality across text types as limitations for LLMFactor and directions for further work.

6 Limitations

  • Converting temporal data into text remains important to financial forecasting, and future work could explore other conversion methods and their integration into LLMs.The authors identify this as a limitation despite relying on extensively discussed factors.Source: §6 · p. 9
  • Variable LLM responses make exact reproduction of the experimental results challenging, motivating further work on reproducibility.Source: §6 · p. 9
  • Benchmark evaluation does not resolve how well extracted factors perform across texts of different lengths and types; the authors plan to improve factor quality.Source: §6 · p. 9

A Templates of SKGP

The final SKGP step compiles earlier responses into LLM prompts, but variability in LLM responses prevents guaranteed exact reproducibility.

A Templates of SKGP

  • The templates for each SKGP step are provided in Table 5 for English and Table 6 for Chinese.Source: A Templates of SKGP · p. 11
  • In the final SKGP step, responses from previous steps are compiled to formulate prompts for LLMs.Source: A Templates of SKGP · p. 11
  • Variable LLM responses mean exact reproducibility cannot be guaranteed; the authors instead demonstrate overall superior performance.Source: A Templates of SKGP · p. 11

B Details of Pre-trained Models

Table 4 provides configuration details for sentiment-based models, with English models listed above Chinese models; keyphrase baselines are accessed through dedicated toolkits.

C Comparative Experimental Results of Factor Templates

Comparisons across factor templates find different best-performing templates for English and Chinese, while LLMFactor remains consistently effective across templates.

Loading 2406.10811v1…