Source-linked AI summary
FNSPID: A Comprehensive Financial News Dataset in Time Series
Zihan Dong, Xinyu Fan, Zhiyuan Peng
TL;DR
Traditional financial forecasting relies on historical quantitative data, while comprehensive datasets combining news sentiment and stock prices remain limited. This paper constructs FNSPID, an integrated time-series dataset, and reports that dataset size and sentiment quality improve prediction performance.
Problem
Traditional financial models rely on historical data that limits anticipation of future shifts, and open access to high-quality financial datasets is limited.
Source: §1 · p. 1, §2.2 · p. 3Method
FNSPID combines time-series financial news and stock prices, with sentiment labels generated using ChatGPT for a small article dataset from 50 prominent US stocks.
Source: §3 · p. 4, §3.1 · p. 4Results
Experiments report improved price-prediction performance with larger datasets and a positive effect of sentiment quality on accuracy.
Source: §1 · p. 2Takeaways & Limitations
FNSPID supports stock-prediction ML and sentiment-price analysis, and the paper provides reproducible examples for expanding the dataset.
Source: §1 · p. 2Future changes to website policies may affect dataset accessibility, and evolving prediction models require ongoing validation on new datasets.
Source: §7.1 · p. 8
Abstract
from arXiv · showhide
Financial market predictions utilize historical data to anticipate future stock prices and market trends. Traditionally, these predictions have focused on the statistical analysis of quantitative factors, such as stock prices, trading volumes, inflation rates, and changes in industrial production. Recent advancements in large language models motivate the integrated financial analysis of both sentiment data, particularly market news, and numerical factors. Nonetheless, this methodology frequently encounters constraints due to the paucity of extensive datasets that amalgamate both quantitative and qualitative sentiment analyses. To address this challenge, we introduce a large-scale financial dataset, namely, Financial News and Stock Price Integration Dataset (FNSPID). It comprises 29.7 million stock prices and 15.7 million time-aligned financial news records for 4,775 S&P500 companies, covering the period from 1999 to 2023, sourced from 4 stock market news websites. We demonstrate that FNSPID excels existing stock market datasets in scale and diversity while uniquely incorporating sentiment information. Through financial analysis experiments on FNSPID, we propose: (1) the dataset's size and quality significantly boost market prediction accuracy; (2) adding sentiment scores modestly enhances performance on the transformer-based model; (3) a reproducible procedure that can update the dataset. Completed work, code, documentation, and examples are available at github.com/Zdong104/FNSPID. FNSPID offers unprecedented opportunities for the financial research community to advance predictive modeling and analysis.
1 INTRODUCTION
Traditional financial analysis and available datasets leave gaps in anticipating market shifts and combining news sentiment with time-series prices. The paper introduces FNSPID and reports improved prediction with larger datasets and sentiment quality.
1 INTRODUCTION
- Existing resources often lack the volume or corresponding stock prices needed for integrated analysis, while news may lack time-series structure.Traditional models also rely on historical data, limiting their effectiveness in anticipating future shifts or unprecedented events.Source: §1 · pp. 1–2
- FNSPID combines time-series news and stock prices to provide a resource for financial market analysis.Its integrated textual and numerical data is intended for stock-prediction ML as well as sentiment-price correlation analyses.Source: §1 · p. 2
- Larger datasets improve price-prediction performance, while sentiment quality positively affects accuracy.The experiments report these effects across FNSPID evaluations.Source: §1 · p. 2
- The paper presents reproducible examples and instructions for expanding FNSPID.It also identifies sentiment analysis, LLM fine-tuning, and deep-learning research as applications.Source: §1 · p. 2
2 RELATED WORK
Related work traces a shift from traditional financial models toward machine learning, sentiment analysis, and language models, while emphasizing shortcomings in existing financial datasets. FNSPID is presented as a broad, accessible collection of news and stock-price data.
2.1 Evolution of Financial Analysis Models
- Traditional financial models support long-term analysis but lack granularity for short-term peaks and troughs; ML methods are explored for market timing and complex patterns.The discussion covers methods from linear regression and SVM to LSTM, RNN, and deep Q-learning.Source: §2.1 · p. 2
- Prior studies connect financial news and sentiment with market movements, while recent work explores language models for sentiment and time-series prediction [21] [24] [40] [42] [51].The cited work also includes GPT-3.5-based news generation and FinGPT [24] [45].Source: §2.1 · pp. 2–3
2.2 Existing Stock Dataset
- Existing datasets offer sentiment labels, entity annotations, or time-series data, but reported gaps include limited news volume, missing timestamps, or insufficient company and trading information [40].Philippe’s Bloomberg and Reuters collection lacks target-sentiment entities in its raw news content.Source: §2.2 · p. 3
- FNSPID covers English and Russian financial news from 1999 to 2023, correlates news with stock prices, and uses trusted financial-news platforms.The paper also frames the dataset as comprehensive and accessible for research where some valuable datasets and models have access restrictions.Source: §2.2 · p. 3
3 CONSTRUCTING FNSPID
FNSPID combines numerical stock data with financial news, then builds summarized-news and sentiment-labeled datasets. Its processing summarizes articles, assigns ChatGPT sentiment scores, and fills dates without news using decay.
3.1 Data mining and processing
- The example prompt supplies a system prompt, two examples, and summarized news, with stock symbol and news text as inputs for sentiment labeling.The resulting sentiment data pairs stock symbols with summaries generated by four methods.Source: Figure 2 · p. 4, Figure 3 · p. 4
3 CONSTRUCTING FNSPID
- FNSPID Task 1 combines Yahoo Finance stock data with news collected from NASDAQ and prior data from Bloomberg, Reuters, Benzinga, and Lenta.The authors say the sources were combined to enhance data integrity and diversity and prevent website bias; they report following site policies and limiting collection to freely available content.Source: §3 · p. 4
3.1 Data mining and processing
- Task 2 summarizes each article with LexRank, Luhn, LSA, and TextRank, producing three-sentence summaries about one-eighth of the original length.A stock-related weighting model emphasizes relevant content; the shorter summaries reduce token usage and support stable ChatGPT prompting.Source: §3.1 · p. 4
- Task 3 uses ChatGPT to assign sentiment scores from 1 (negative) to 5 (positive) to summaries produced by the LSA summarizer.The authors process up to 10 entries at a time with temperature 0 and normalize scores before combining them with other model features.Source: §3.1 · p. 4
- For dates without news, sentiment is extrapolated from the previous news day by exponential decay toward neutral; multiple articles on a date are averaged.The decay target is score 3, and the averaging is intended to represent the day's overall sentiment while limiting the influence of any single article.Source: §3.1 · p. 4, Figure 4 · p. 5
4 FNSPID PROPERTY
FNSPID contains varied numerical and sentiment data, with the authors reporting a dataset exceeding 30 GB and 402,546 sentiment-labeled news items. Its evaluation describes language, stock-symbol, and year-by-year news coverage.
4 FNSPID PROPERTY
- FNSPID exceeds 30 GB and includes numerical stock-price data alongside news text, sentiment scores, and article summaries produced by four methods.The authors report that assembling the dataset required approximately 4 TB of computing power and 45 days.Source: §4 · p. 5
- 402,546 news items received sentiment scores from a batch of 50 influential S&P 500 stocks selected using 2024 rankings.These sentiment-labeled items are part of the dataset's sentiment analysis work.Source: §4 · p. 5
4.1 Evaluation
- Figure 5 summarizes news containing stock symbols, English and Russian language distribution, included URLs, and existing versus potentially expandable news text.The authors describe language distribution and the distinction between articles with and without stock symbols as aspects of the dataset evaluation.Source: Figure 5 · p. 5, §4.1 · p. 5
- Figure 6 tracks annual news-article counts from 1999 to 2023, providing a view of the dataset's temporal distribution.The authors frame this analysis as showing trends and patterns in financial-news coverage over time.Source: Figure 6 · p. 5, §4.1 · pp. 5–6
- The authors present FNSPID's multilingual range and temporal depth as support for research in financial sentiment analysis and time-series prediction.They describe the dataset as a resource for financial modeling and analysis.Source: §4.1 · p. 6
5 EXPERIEMENT
Section 5 evaluates FNSPID through stock-price prediction experiments testing training-set quantity and sentiment quality. Results favor larger datasets and higher-quality sentiment, with gains varying by model.
5 EXPERIEMENT
- Experiments tested prediction models with different training-set sizes and compared FNSPID sentiment with TextBlob sentiment using matched sample-model parameters.Source: §5.1 · p. 6, §5.2 · p. 6
5.1 Quantity Test
- 6.29% average R² improvement from 5 to 25 training stocks appeared across six models; Transformer led at R²=0.988 versus LSTM at R²=0.856.Source: §5.1 · p. 6, Table 3 · p. 7
5.2 Quality Test
- For Transformer, FNSPID Task 3 sentiment improved overall stock-price prediction by 0.2%, while TextBlob sentiment had a -1.16% impact.Source: §5.2 · p. 6, Table 3 · p. 7
- Only Transformer showed a positive effect from sentiment across repeated experiments, while TimesNet did so occasionally.Source: §5.2 · p. 6
- With only 5 news items, LSTM outperformed Transformer in training; as the dataset grew, Transformer prediction accuracy improved significantly.Source: §5.2 · p. 7
- FNSPID experiments found that dataset quality and quantity affect stock-price prediction, high-quality sentiment helps Transformer training, and Transformer surpasses traditional time-series models and TimesNet.Source: §5.2 · p. 7
6 FNSPID APPLICATION AND ETHICS
Section 6 discusses FNSPID applications and ethics, alongside limitations in sentiment scoring. It describes uses spanning multimodal prediction, market analysis, risk management, and financial AI.
6.1 Challenges on FNSPID Construction
- Simple sentiment tools and BERT struggled with complex financial paragraphs, so authors excluded sentiment scores; ChatGPT scoring also became unstable for summaries over three sentences or long news inputs.Source: §6.1 · p. 7
6.2 FNSPID Applications
- FNSPID is proposed for multimodal stock prediction and sentiment analysis, parallel processing of news across stocks, and strengthening reinforcement-learning algorithms.Source: §6.2 · p. 8
- Aligned sentiment and numerical data support correlation analysis and more accurate sentiment labeling; news patterns preceding recessions are proposed for anomaly detection and risk management.Source: §6.2 · p. 8
6.3 Dataset Ethics
- The ethics discussion describes anonymization, GDPR and CCPA compliance, fairness audits, usage guidelines, and transparent data marking as measures for privacy, misuse prevention, and reproducibility.Source: §6.3 · p. 8
7 LIMITATIONS AND FUTURE WORK
The authors identify constraints on dataset accessibility and model validation, alongside limited news-sentiment alignment. They propose expanding and updating FNSPID and exploring additional research uses.
7.1 Limitations
- Future website-policy changes could affect dataset accessibility, and evolving prediction methods require ongoing validation with new datasets.Changes to policies may also necessitate adjustments to data collection.Source: §7.1 · p. 8
7.2 Future work
- Only 20% of stock-price timestamps align with news sentiment, and the authors plan an automated system to keep the dataset up to date.The amount of collectible news data is limited because popular stock-news data has accumulated in a short period.Source: §7.2 · p. 8
- Future uses include multimodal modeling, studying sentiment’s influence on price fluctuations, developing sentiment algorithms, and stock-correlation analysis.Source: §7.2 · p. 8
A.1 Asset models
This appendix describes two asset-pricing models: the FFE model integrates market, size, and value factors, while APT uses multiple macroeconomic risk factors.
A.1 Asset models
- The FFE model explains asset price using market excess return, size premium, and value premium factors.It builds on CAPM and integrates market risk, size, and value factors.Source: §A.1 · p. 10
- APT frames portfolio return as explainable by multiple risk factors, including industrial production, inflation, and bond-return spreads.The described factors include changes in industrial production, expected and unexpected inflation, and excess returns on corporate and government bonds.Source: §A.1 · p. 10
A.2 Summarize Algorithm
The summarization algorithm weights sentences to direct attention toward the related stock, then sorts sentences by weight to produce a summary.
A.2 Summarize Algorithm
- Sentences containing the stock symbol receive weight m=1, while other sentences receive zero sentence weight.The method combines sentence and summary weights to rank sentences.Source: §A.2 · p. 10
- A selected summary sentence receives n=1 when it is included in the longer sentence; otherwise its summary weight is zero.Source: §A.2 · p. 10
- The method sorts the sentence dictionary by weight and generates the final summarized sentence.Source: §A.2 · p. 10
A.3 ML models Normalizer 𝑆𝑛= 𝑉𝑛
The normalization discussion covers relative change from an initial value and min-max scaling to a consistent range.
A.3 ML models Normalizer 𝑆𝑛= 𝑉𝑛
- Equation (7) calculates normalized change relative to an initial value as S_n=(V_n/V_0)-1.It expresses change from the initial value as a fraction of that value.Source: §A.3 · pp. 10–11
- Min-max normalization rescales a variable using its minimum and maximum values to support comparison and analysis.Equation (8) scales the variable to a range between 0 and 1.Source: §A.3 · p. 11