Source-linked AI summary
Applications of deep learning in stock market prediction: recent progress
Weiwei Jiang
TL;DR
Rapid progress in deep learning for stock market prediction has made it difficult to keep up with recent developments. This survey reviews more than 100 related articles, examines data, models, evaluation, and reproducibility, and reports that deep learning models outperform linear and machine learning models on stock market prediction tasks.
Problem
Rapid progress in the field makes it difficult for novices to catch up with the latest developments.
Method
The survey reviews more than 100 published articles on deep learning models for stock market prediction and investigates surveyed papers' data and code availability.
Results
Deep learning models show better performance than both linear and machine learning models on stock market prediction tasks.
Takeaways & Limitations
The survey aims to help researchers keep up with progress and boost future research in related topics.
Takeaways & Limitations
The survey focuses on stock market applications and does not cover all related machine learning problems.
Abstract
from arXiv · showhide
Stock market prediction has been a classical yet challenging problem, with the attention from both economists and computer scientists. With the purpose of building an effective prediction model, both linear and machine learning tools have been explored for the past couple of decades. Lately, deep learning models have been introduced as new frontiers for this topic and the rapid development is too fast to catch up. Hence, our motivation for this survey is to give a latest review of recent works on deep learning models for stock market prediction. We not only category the different data sources, various neural network structures, and common used evaluation metrics, but also the implementation and reproducibility. Our goal is to help the interested researchers to synchronize with the latest progress and also help them to easily reproduce the previous studies as baselines. Base on the summary, we also highlight some future research directions in this topic.
1. Introduction
Stock market prediction is a longstanding finance and computer-science problem, evolving from fundamental, technical, linear, and machine-learning approaches toward deep learning. This survey reviews recent deep-learning progress, prediction workflows, implementation and reproducibility, and future research directions.
- The survey focuses on recent deep-learning applications represented by diverse deep neural-network structures.
- Deep learning models have shown better performance than linear and machine-learning models on stock market prediction tasks.
- Recent developments include online news and Twitter data sources and graph neural networks using knowledge-graph data.
- The survey summarizes especially recent work from the past three years and presents trends across prediction-workflow steps.
- It emphasizes implementation and reproducibility by addressing data and code availability and helping readers reproduce prior models as baselines.
- The authors identify future research directions based on their survey of recent studies.
2. Related Work
Prior surveys span neural networks, text mining, evolutionary computation, machine learning, and financial forecasting, but this paper focuses on recent deep-learning work in stock prediction. It emphasizes implementation, reproducibility, and stock-specific evaluation concerns.
- Prior surveys: Earlier reviews cover neural networks, text mining, evolutionary computation, financial forecasting, and broader quantitative-finance applications.Their scopes include stock prices, financial text, preprocessing, clustering, forecasting, trading systems, and portfolio optimization.
- Prior surveys: Recent reviews address machine-learning methods, financial instruments, financial time series, and fundamental or technical analysis, but do not share one narrow scope.The cited reviews differ in community coverage, instrument coverage, and methodological emphasis.
- Open issues: Existing literature includes unsuitable benchmarks, short evaluation periods, and nonoperational trading strategies.These issues are identified as problems in prior work rather than as findings from a new empirical benchmark.
- This survey: This survey targets deep-learning applications to stock price and market-index prediction during 2017–2019.Its scope is narrower and more recent than a review covering deep learning for financial time-series forecasting from 2005 to 2019.
- This survey: The review prioritizes implementation workflow and reproducibility, including dataset and code availability.It treats reproducibility as a problem that has drawn attention from AI researchers.
- This survey: It also highlights stock-market-specific evaluation, including profitability alongside prediction accuracy.The survey further notes challenges involving live testing, algorithmic trading, long-term prediction, and sentiment analysis on company filings.
3. Overview
The survey covers 124 papers on stock and market-index prediction, classifying tasks by target and frequency and cataloguing markets, sources, venues, and data availability. Most studies use daily prediction, while the review spans mature and emerging markets.
- Corpus: The review collects papers from Google Scholar and includes 56 journal papers, 58 conference papers, and 10 arXiv preprints.The preprints are included to keep the survey updated with recent progress.
- Scope: The study focuses on close-price prediction for individual stocks and market indexes, including ETFs, equity-index futures, and intraday limit-order-book mid-prices.Bond-price and cryptocurrency-price prediction are outside the stated coverage.
- Prediction problems: 105 of 124 papers study daily prediction, 18 of 124 study intraday prediction, and only one studies both frequencies.The survey attributes the imbalance partly to the greater availability of daily historical prices and news titles than intraday data.
- Prediction problems: 124 papers divide into 52 daily-classification, 54 daily-regression, 8 intraday-classification, and 11 intraday-regression studies.The four categories are defined by target output and prediction frequency.
- Markets: Most studies focus on one market, while some evaluate models across multiple markets; both mature markets such as the US and emerging markets such as China receive substantial attention.Surveyed markets and their major indexes are listed in Table 3.
4. Prediction Workflow
The survey organizes prior stock-prediction studies into a reproducible four-step workflow. The steps are Raw Data, Data Processing, Prediction Model, and Model Evaluation.
- Prediction Workflow: The review summarizes studies through a general workflow with four steps: Raw Data, Data Processing, Prediction Model, and Model Evaluation.The workflow is intended to classify and summarize previous studies systematically.
- Prediction Workflow: The workflow presents a general approach that future work can reproduce across different data-source combinations.The survey discusses each step separately and includes data obtaining, processing, and usage of different data types.
4.1. Raw Data
The survey organizes stock-prediction inputs into intrinsic market history and diverse extrinsic sources, whose availability, cost, and processing difficulty shape their use. Most studies use market data, while recent work increasingly combines more diverse sources and generally targets short horizons.
- Data categories: Stock-prediction studies use intrinsic historical prices and extrinsic sources such as market, fundamental, text, macroeconomic, knowledge-graph, image, and analytics data.The survey frames data collection as the first prediction step and notes that extrinsic sources affect stock markets.
- Data availability: Market data is used most because it supplies large amounts of accessible trading information for complex deep-learning models.Text data is used next most, while analytics data is absent from the surveyed studies because of sparsity and access cost.
- Research trends: More diverse data types appear in studies from 2018 and 2019 than in 2017, alongside newer tools such as graph neural networks for knowledge-graph data.The survey states that this trend indicates difficulty obtaining better results using only market data.
- Data length: Choosing the historical-data length involves a tradeoff: short periods increase insufficiency and overfitting risk, whereas long periods may span different market styles and become outdated.Availability and cost also constrain the choice of data length.
4.2. Data Processing
The survey describes preprocessing steps that address missingness, noise, feature construction, and feature scale before prediction. These steps include forward propagation for aligning frequencies, denoising, engineered representations, dimensionality reduction, and normalization.
- Data alignment: Lower-frequency data are forward-filled with the last valid observation when aligning sources with different sampling frequencies, preventing future-information leakage.The procedure is illustrated for market and fundamental data.
- Denoising: Stock-market data contain noise that can misrepresent price trends and misguide prediction, motivating wavelet transforms and kNN-based data preparation.Sun et al. use two training sets with different labels in a data-preparation layer.
- Feature extraction: Feature engineering extracts domain-informed inputs such as technical indicators, text embeddings, event representations, sentiment vectors, and linguistic features.Examples include word2vec, GloVe, neural tensor networks, CNNs, NLTK, and LIWC.
- Text features: Commercial linguistic software can substantially boost machine-learning model performance, while sentiment analysis produces positive, neutral, or negative factors for prediction.LIWC counts words associated with emotions, thinking styles, and social concerns.
- Dimensionality reduction: Dimensionality reduction addresses correlated technical indicators and the associated overfitting risk in deep-learning models.PCA projects inputs into a lower-dimensional space using singular value decomposition; PCA-ANN achieved slightly higher next-day SPY direction accuracy than FRPCA and KPCA.
4.3. Prediction Model
The survey categorizes prediction models into standard architectures, variants, hybrid models, and other approaches, emphasizing feedforward, convolutional, and recurrent families. It also notes continued model development and recurring limitations of particular architectures.
- Learning paradigms: Most prediction models use supervised learning, whereas only a few studies use semi-supervised learning.The surveyed workflow separates training-set fitting from test-set evaluation.
- Standard models: Standard deep-learning models are organized into feedforward neural networks, convolutional neural networks, and recurrent neural networks.The survey also categorizes generative adversarial, transfer-learning, and reinforcement-learning approaches as other models.
- Convolutional models: CNNs process spatial inputs through shared-weight filters and pooling, while 1D CNNs adapt convolution to time-series classification and prediction.Weight sharing reduces parameters relative to densely connected feedforward networks.
- Recurrent models: RNN connections form cycles across temporal sequences, but normal RNNs are affected by vanishing gradients; LSTM and GRU variants use recurrent gates.GRUs use fewer parameters than LSTMs, and bidirectional variants connect opposite-direction hidden layers.
- Variants and hybrids: Variants and stacked or hybrid models are developed to improve prediction performance, including CNN–RNN combinations and models using multiple recurrent architectures.The survey groups prediction models into standard models, variants, hybrid models, and other models.
- Survey observations: The survey reports that RNN models are used most often, their baseline share has increased over the past three years, and capsule networks are effective for this task.These are survey-level observations about model usage and reported effectiveness.
4.4. Model Evaluation
The survey organizes model evaluation into regression, classification, profit, and significance analyses, documenting commonly used metrics and statistical tests.
- Classification metrics: Classification evaluation measures movement-prediction performance using accuracy, precision, recall, sensitivity, specificity, F1, MCC, and AUC.
- Regression metrics: Regression evaluation measures stock or index price-prediction performance using metrics such as MAE, MSE, RMSE, MAPE, NMSE, and R2.
- Profit analysis: Profit analysis evaluates return and risk, including portfolio-value change, maximum drawdown, annualized volatility, and Sharpe Ratio.
- Significance analysis: Kruskal-Wallis and Diebold-Mariano tests can assess whether deep learning predictions differ significantly from baselines, but few stock-prediction studies use them.
- Metric usage: RMSE and MAPE are the most often used regression metrics, followed by MAE and MSE.
5. Implementation and Reproducibility
The survey reviews implementation choices and reproducibility resources, finding Python-based frameworks dominant while emphasizing data and code availability as prerequisites for replication.
- Programming languages: Python is becoming the dominant programming language for stock market prediction research.
- Frameworks: Keras and TensorFlow are the dominant frameworks for deep learning-based stock market prediction research.
- Computational resources: Deep learning training requires substantial computation, with GPUs used to accelerate convolutional operations and expected to remain increasingly important for text data.
- Reproducibility: The survey investigates data and code availability because both are important aspects of result reproducibility.
- Data resources: Authors’ exact datasets and public links would make replication more convenient, while cloud services are not widely adopted or explicitly reported in the surveyed studies.
6. Future Directions
The survey identifies future directions in newer neural architectures, multimodal data, cross-market learning, and practical trading evaluation.
- New model structures: Newer neural-network structures remain insufficiently studied, including attention mechanisms, generative adversarial networks, Transformers, and pretrained BERT for text sentiment analysis.
- Multiple data sources: Using multiple data sources, especially underexplored sources, is proposed instead of relying on a single market-data source.
- Cross-market learning: Cross-market analysis and transfer learning merit further study because markets differ in trading rules while sharing phenomena that may support prediction.
- Cross-market learning: A model trained on US stock-market data was tested on markets in 31 countries over 12 years in one cross-market study.
- Trading systems: Practical algorithmic trading requires more realistic strategies that account for trading rules, transaction costs, market styles, and lengthy deep-learning training.
- Reinforcement learning: Deep reinforcement learning is identified as a direction for simulating more cases and making faster trading choices than humans.
7. Conclusion
The survey reviews more than 100 articles from 2017–2019 across the stock-prediction pipeline, implementation, reproducibility, and future research directions.
- The survey covers more than 100 related published articles from the past three years and presents research trends from 2017 to 2019.
- Its coverage spans raw-data collection, data processing, prediction models, and model evaluation.
- The survey gives special attention to deep-learning implementation and published-article reproducibility to support adoption of models as baselines.
- It highlights future directions intended to provide insight and support further research on stock market prediction.