Source-linked AI summary
Deep Learning in Finance
J. B. Heaton, N. G. Polson, J. H. Witte
TL;DR
Financial prediction involves large, heterogeneous data whose complex interactions are difficult to specify in existing economic models. The paper applies deep learning hierarchical decision models to financial prediction and classification, finding potential improvements in predictive performance, sometimes dramatically, while noting challenges for current financial theory and market-efficiency thinking.
Problem
Financial prediction may require large, diverse data and complex interactions that financial economic theory does not fully specify.
Method
The paper introduces deep learning hierarchical decision models for financial prediction and classification, using regularization and cross-validation to support model selection and generalization.
Results
Deep learning has the potential to improve predictive performance in conventional financial applications, sometimes dramatically.
Takeaways & Limitations
Deep learning may exert increasing influence in finance, particularly where prediction is paramount.
Takeaways & Limitations
The approach presents significant challenges to current thinking in finance, including the concept of market efficiency, and its implications remain unresolved.
Abstract
from arXiv · showhide
We explore the use of deep learning hierarchical models for problems in financial prediction and classification. Financial prediction problems -- such as those presented in designing and pricing securities, constructing portfolios, and risk management -- often involve large data sets with complex data interactions that currently are difficult or impossible to specify in a full economic model. Applying deep learning methods to these problems can produce more useful results than standard methods in finance. In particular, deep learning can detect and exploit interactions in the data that are, at least currently, invisible to any existing financial economic theory.
1 Introduction
Financial prediction draws on large, diverse data with complex interactions that financial theory does not fully specify. The paper proposes deep learning hierarchical decision models to learn these features while incorporating broad inputs and reducing over-fitting.
- Motivation: Financial prediction may depend on information distributed across economic and other data sources.Different market participants monitor disparate sources for clues about future price movements.
- Problem: Large data collections and complex nonlinear interactions are difficult to specify using financial economic theory.This contributes to many weakly justified models with over-fitting and poor out-of-sample performance.
- Contribution: The paper introduces deep learning hierarchical decision models for financial prediction and classification.The proposed method is intended to learn complex input features that support predictions of assets or portfolio returns.
- Advantages: Deep learning permits expanding inputs to include all items of possible relevance to a prediction problem.This addresses the breadth of potentially informative financial data.
- Advantages: Deep learning accounts for nonlinearities and complex interactions among inputs, potentially increasing in-sample fit versus traditional models.The paper also identifies easier avoidance of over-fitting as an advantage.
- Modeling principle: The paper emphasizes high-dimensional inputs as a guiding principle for predictive models.It contrasts predictive algorithmic modeling with traditional statistical approaches.
2 Deep Learning
The paper begins by introducing a general theoretical deep learning framework and several specifications.
- Framework: The paper first introduces a general theoretical deep learning framework.
- Framework: The framework includes several model specifications.
- Framework: This introduction precedes the paper’s finance applications.
2.1 Architecture
Deep learning maps high-dimensional inputs to numeric or categorical outputs through layered nonlinear transformations. Hidden features extracted at successive layers provide flexible representations of nonlinear data structure, including financial portfolios of payouts.
- Concept: A learning machine maps a high-dimensional input X to an output Y, which may be continuous, discrete, or mixed.Classification outputs index categories.
- Concept: Deep learning trains models on data and passes learned features through multiple levels of abstraction.Each layer transforms extracted factors into features for the next deeper layer.
- Architecture: Activation functions transform weighted data, while the architecture must specify the number of hidden units in each layer.The predictor is formed as a composite map of these layer transformations.
- Prediction rule: The output is produced from final hidden features through weights and activation levels.The hidden features Z(l) are factors extracted by the algorithm.
- Architecture: Matrix-valued weights give the predictor flexibility to uncover nonlinear features, while hidden-unit dimensions determine which hierarchical terms remain.Estimated hidden features can represent portfolios of payouts in finance.
- Architecture: A deep architecture applies L nonlinear transformations to X, with depth represented by the number of layers L.The final prediction is the output of the layered hierarchy.
- Activation functions: Common activation choices include sigmoid, hyperbolic, gate, and ReLU functions, with ReLU especially suited to rapid dimension reduction.
2.2 Training a Deep Architecture
Training estimates network parameters by minimizing a loss, typically with regularization and validation-based model selection. Cross-validation and stochastic optimization target generalization, while computational demands arise from the model’s multimodal objective.
- Training objective: Training uses input-output pairs, a loss function, and estimated weights and offsets to fit the deep predictor.The learned parameters are denoted by Ŵ and b̂.
- Loss functions: Mean-squared error is a standard training loss when an L2-norm error measure is used.
- Regularization: Regularization adds a penalty weighted by λ > 0 to stabilize predictions and avoid over-fitting.The amount of regularization controls the trade-off between fitting and generalization.
- Regularization: Ridge or L2 regularization is a useful default, while lasso or L1 regularization can induce sparsity in weights and offsets.
- Probabilistic interpretation: The probabilistic formulation yields a regularized maximum a posteriori estimator.For default prediction, multinomial logistic regression leads to cross-entropy loss; L2 loss suits multivariate normal models.
- Cross-validation: Cross-validation reduces over-fitting, assesses out-of-sample performance, and selects regularization, layer depth L, and hidden-unit sizes Nl.It addresses both model selection and estimation choices.
- Optimization: Back-propagation uses stochastic gradient descent, but multimodality can slow convergence and increase reliance on computational power.Mini-batches are used for ultra-large data sets.
2.3 Predictive Performance
The section frames predictive performance and model selection as central training problems for deep architectures. It emphasizes choosing regularization and architecture size using out-of-sample predictive loss.
- Deep learning training must address both regularization strength and the size and depth of each layer.
- Cross-validation with an independent hold-out set selects regularization by optimizing out-of-sample predictive loss.SURE is presented as an alternative approach.
- Model selection determines the number of layers L and hidden-unit dimensions N = (N1, . . . , NL).Dropout is introduced as a technique for addressing this problem.
- The chain rule makes derivatives available across the composition of L layers, supporting predictive-risk calculations for large data sets.
- Models with the best predictive MSE are favored.
2.4 Dropout for Model Selection
Dropout regularizes deep predictors by randomly removing inputs or hidden units, reducing reliance on small subsets of data and helping select architecture dimensions.
- Dropout randomly removes input dimensions with probability p to avoid over-fitting during training.
- The dropout architecture replaces X with D ⋆X, where D contains independent Bernoulli Ber(p) random variables.
- Marginalizing dropout randomness yields an objective interpretable as Bayesian ridge regression with a g-prior.
- Dropout reduces the likelihood of over-reliance on small sets of input data during training.
- Dropping hidden units can select the number of hidden units, but removing a layer variable also removes all higher-layer terms in the composite predictor.
2.5 Auto-encoder
Auto-encoders learn cost-effective lower-dimensional representations by reconstructing their inputs through a bottleneck. In dynamic financial settings, the encoder captures current and historical information for reconstruction.
- An auto-encoder trains a deep architecture to approximate X by itself through a bottleneck structure.
- For training data {X1, X2, . . .}, the reconstruction targets are set equal to the inputs, Yi = Xi.
- A hidden layer with M ≪ N units provides a lower-dimensional encoding of an N-dimensional input.
- Simultaneously estimating both weight matrices produces a two-layer deep learning model, whereas estimating them differently yields factor-model or sliced-inverse-regression variants.
- In a dynamic auto-encoder, the state equation encodes Yt and a matrix W decodes it into its history Yt−1 and current state Xt.
- The auto-encoder is directly predictive and does not require explicit modeling of the variance-covariance matrix.
2.6 Long Short Term Memory Models (LSTMs)
RNNs model temporal sequences but can struggle with long-term dynamics because of vanishing and exploding gradients. LSTMs address this with gated memory states that regulate retained and newly incorporated information.
- RNNs: RNNs compute updates sequentially for length-T input sequences and can learn complex temporal dynamics.
- RNNs: RNNs have difficulty learning long-term dynamics partly because gradients can vanish or explode across recurrent layers.
- LSTMs: LSTMs add an input gate, forget gate, input modulation gate, and memory cell to recurrent networks.
- LSTMs: The forget and input gates regulate which past information is discarded and which current information updates the cell state.
- LSTMs: LSTMs may improve financial predictors by memorizing volatility patterns from previous periods and identifying temporal relations automatically.This comes at the cost of larger parameter sets to train.
- LSTMs: LSTM models can capture long-memory effects in the underlying structure of asset-return movements.
3 Finance Applications
The paper applies deep learning to financial prediction and classification, including factor modeling, default-probability estimation, and event-study analysis. Its hierarchical models jointly learn representations and predictions while accommodating nonlinear relationships and high-dimensional inputs.
- Deep factor models: Shallow dimension-reduction methods can discard information valuable for predicting the desired output because they are independent of the response variable.
- Deep factor models: Deep learning jointly estimates layer functions using the full training data, incorporating information on outputs, inputs, and their relationships.
- Deep factor models: Deep learning models financial data as hierarchical nonlinear factor structures, with recursively defined nonlinear link functions across layers.
- Default probabilities: For credit risk, deep learning combines asset returns, earnings-call text, and accounting data to form a feature representation of firm creditworthiness.
- Default probabilities: Cross-entropy minimization is equivalent to a multiclass logistic likelihood, while feature extraction produces nonlinear summaries of firms’ tendency to default.
- Event studies: Event-study deep learners use input event embeddings and pooling choices to measure the effect of previous events on today’s return.
4 Example: Smart Indexing
The smart-indexing example compares auto-encoder and deep feature policy approaches for approximating the S&P500. Although the DFP fits training data more closely, the simpler auto-encoder basis is more consistent out of sample.
- Indexing approaches: Index replication can select stocks historically similar to the index or stocks representing a disproportionately large share of aggregate index information.
- Auto-encoder basis: A deep auto-encoder directly approximates aggregate information and treats stocks closest to its compressed core as a nonlinear basis for the index.
- Auto-encoder basis: The ten stocks with the highest communal information content form an equally weighted S&P500 approximation after ranking stocks by proximity to their auto-encoded versions.
- Deep feature policy: A DFP combines the ten most communal and ten most individualistic stocks, then optimizes approximation actions from hierarchical nonlinear features.
- In-sample comparison: The DFP improves in-sample fitting, including the final six months of training, whereas the ten-stock auto-encoder basis deviates from the S&P500 near the training period’s end.
- Out-of-sample comparison: Out-of-sample testing finds the DFP unreliable while the simple ten-stock auto-encoder basis consistently replicates the index.
5 Conclusion
The paper presents deep learning as a flexible framework for optimizing predictive performance with large data sets in finance. It may increasingly influence financial practice while challenging current thinking about market efficiency.
- Deep learning hierarchical decision models provide a general framework for financial prediction and classification using large data sets.The framework is presented as suitable for practical and theoretical finance problems.
- Deep learning can improve predictive performance in conventional financial applications, sometimes dramatically.
- Deep learning may challenge existing financial-economic thinking, especially the concept of market efficiency.The paper raises open questions about whether new tests of market efficiency will be necessary.
- Prediction-focused areas of finance are likely to experience increasing influence from deep learning models.