PaperlayerView exact revision on arXiv

Source-linked AI summary

Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics

Amir Mosavi, Pedram Ghamisi, Yaser Faghan, Puhong Duan

arXiv:2004.01509v1q-fin.STcs.LGecon.GNstat.ML

TL;DR

Economic applications face uncertainty, risk, and high-dimensional data that challenge traditional forecasting and optimization methods. This review examines DL, RL, and DRL applications and concludes that newer DL and DRL approaches generally improve performance and accuracy over classical machine learning.

  • Problem

    Economic forecasting and strategy design remain challenging under uncertainty, risk, and high-dimensional market conditions.

  • Method

    The paper reviews and compares deep learning, reinforcement learning, and deep reinforcement learning methods across multiple economic domains.

  • Results

    Across economic applications, DL and DRL approaches perform better and achieve higher accuracy than classical machine learning methods.

  • Takeaways & Limitations

    DL and DRL offer practical approaches for forecasting, strategy optimization, profitability, and risk-aware decision-making in competitive markets.

  • Takeaways & Limitations

    Q-learning can overestimate action values, producing upward bias in the algorithm.

Abstract

from arXiv · show

The popularity of deep reinforcement learning (DRL) methods in economics have been exponentially increased. DRL through a wide range of capabilities from reinforcement learning (RL) and deep learning (DL) for handling sophisticated dynamic business environments offers vast opportunities. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this work, we first consider a brief review of DL, RL, and deep RL methods in diverse applications in economics providing an in-depth insight into the state of the art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher accuracy as compared to the traditional algorithms while facing real economic problems at the presence of risk parameters and the ever-increasing uncertainties.

Deep Learning Methods … Convolutional Neural Networks (CNNs)

The reviewed deep learning methods include SAEs, DBNs, CNNs, and RNNs. SAEs and DBNs learn hidden or hierarchical representations, while CNNs extract and reduce feature dimensions through convolution and pooling.

  • Deep Learning Methods: The reviewed deep learning algorithms comprise stacked auto-encoders, deep belief networks, convolutional neural networks, and recurrent neural networks.
  • Stacked Auto-Encoders (SAEs): Stacked auto-encoders use an auto-encoder with visible and hidden layers trained in two steps to reconstruct input data.
  • Stacked Auto-Encoders (SAEs): The auto-encoder transforms inputs into hidden representations and estimates reconstruction error by minimizing Euclidean distance between input x and output y.The hidden layer is the encoder, with weights, biases, and an activation function defining the transformation.
  • Deep Belief Networks (DBNs): Deep belief networks are built by stacking restricted Boltzmann machines, whose conditionally independent hidden units capture hierarchical representations of training data.RBMs use visible and hidden units and are trained layer by layer.
  • Convolutional Neural Networks (CNNs): CNNs combine convolutional, pooling, and fully connected layers to transform inputs and intermediate features into feature maps and outputs.Kernels generate feature maps, while pooling reduces their dimensions and the number of network parameters.
  • Convolutional Neural Networks (CNNs): Convolutional neural networks apply kernels to inputs and intermediate features, with activation functions enhancing network nonlinearity.The convolution operation produces outputs across multiple feature-map channels.
  • Convolutional Neural Networks (CNNs): ReLU is currently the most popular CNN activation function because it supports notably rapid convergence and robustness against gradient vanishing.

Recurrent Neural Networks (RNNs)

RNNs extend feedforward neural networks with recurrent connections and hidden states to identify patterns in sequential and dynamic temporal data. Because traditional RNNs suffer from vanishing gradients, LSTMs and gated recurrent units were introduced to handle large sequential datasets.

  • Recurrent Neural Networks (RNNs): RNNs use recurrent connections and hidden states to model sequential data and dynamic temporal specifications.The hidden state is updated from the current input and prior recurrent activations, enabling temporal pattern processing.
  • Recurrent Neural Networks (RNNs): The recurrent hidden-state update uses nonlinear functions, coefficient matrices, and bias vectors, while the output is computed from the current recurrent activation.The parameters distinguish contributions from the current input, prior hidden-unit activation, and current-step activation.
  • Recurrent Neural Networks (RNNs): Traditional RNNs experience vanishing gradients, motivating LSTM and gated recurrent unit architectures for handling large sequential datasets.These architectures were introduced specifically to address the long-term-memory limitation associated with traditional RNNs.

Deep Reinforcement Learning Methods

This section reviews value-based, policy-gradient, and model-based deep reinforcement learning methods. It frames RL as an agent–environment problem focused on maximizing cumulative rewards through incremental learning and trial-and-error experience.

  • Deep Reinforcement Learning Methods: Deep RL methods are organized into value-based, policy-gradient, and model-based approaches.These are identified as the most commonly used deep RL algorithms.
  • Deep Reinforcement Learning Methods: RL formulates an agent–environment interaction in which the agent seeks to maximize cumulative rewards.The section introduces value and Q-value functions for evaluating expected cumulative rewards.
  • Deep Reinforcement Learning Methods: RL agents learn good behavior incrementally while using trial-and-error experience when interacting with the environment.The passage identifies incremental learning and trial-and-error interaction as two important RL characteristics.
  • Deep Reinforcement Learning Methods: The section presents the general structure of deep reinforcement learning approaches through an agent–environment interaction architecture.Figure 6 depicts agent–environment interaction, while Figure 7 presents the general DRL structure.

Value-based Methods · Deep Q-networks (DQN)

Value-based methods define policies through value functions, progressing from Q-learning and Bellman-based optimality to DQN’s neural estimation of nonlinear Q-values. DQN improves scalability and stability through target networks, replay memory, reward clipping, and Double DQN’s reduction of positive action-estimation bias.

  • Value-based Methods: Value-based algorithms construct value functions for defining policies, including Q-learning, DQN, and improved DQN variants.The discussion places DQN within the broader value-based algorithm class.
  • Value-based Methods: Q-learning uses a value function and the Bellman equation to characterize the optimal Q-learning value.The Bellman operator defines the corresponding value-function update.
  • Value-based Methods: The optimal Q-value function is Q*(s, a), while parameterized value functions can address high-dimensional and potentially continuous spaces.The passage also references optimality analysis in discrete spaces under sufficient exploration.
  • Deep Q-networks (DQN): DQN uses a neural network to estimate a complex, nonlinear Q-value function and achieved good online results in ATARI games.The algorithm was presented by Mnih et al. for online ATARI gameplay.
  • Deep Q-networks (DQN): DQN updates its target-network parameters only every A iterations to improve stability and reduce divergence risk.It also uses a target Q-network, replay memory, and reward clipping to control instabilities and maintain reasonable target values.
  • Deep Q-networks (DQN): Standard Q-learning can overestimate action values because it uses similar information to select and evaluate actions, creating upward bias.Double estimation is introduced to address this limitation.
  • Deep Q-networks (DQN): Double DQN improves stability and produces more accurate Q-value functions than the standard Q-network.The double estimator is used to remove positive bias in action estimation.

Distributional DQN · Policy Gradient Methods · Stochastic Policy Gradient (SPG)

The section contrasts distributional value modeling with policy-gradient approaches for optimizing expected cumulative rewards. It also describes stochastic policy-gradient estimation, value-function-based updates, and entropy-assisted exploration.

  • Distributional DQN: Distributional DQN represents return distributions rather than only expected cumulative returns, capturing stochastic rewards and agent transitions.
  • Distributional DQN: The distributional formulation defines a random distribution return function for a policy and expresses Q values using random state-action pairs.
  • Distributional DQN: Deep-learning distributional Bellman approximation enables risk-aware behavior and richer training signals that improve learning.
  • Policy Gradient Methods: Policy-gradient methods use neural-network parameterized policies to maximize expected cumulative reward.
  • Stochastic Policy Gradient (SPG): Stochastic policy-gradient methods derive an estimator for the gradient used to optimize the policy.
  • Stochastic Policy Gradient (SPG): Policy evaluation estimates the Q-function, while policy improvement takes gradient steps using value-function approximations.
  • Stochastic Policy Gradient (SPG): Cumulative returns from entire trajectories can replace direct Q-function estimation, while actor-critic methods estimate returns efficiently.
  • Stochastic Policy Gradient (SPG): Entropy functions regulate policy randomness and support efficient exploration.

Deterministic Policy Gradient (DPG) · Actor-Critic Methods · Combining policy gradient and Q-learning

The section reviews deterministic policy gradients, actor-critic architectures, and combinations of policy-gradient and value-based methods for handling high-dimensional or continuous-action reinforcement-learning problems. It emphasizes their respective gradient, value-estimation, exploration, and structural properties.

  • Deterministic Policy Gradient (DPG): DPG computes the expected gradient of the action-value function without integrating over the action space.This approximation is suited to high-dimensional action spaces.
  • Deterministic Policy Gradient (DPG): DPG algorithms can outperform stochastic policy-gradient algorithms in high-dimensional action spaces.DDPG and NFQ with continuous actions address action-space problems described for NFQ and DQN.
  • Actor-Critic Methods: Actor-critic architectures use an actor to update the policy distribution and a critic to estimate the current policy’s value function.In deep reinforcement learning, both functions can be parameterized by nonlinear neural networks.
  • Actor-Critic Methods: The behavior policy can bias the gradient, while the critic estimates Q(s, a; θ) for the current policy π.The cited formulation identifies the behavior policy as β and the critic parameter as θ.
  • Combining policy gradient and Q-learning: Policy-gradient methods combine sample-efficient learning with value-function approximation and support continuous actions, explicit exploration, and multiagent settings.The passage frames these capabilities as tools for improving reinforcement-learning policy strategies.
  • Combining policy gradient and Q-learning: Value-based and policy-based approaches can have quite similar structures in some specific settings.This similarity motivates examining combinations of policy-gradient and Q-learning methods.

Model-based Methods … Deep Learning Application in Economics

The section surveys model-based reinforcement learning, including learned-environment models, search and optimization methods, and integration with model-free techniques. It also introduces the paper’s broader reviews of deep learning and deep reinforcement learning applications in economics.

  • Model-based Methods: Model-based methods explicitly represent environment dynamics and reward functions, unlike the previously discussed value-based and policy-based model-free approaches.
  • Pure model-based methods: When the environment model is unknown, function approximators can learn it from experience to recommend actions.
  • Pure model-based methods: Discrete-action model-based methods commonly use lookahead search, whereas continuous-action settings can use trajectory optimization.
  • Pure model-based methods: Lookahead search generates potential trajectories but faces an exploration–exploitation trade-off when sampling them.
  • Integrating model-free and model-based methods (IMF&MBM): Model-free and model-based approaches can be combined, with model-based methods handling primary tasks and model-free fine-tuning supporting successful goal achievement.
  • Integrating model-free and model-based methods (IMF&MBM): Neural networks can integrate both approaches through value-function estimation with backpropagation and structured architectures that support robust generalization.
  • Review Section: The review covers diverse uses of deep learning and deep reinforcement learning in economics.
  • Deep Learning Application in Economics: It further discusses recent applications of deep learning across varied economics domains.

Deep Learning in Stock Pricing · Deep Learning in Insurance · Deep Learning in Auction Mechanisms

The reviewed applications span stock-price prediction, insurance fraud detection and risk measurement, and auction mechanism design. In auction design, deep learning supports complex multi-bidder and constrained auction settings, with RegretNet reporting higher revenue than specified baselines.

  • Deep Learning in Stock Pricing: Deep learning addresses stock-price forecasting under market uncertainty and risk, where investors seek improved returns.The section identifies stock-market value and development as important to business growth and forecasting as a challenging research problem.
  • Deep Learning in Stock Pricing: Deep learning applications in stock-price prediction are summarized in Table 1.The supplied passage identifies the table as covering applications of deep learning in stock price prediction, without providing its individual entries.
  • Deep Learning in Insurance: Insurance applications focus on fraud detection and practical measurement of diverse risks using machine-learning methods.The passage connects these applications to strong market demand for alternatives to traditional risk-measurement approaches.
  • Deep Learning in Auction Mechanisms: Auction design seeks trustworthy mechanisms that induce bidders to choose profit-maximizing strategies, building on prior single-item and partially optimal results.The section presents auction design as practically important for enabling organizations to provide better services.
  • Deep Learning in Auction Mechanisms: Multi-layer neural networks with augmented Lagrangian optimization enabled compatible multi-bidder auctions for more complex tasks than an LP-based approach.The proposed approach was reported as suitable for large settings with high profit and low regret.
  • Deep Learning in Auction Mechanisms: Neural networks were also used to design optimal-revenue auctions under budget constraints, Bayesian compatibility, multiple valuation distributions, and multiple items.The reported method improved on prior state-of-the-art approaches while targeting high-profit auction design.

Deep Learning in Banking and Online Market · Deep Learning in Macroeconomic

The section reviews deep-learning applications in fraud detection, macroeconomic forecasting, unemployment prediction, and tactical asset allocation. Reported results show improved predictive performance, including AE’s lowest average MAE and backpropagation-based simulation outperforming ARIMA.

  • Deep Learning in Banking and Online Market: Deep learning is applied to fraud detection in online shopping and credit cards, driven by demand for efficient systems that protect involved systems.The review identifies fraud detection as a challenging application requiring secure and efficient systems.
  • Deep Learning in Macroeconomic: Macroeconomic forecasting methods address economic growth and business changes but require substantial data and can suffer from model dependency.Recent approaches are presented as more acceptable than previous methods in the cited review.
  • Deep Learning in Macroeconomic: Encoder-decoder deep neural architectures improve unemployment-indicator prediction accuracy while requiring less data, evaluated using mean absolute error.The study compares average MAE values for CNN, LSTM, AE, DARM, and the Survey of Professional Forecasters.
  • Deep Learning in Macroeconomic: AE achieved the lowest average MAE, followed by SPF, with advantages in single-series efficiency, prediction accuracy, and model specification.The comparison is reported in Fig. 13 across CNN, LSTM, AE, DARM, and SPF.
  • Deep Learning in Macroeconomic: Backpropagation-based model simulation outperformed ARIMA in forecasting macroeconomic indicators, with performance evaluated using average RMSE.The comparison involved ANN, autoregressive, and ARIMA techniques, as shown in Fig. 14.
  • Deep Learning in Macroeconomic: Feed-forward neural networks were used for tactical asset allocation using macroeconomic indicators and price-volume trends.Chakravorty et al. proposed two different methods for investment decisions based on macroeconomic data.

Deep Learning in Financial Market (Service & risk management) · Deep Learning in Investment · Deep Learning in Retail

The reviewed applications span credit-risk forecasting, financial investment analysis, and retail experiences and operations. Deep learning supports reliable modeling of complex data, augmented-reality product assistance, and robust retail classification.

  • Deep Learning in Financial Market (Service & risk management): Deep learning is presented as a tool for forecasting credit risk in banking systems using big-data-driven financial models.The financial-market review identifies credit-risk handling as a crucial service and risk-management task.
  • Deep Learning in Investment: Financial investment problems require models that integrate multiple data sources and handle unusual feature interactions for efficient forecasting.Table 10 summarizes recent deep-learning applications in financial investment and stock-price prediction.
  • Deep Learning in Retail: Augmented reality is used in retail to improve customers’ experiences when buying or finding products in physical stores.The review identifies augmented reality as a frequently used approach and summarizes notable retail studies in Table 11.
  • Deep Learning in Retail: A mobile application combined deep learning image classification with augmented reality to locate clients and provide product information in large stores.The approach was designed to help clients find selected products and access related information.
  • Deep Learning in Retail: The proposed retail model performed better than common CNN methods on both datasets and was described as robust.The comparison is reported in Fig. 18 for the study by De Sousa Ribeiro et al.
  • Deep Learning in Retail: An adaptable CNN framework was developed to identify product use-by dates automatically and reduce mislabeling risks in food retail.The framework combines k-means and k-nearest-neighbor methods with CNNs to separate and adapt calculated centroids.

Deep Learning in Business (Intelligence) … Discussions

The review surveys deep learning and deep reinforcement learning for business intelligence and economics, emphasizing scalability, high-dimensional problem handling, market constraints, risk management, and profitability. It concludes that DL and DRL offer comparable prediction quality, while DRL better accommodates realistic market constraints and scalability demands.

  • Deep Learning in Business (Intelligence): DL techniques are presented as useful for complex business-intelligence problems involving market data and big-data-driven business services.Table 12 summarizes notable deep-learning studies in business intelligence.
  • Deep Reinforcement Learning Application in Economics: DRL captures substantial market conditions while offering scalability and the ability to handle high-dimensional economic problems.These capabilities motivate the review of recent deep-reinforcement-learning applications in economics and financial markets.
  • Deep Reinforcement Learning in Stock Trading: Traditional trading methods are difficult to test with transaction costs, while conventional RL lacks sufficient scalability for high-dimensional stock-market strategies.Financial companies seek optimal strategies in dynamic, complicated trading environments to maximize revenue.
  • Discussions: DL and DRL models are compared across economic domains using complexity, accuracy, speed, datasets, detection properties, profitability, revenue, and risk management.Table 13 provides the comparative study of DL and DRL models in economics.
  • Discussions: Both DL and DRL algorithms support prediction with almost the same statistical prediction quality across the reviewed economic applications.The discussion compares their broader performance across multiple economic domains.
  • Discussions: AE for risk management and LSTM-SVR for investment can considerably increase revenue while maintaining risk constraints and reasonably high performance.Reinforcement learning also simulates more efficient models under realistic market constraints, while deep RL addresses RL scalability limitations.

Conclusion · Acronyms

The conclusion emphasizes that DL and deep RL can improve economic forecasting, strategy detection, model performance, and accuracy relative to classical ML approaches. The acronym section defines recurring methods and concepts used throughout the paper.

  • Conclusion: DL can forecast and detect complex market trends, while DL and deep RL techniques offer different strengths and weaknesses for economic applications.These methods are presented as mechanisms for improving productivity, product quality, and market-strategy detection.
  • Conclusion: DL and deep RL approaches are reported to perform better than classical ML approaches on real-world economic problems.The conclusion also links these approaches to improved model performance and accuracy.
  • Acronyms: The acronym list defines AE as Autoencoder, ARIMA as Autoregressive Integrated Moving Average, and CNN as Convolutional Neural Network.It also includes DBNs, DDPG, DDQN, DDRL, DL, DNN, DPG, DQN, and DRL.
  • Acronyms: The acronym list includes BI for Business Intelligence, CNNR for Convolutional Neural Network Regression, and NLP for Natural Language Processing.It also identifies PCA, PDE, PG, PPO, PVM, RBM, RCNN, and RL among the paper’s terminology.
  • Acronyms: The acronym list defines OSBL as Online Stochastic Batch Learning, PCA as Principal Component Analysis, and PPO as Proximal Policy Optimization.These terms appear alongside reinforcement-learning, neural-network, and economic-modeling abbreviations.
  • Acronyms: The acronym list includes RNN for Recurrent Neural Network, R-NN for Random Neural Network, RS for Risk Score, and RTB for Real-Time Bidding.It also lists SAE, SPF, and SPG among the paper’s abbreviations.
Loading 2004.01509v1…