Source-linked AI summary
Reinforcement Learning in Economics and Finance
Arthur Charpentier, Romuald Elie, Carl Remlinger
TL;DR
Reinforcement learning addresses sequential decision problems in which actions affect both rewards and future states, making long-term policy learning important. The paper synthesizes reinforcement learning techniques and applications across economics, games, operations research, and finance, while identifying computational and practical limits. Its applications include efficient auction algorithms and learning-based investment strategies, but financial deployment is constrained by unrealistic simulators, nonstationary markets, and difficult-to-control black-box policies.
Problem
Sequential decision problems require learning from feedback when actions affect rewards and future states, but many economic and financial problems face computational difficulties.
Method
The paper presents a state-of-the-art synthesis of reinforcement learning techniques and applications in economics, game theory, operations research, and finance.
Results
The reviewed applications include efficient automated auction algorithms, learned investment strategies, game-learning results, and reinforcement learning approaches to complex economic and financial problems.
Takeaways & Limitations
Recent computational advances make reinforcement learning a promising framework for analyzing complex economic and financial problems.
Takeaways & Limitations
Financial applications are limited by unrealistic market simulators, nonstationary dynamics, high computational demands, and poorly interpretable black-box policies.
Abstract
from arXiv · showhide
Reinforcement learning algorithms describe how an agent can learn an optimal action policy in a sequential decision process, through repeated experience. In a given environment, the agent policy provides him some running and terminal rewards. As in online learning, the agent learns sequentially. As in multi-armed bandit problems, when an agent picks an action, he can not infer ex-post the rewards induced by other action choices. In reinforcement learning, his actions have consequences: they influence not only rewards, but also future states of the world. The goal of reinforcement learning is to find an optimal policy -- a mapping from the states of the world to the set of actions, in order to maximize cumulative reward, which is a long term strategy. Exploring might be sub-optimal on a short-term horizon but could lead to optimal long-term ones. Many problems of optimal control, popular in economics for more than forty years, can be expressed in the reinforcement learning framework, and recent advances in computational science, provided in particular by deep learning algorithms, can be used by economists in order to solve complex behavioral problems. In this article, we propose a state-of-the-art of reinforcement learning techniques, and present applications in economics, game theory, operation research and finance.
1 Introduction
Reinforcement learning studies how agents use repeated experience to learn long-term optimal decisions, linking behavioral learning with sequential control. The paper connects this framework to economics, games, operations research, and finance.
- Historical and conceptual foundations: Reinforcement learning explains how agents learn optimal decisions through repeated experience and maximize cumulative discounted rewards.It also addresses how actions, states, and rewards are matched over time.
- Formal setup: The framework models an agent moving from state s_t to s_t+1 after taking action a_t and receiving reward r_t, with policies mapping states to actions.The objective is to learn an optimal policy from past actions and rewards.
- Historical and conceptual foundations: Actions can have delayed consequences, requiring agents to sacrifice immediate rewards for larger long-term returns.This sequential structure distinguishes reinforcement learning from simple stimulus-response learning.
- Connections to related fields: Because actions are interventions with consequences, reinforcement learning has strong connections to causality modeling.The framework can be used to infer consequences of past actions.
- Applications: Reinforcement learning has applications in economic modeling, operations research, game theory, and finance.The paper discusses consumption dynamics, traveling-salesman problems, stochastic games, auctions, risk management, portfolios, and market making.
2 From Machine to Reinforcement Learning
The paper develops reinforcement learning from machine learning, online learning, and bandits by emphasizing sequential decisions, feedback, and regret. It presents optimization objectives and the exploration–exploitation problem underlying incomplete-information decisions.
- From machine learning to reinforcement learning: Machine learning frames learning as optimizing a loss or reward, while reinforcement learning extends this to sequential decisions with feedback and changing consequences.Unlike fixed-data learning, reinforcement learning involves active experimentation.
- Supervised learning: Supervised learning maps inputs to actions using losses, risks, empirical risk minimization, and an oracle target defined over a function class.The training objective is evaluated against unseen observations while avoiding over-fitting.
- Regret: Regret measures the cumulative loss or reward gap between the learner’s sequence of actions and an optimal oracle or achievable alternative.For reward maximization, regret is the difference between what could have been achieved and what was actually achieved.
- Online learning: Online learning updates predictions sequentially as observations arrive, using past data to produce the next prediction and then observing its loss.The observations need not be i.i.d.
- Bandits: Multi-armed bandits require balancing exploration of uncertain options against exploitation of the empirically best option.Bandits are a special case of reinforcement learning because actions do not affect the agent’s state.
3 Reinforcement Learning
This section introduces reinforcement learning principles and possible extensions before turning to applications and known difficulties of the models.
- Scope: The section focuses on common reinforcement learning approaches, their links with economics and finance, and known difficulties.It prepares the discussion of principles and extensions.
3.1 Mathematical context
The mathematical framework represents sequential decision problems as Markov decision processes and seeks policies maximizing expected discounted rewards. Value and Q-functions characterize policy quality, while exploration is required when rewards or transitions are unknown.
- Markov decision processes: A Markov decision process consists of state and action spaces, transition and reward functions, and a discount factor γ.A policy π maps states to actions.
- Value functions: The value of a state under a policy is its expected reward from that state, while the Q-value evaluates taking a particular action before following the policy.These quantities support policy comparison and optimization.
- Optimal policies: The optimal policy π⋆ assigns each state its highest-valued action, making computation of π⋆ the central solution task.Knowing the Q-function directly provides the derivation of an optimal policy.
- Finite-state representation: For finite state and action spaces, Q-values can be represented as a vector of size |S||A|, with transition constraints depending on the policy.The matrix representation tracks transitions from (s, a) to (s′, π(s′)).
- Learning and exploration: When reward and transition functions are unknown, agents must explore through actions that generate feedback usable for learning.Exploration strategies include ε-greedy, visit-based exploration, and Boltzmann selection.
3.2 Some Dynamical Programming principles
Dynamic programming evaluates and improves policies using value functions, but classical procedures assume a known environment and may be computationally lengthy. Reinforcement learning addresses unknown rewards and transitions by estimating them from observed or simulated experience.
- Value functions support action selection and optimal-policy construction in dynamic programming and reinforcement learning.
- Policy iteration repeatedly evaluates a policy, improves it by selecting higher-value actions, and continues until an optimal policy is found.The procedure starts from an initial policy π0 and alternates value computation with improvement.
- Known transition and reward functions permit direct solution of value equations, whereas real-world applications often require sampling to learn the model.The paper notes that full information is not always available and that policy iteration can be very long.
- Parametric approximations represent the unknown action-value function with basis functions and estimate coefficients through least squares, minimizing Bellman residuals.The resulting system is over-constrained, motivating a least-square solution.
- Least-square policy iteration uses observed state-action-reward samples and basis functions to approximate policy values when rewards and transitions are unknown.With uniformly observed states and actions, the estimated coefficients converge toward their population counterparts.
- Model-based learning estimates transition and reward functions from simulated experience, with empirical estimates converging to the true functions as generations increase.This convergence follows from the law of large numbers.
3.3 Some Solution Methods
The paper surveys common reinforcement-learning solution methods, including Q-learning, policy-gradient learning, and actor-critic architectures. These methods differ in whether they learn values, policies, or both, and in their treatment of action spaces and environments.
- Q-learning: Q-learning is an off-policy temporal-difference control method based on Bellman optimality and iterative updates from an initial Q-function.
- Q-learning: Neural networks can approximate the Q-function in Q-learning.
- Policy Gradient: Policy learning directly estimates a parameterized mapping from states to actions rather than comparing expected returns for every action.The approach is described as converging faster than value-based learning and supporting continuous action spaces.
- Actor-Critic: Actor-critic methods combine an actor that updates policy parameters with a critic that updates a value or action-value function.The combination is presented as joining policy-learning speed and continuous-action handling with value-learning sample efficiency and stability.
3.4 Inverse Reinforcement Learning
Inverse reinforcement learning infers the reward function from observed decisions and then uses reinforcement-learning techniques to recover an optimal policy. The discussion connects this framework to structural economic models of choice and highlights their computational complexity.
- Economic applications include job matching, patent obsolescence, fertility and child mortality, investment, and bus-engine replacement decisions.
- Dynamic discrete-choice models use conditional choice probabilities over a finite action set to represent which action is optimal in each state.
- Under Gumbel-distributed rewards, the model becomes multinomial logit, with log-odds ratios proportional to the value function.The bus-repair example represents operating or replacing a bus, with operating and replacement costs entering utility.
- The mapping between conditional choice probabilities and choice-specific value functions is invertible.
- Structural estimation in observed bus-maintenance decisions is computationally complex.
- Inverse reinforcement learning learns a reward function from an agent’s decisions before finding the policy that maximizes that reward.
4.1 Applications in Economic Modeling
Reinforcement learning provides a framework for economic models involving dynamic decisions, uncertainty, bounded rationality, and learning. The section applies recursive methods to consumption and income dynamics and relates reinforcement learning to multi-agent economics, firm learning, and adaptive experimentation.
- Reinforcement learning is connected to longstanding computational-economics work and more recent surveys of applications to economic problems.
- Consumption and Income Dynamics: Consumption and income dynamics model an infinitely living agent choosing consumption under stochastic income and a nonnegative-wealth constraint.
- Consumption and Income Dynamics: Under the stated regularity conditions, contraction mapping yields a unique value-function solution and a decision rule prescribing the best action in each state.
- Consumption and Income Dynamics: For any policy, value evaluation can use the matrix formula vπ = (I_n − γTπ)^−1uπ, after which the optimal value is the maximum across policies.
- Economic models study learning through experience, imitation, Bayesian updating, and adaptive processes when agents face uncertainty or multiple equilibria.
- Bounded-rationality research includes limits on thinking time, memory, and model simplification, while sample complexity measures observations needed for near-optimal action.
- Reinforcement learning is applied to firms’ intertemporal investment decisions, sequential energy markets, and experiments that adapt treatment assignment across waves.
4.2 Applications in Operations Research and Game Theory
Reinforcement learning is applied to sequential optimization problems, graph algorithms, auctions, and strategic multi-agent settings. These applications also expose challenges from nonstationarity, equilibrium selection, and scalability.
- Operations research: 2-opt and related heuristic procedures can be formulated within the reinforcement learning framework, including short-term exploratory moves.The 2-opt procedure iteratively replaces two edges, while other approaches include Christofides and ant-colony-inspired methods.
- Operations research: Reinforcement learning frames sequential graph optimization problems such as TSP through states, actions, and rewards.For TSP, states are edge subsets forming a cycle, actions permute nodes, and rewards reflect changes in the loss function.
- Operations research: Neural-network methods can complement or replace heuristic procedures for combinatorial optimization despite computational expense.Deudon et al. combine machine learning with existing heuristics, while Bello et al. replace the heuristic procedure with neural networks.
- Game theory: A one-parameter reinforcement learning model robustly outperforms equilibrium predictions in many games.Adding forgetting and valuing experimentation improves predictive power; multi-agent reinforcement learning nevertheless lacks scalability as the number of agents grows.
- Game theory: In multi-agent games, each player’s opponents become part of the environment, making the environment nonstationary during simultaneous learning.Single-agent algorithms may cycle among equilibria rather than converge, while stabilizing procedures such as fictitious play can reach equilibria in some settings.
- Auctions and real-time bidding: Deep reinforcement learning uses neural networks to handle the scale of real-time bidding and produces efficient auction algorithms that outperform more classical ones.Auction states include auction information and campaign parameters, while the action is the bid price.
4.3 Applications in Finance
Finance applications use reinforcement learning for risk management, derivative valuation and hedging, portfolio allocation, and market microstructure. The approach offers model-free approximation possibilities but remains constrained by simulator assumptions and changing market dynamics.
- Risk management and derivatives: High-dimensional derivative valuation and hedging with realistic market frictions lack efficient numerical approximations and tractable analytical solutions.Reinforcement learning is introduced as a model-free numerical approximation approach using simulated market trajectories and neural networks.
- Risk management and derivatives: Financial-market simulators still require model-driven assumptions, including a volatility-model class and its calibration.Model-free simulator designs based on generative methods are described as recent attempts to reduce this dependence.
- Portfolio allocation: Reinforcement learning is applied to dynamic investment strategies, although simpler online or bandit methods may suffice for some portfolio-allocation problems.The distinction becomes less applicable when investors significantly affect financial-asset dynamics.
- Market microstructure: At high trading frequencies or large volumes, reinforcement learning can optimize the dynamic use of market and limit orders using order-book dynamics.Because order-book dynamics aggregate other traders’ actions, the setting is more precisely a multi-agent learning problem.
- Market microstructure: Financial-market applications face two main practical drawbacks: realistic simulator design and nonstationary dynamics that limit transfer from past to future markets.Black-box policies may also be difficult to interpret and control in aggregate market use.
5 Conclusion
Deep reinforcement learning has become a prominent approach for learning near-optimal strategies by experience, with applications to autonomous systems and economic behavior. Its promise is tempered by substantial information and computing requirements, as well as finance’s nonstationarity.
- Conclusion: Deep reinforcement learning is presented as a popular technique for artificial agents to learn near-optimal strategies through experience.The paper mentions applications including self-driving cars and modeling animal, consumer, and investor behavior.
- Conclusion: Recent computational advances make it promising to revisit complex economic and financial problems using reinforcement learning.The paper connects this promise to earlier economic groundwork whose progress was slowed by computational difficulties.
- Conclusion: Deep reinforcement learning performs well when substantial information is available, but exploration can require very large amounts of computing power.The paper also notes growing popularity in finance despite the market’s lack of stationarity.