Source-linked AI summary
Recent Advances in Reinforcement Learning in Finance
Ben Hambly, Renyuan Xu, Huining Yang
TL;DR
Financial decision-making methods face a trade-off between tractable but potentially oversimplified models and realistic models that can become mathematically and computationally intractable. This survey introduces RL theory and algorithms, evaluates recent theoretical results, and reviews applications across major financial problems.
Problem
Classical financial decision methods must balance tractability and applicability, since simple models may oversimplify markets while realistic models can be mathematically and computationally intractable.
Method
The survey provides a systematic introduction to Markov decision processes, RL algorithms, performance evaluation, and applications across several financial decision-making problems.
Results
The survey summarizes recent RL theory, including convergence guarantees for value- and policy-based methods, and reviews applications in optimal execution, portfolio optimization, and other financial problems.
Takeaways & Limitations
RL offers a framework for financial decisions that can incorporate market conditions more flexibly than pre-planned strategies based on restrictive dynamics and price-impact assumptions.
Takeaways & Limitations
REINFORCE can suffer from high-variance policy-gradient estimates and requires complete trajectories before collecting episode data.
Abstract
from arXiv · showhide
The rapid changes in the finance industry due to the increasing amount of data have revolutionized the techniques on data processing and data analysis and brought new theoretical and computational challenges. In contrast to classical stochastic control theory and other analytical approaches for solving financial decision-making problems that heavily reply on model assumptions, new developments from reinforcement learning (RL) are able to make full use of the large amount of financial data with fewer model assumptions and to improve decisions in complex financial environments. This survey paper aims to review the recent developments and use of RL approaches in finance. We give an introduction to Markov decision processes, which is the setting for many of the commonly used RL approaches. Various algorithms are then introduced with a focus on value and policy based methods that do not require any model assumptions. Connections are made with neural networks to extend the framework to encompass deep RL algorithms. Our survey concludes by discussing the application of these RL algorithms in a variety of decision-making problems in finance, including optimal execution, portfolio optimization, option pricing and hedging, market making, smart order routing, and robo-advising.
1 Introduction
The survey motivates reinforcement learning as a data-driven alternative to model-dependent financial decision methods and reviews its theory, algorithms, and applications across finance.
- Classical stochastic-control models trade tractability and implementability against realistic market representation, which can make strategies sub-optimal in practice.More realistic models may become mathematically and computationally intractable.
- Large transaction, quote, and order-flow datasets have created new opportunities and challenges for financial modeling, motivating reinforcement-learning approaches.RL is presented as a way to use this information in contrast to classical stochastic control.
- The survey reviews a broad spectrum of reinforcement learning activity rather than focusing on a specialized application or algorithm family.
- It introduces Markov decision processes, value-based and policy-based methods, deep reinforcement learning, and applications including execution, portfolios, hedging, market making, routing, and robo-advising.
2 The Basics of Reinforcement Learning
The paper introduces reinforcement learning as goal-directed learning from interaction and presents its historical foundations, modern deep-learning extensions, and finance-oriented theoretical framework.
- Reinforcement learning formalizes agent-environment interaction through states, actions, and rewards without requiring exemplary supervision or complete environment models.
- Modern RL combines trial-and-error learning with optimal-control ideas based on value functions and dynamic programming.The Bellman equation is identified as a foundation of algorithms including Q-learning and Actor-Critic.
- Deep learning and functional approximation have extended RL to diverse tasks in discrete and continuous spaces and in real or simulated environments.
- Classical algorithms such as temporal-difference learning and Q-learning solve small-scale problems under enumerable-state or restricted-function assumptions.
- The section establishes MDPs, unknown-dynamics learning procedures, performance criteria, and model-free RL algorithms as the survey’s foundational framework.
2.1 Setup: Markov Decision Processes
The survey formulates financial decision problems as finite- or infinite-horizon Markov decision processes, then describes policies, value equations, approximation classes, and key scope assumptions.
- Finite Time Horizon: Finite-horizon MDPs model short-term problems with time-dependent transitions and rewards, terminal rewards, and policies that may be deterministic or randomized.Optimal execution is given as a financial example, while portfolio optimization commonly uses long horizons.
- Infinite Time Horizon: Infinite-horizon discounted MDPs optimize expected discounted return by selecting actions through a policy based on the current state.
- Dynamic Programming: The Bellman and Q-function equations characterize optimal value and action-value quantities, with the optimal policy selecting actions that maximize Q(s, a).
- Scope: The survey does not cover infinite-horizon average-reward, or ergodic-reward, settings because they are considered less relevant to financial applications.
- Financial Modeling: Financial applications particularly require time-varying transition kernels and reward functions because financial time-series data are typically non-stationary.
- Optimal Policies: For finite state and action spaces with bounded rewards, discounted infinite-horizon MDPs admit an optimal deterministic stationary policy.
- Functional Approximation: Linear MDPs and linear functional approximations represent transitions, rewards, or value functions through feature mappings and unknown parameters.The survey notes equivalence between the two formulations under mild conditions.
- Functional Approximation: Nonlinear functional approximation, especially neural networks, leads to deep RL and can potentially address functional-space misspecification.
2.2 From MDP to Learning
The paper turns MDPs with unknown dynamics or rewards into reinforcement-learning problems, emphasizing agent interaction, exploration, simulators, and performance evaluation.
- From MDP to Learning: When transition dynamics and rewards are unknown, RL seeks an optimal policy while learning them explicitly or implicitly through model-based or model-free approaches.
- Agent-Environment Interface: Agents interact with environments through sequential state, action, reward, and next-state samples that form the history used by RL algorithms.
- Exploration vs Exploitation: Exploration can improve a sub-optimal solution but consumes time and computation, whereas over-exploration may impair convergence.
- Simulator: Online exploration is challenging because the next state follows the preceding action, and ε-greedy Q-learning may require exponentially many episodes.
- Simulator: A simulator permits arbitrary state-action queries and restarts, substantially alleviating exploration difficulty.
- Policy Choice: Randomized policies are commonly used to encourage exploration even when deterministic optimal policies exist in discounted infinite-horizon MDPs.
- Performance Evaluation: Performance is assessed using sample complexity, convergence rate, regret, and asymptotic convergence across episodic and infinite-horizon settings.
2.3 Value-based Methods
Value-based methods learn value functions from observed interactions when reward and transition models are unknown. The section introduces temporal-difference learning, Q-learning, and SARSA, alongside convergence and sample-complexity results.
- Value-based methods: Value-based RL operates in finite state and action spaces with unknown reward and transition functions, learning from observed samples rather than explicit system models.The section presents this as the tabular setting and motivates sample-based value-function estimation.
- Temporal-difference learning: Temporal-difference learning updates value estimates using observed transitions, with the learning rate balancing the current estimate against a new reward-based estimate.The TD error measures the difference between the current estimate and the target formed from reward plus discounted successor value.
- Q-learning and SARSA: Q-learning updates Q-values toward the observed reward plus the discounted maximum next-action value, making it a stochastic approximation to the Bellman equation.Its samples may be collected under any policy, distinguishing it from on-policy methods such as SARSA.
- Convergence and sample complexity: Q-learning converges asymptotically to the optimal Q-function with probability 1 under bounded rewards, finite state and action spaces, and suitable learning-rate conditions.The result does not specify how many samples are required to reach a given accuracy; later theory studies sample complexity.
- Q-learning and SARSA: SARSA selects the next action using the agent’s current ε-greedy policy, whereas Q-learning uses the maximum next-action value without that exploratory choice.The ε-greedy policy chooses a maximizing action with probability 1 − ε and samples uniformly from the action set with probability ε.
- Convergence and sample complexity: Recent value-based RL theory studies sample complexity, including settings where transition kernels are approximately linear with small errors to address model misspecification.The cited result is expressed in terms of the feature dimension d, although the supplied passage truncates its exact order.
2.4 Policy-based Methods
Policy-based reinforcement learning methods directly parameterize and optimize policies, offering an alternative when value-based methods become computationally difficult in large or continuous spaces. The section covers policy-gradient estimation, REINFORCE, Actor-Critic methods, stabilizing variants, and convergence results.
- Policy-based methods: Policy-based methods directly parameterize a policy instead of learning a value function, which can be more effective as state or action spaces become large or continuous.The section considers model-free policy-based methods with stationary policies, including REINFORCE, Actor-Critic, TRPO, and PPO.
- Policy-gradient methods: Policy-gradient methods update policy parameters by ascending an estimated gradient of the objective, under differentiability and stationary-distribution assumptions.The policy-gradient theorem requires a differentiable policy and, in the stated form, a stationary distribution independent of the initial state.
- REINFORCE: REINFORCE estimates policy gradients from complete sampled trajectories using the discounted return G_t as an approximation to the Q-function.The algorithm repeatedly simulates finite trajectories and updates the policy parameter using returns at each time step.
- REINFORCE: REINFORCE provides an unbiased policy-gradient estimate but has high variance and may converge slowly; subtracting a value-function baseline reduces variance without changing the mean.Because it uses complete trajectories, REINFORCE also waits until the end of each episode to collect data.
- Actor-Critic methods: Actor-Critic methods learn both a value or Q-function and a policy, allowing critic information to improve policy updates and enabling learning from incomplete experience online.The critic updates value-function parameters while the actor updates policy parameters; nested-loop, two-time-scale, and single-scale executions are described.
- Variants and convergence: Policy-based variants target faster, more stable, or less sample-intensive learning, while theory establishes local and non-asymptotic convergence guarantees for several methods.Examples include Fisher-information updates, KL-based or clipped objectives, deterministic policies, experience replay, and convergence analyses for Actor-Critic, REINFORCE, TRPO, and related algorithms.
2.5 General Discussion
The survey covers finite- and infinite-horizon RL, including model-based and model-free methods, while emphasizing challenges from exploration, nonstationarity, and high-dimensional spaces. It reviews theoretical guarantees and approaches intended to reduce dependence on state-action-space size.
- Finite- and infinite-horizon RL: Finite-horizon RL includes both model-based and model-free methods, while infinite-horizon RL includes model-free and model-based cases.The survey discusses performance through sample complexity and regret analysis.
- Theoretical guarantees: Regret and sample-complexity bounds for many algorithms depend on the sizes of the state and action spaces, creating a curse of dimensionality.The survey notes this dependence across several finite-horizon results.
- Theoretical guarantees: MatrixRL addresses this issue by learning a low-dimensional transition-model representation, yielding a regret bound that depends on the number of features d rather than |S| and |A|.This replaces direct dependence on state and action-space sizes with feature dimension in the stated bound.
- Exploration: Exploration remains challenging because committing too quickly can lead to local minima or failure, even when exploitation is efficient.The survey contrasts strong return optimization with less-understood efficient exploration.
- Exploration: Classical exploration methods include ε-greedy, UCB, Boltzmann exploration, and Thompson sampling, while deep RL often uses entropy regularization or noise-based exploration.These methods alter action selection, objectives, observations, actions, or parameters to support exploration.
- Nonstationary RL: Financial environments are often nonstationary, motivating formulations for episodic and infinite-horizon non-stationary RL.Existing theoretical work commonly assumes prior knowledge of the degree of nonstationarity, which the survey describes as impractical for many applications.
3 Deep Reinforcement Learning
Deep RL uses neural-network function approximations to handle high-dimensional or continuous financial state and action spaces. The survey introduces feed-forward, convolutional, and recurrent architectures and reviews convergence results for neural RL algorithms.
- Motivation: Neural-network parameterizations make value functions and policies more practical when state or action spaces are large, high-dimensional, or continuous.The survey focuses on parametrized Q-functions, V-functions, and policies.
- Motivation: Neural networks suit time-series inputs and can be trained incrementally as additional samples become available.The survey identifies these as advantages for financial applications.
- Neural architectures: The survey introduces fully-connected, convolutional, and recurrent neural networks as architectures used in deep RL.CNNs process spatially structured inputs, while RNNs process sequential data such as financial time series.
- Neural architectures: CNNs extract features through trainable convolutional filters and reduce computational cost through pooling layers.Max pooling aggregates information by taking the maximum over a local spatial neighborhood.
- Neural architectures: RNNs use internal memory and recursive hidden states to process arbitrary input sequences, while LSTMs and GRUs improve handling of long-term dependencies.In LSTMs, gates regulate information transferred into the cell state and revealed through the hidden state.
- Convergence guarantees: DQN approximation error comprises statistical and algorithmic components, with the algorithmic error decaying geometrically during training.The statistical component reflects bias and variance from neural-network Q-function approximation.
- Convergence guarantees: Neural PPO, TRPO, and Actor-Critic methods have sublinear convergence results under the surveyed parameterizations, whereas DDPG lacks an established convergence guarantee.For one Actor-Critic variant, stationary points are globally optimal under mild regularity conditions.
4 Applications in Finance
The survey reviews reinforcement-learning applications across electronic-market finance, emphasizing decisions that can use market data and adapt beyond fixed model assumptions. It covers optimal execution, portfolio optimization, option pricing and hedging, market making, smart order routing, and robo-advising.
- Electronic markets: Electronic markets, including limit order books, provide the setting for many reviewed financial decision-making problems.The survey discusses central limit order books and electronic over-the-counter markets before introducing applications.
- Optimal execution: Closed-form optimal-execution strategies can depend heavily on assumed dynamics and linear price impacts, whereas RL can incorporate real-time market conditions.The survey notes that model misspecification may produce undesirable strategies and that pre-planned strategies can miss market opportunities.
- Optimal execution: RL methods for optimal execution report improvements across trading-cost, implementation-shortfall, PnL, and benchmark comparisons.Examples include up to 10.3% average improvement in Implementation Shortfall, outperformance of TWAP on seven of nine stocks, and lower costs during the 2010 flash crash.
- Other applications: The reviewed applications extend beyond execution to portfolio optimization, option exercise and hedging, market making, smart order routing, and robo-advising.Reported examples include larger American-option payoffs than Longstaff-Schwartz, improved market-maker reward-to-risk performance, and consistent outperformance of the S&P 500 benchmark by a robo-advising pipeline.
- Other applications: Robo-advising can combine inverse portfolio optimization with deep RL to infer preferences and formulate multi-period allocation decisions.The two-agent pipeline uses historical allocation data to infer risk preferences and expected returns before applying DDPG-based deep RL.
5 Further Developments for Mathematical Finance and Reinforcement Learning
The section identifies open challenges for applying reinforcement learning in finance, including risk sensitivity, limited exploration, offline learning, sample efficiency, transfer, and multi-objective decisions. It also highlights opportunities to combine stochastic-control domain knowledge with RL learning capacity and to extend RL to underexplored financial settings.
- Integrated Modeling: Integrating stochastic-control domain knowledge with RL could incorporate application-specific risks while retaining the learning power of RL algorithms.Examples include adverse-selection risk in market making and execution risk in optimal liquidation.
- Risk-Aware RL: Risk-sensitive RL must account for financial risks such as drawdown, variance, or lower-tail PnL while addressing nonlinear objectives and risk-aware exploration.Existing work includes risk-sensitive temporal-difference and Q-learning methods, constrained RL, and robust risk-aware frameworks.
- Offline Learning and Exploration: Offline learning is a plausible alternative to real-time updating because financial algorithms can collect data during trading and update after the market closes.Existing offline regression and batch-RL methods are described as general methodologies rather than finance-specific solutions.
- Offline Learning and Exploration: Exploration requires careful budgeting because it can consume substantial time and computation, impose financial costs, and require institutional justification for black-box strategies.This motivates methods that improve policies under constrained exploration effort.
- Other Open Directions: Further directions include model-free order routing across lit and dark pools, unconstrained robo-advising, and RL methods for balancing expected return against portfolio risk.These settings involve contrasting market mechanisms, broader strategy choices, or conflicting investment objectives.
- Data Efficiency and Transfer: Financial RL still faces large sample requirements, non-stationary time series, limited historical relevance, and unresolved transfer or cold-start problems for new assets.The survey highlights transferring experience between similar assets and initializing strategies for newly issued assets with limited data.