Source-linked AI summary
Market Making via Reinforcement Learning
Thomas Spooner, John Fearnley, Rahul Savani, Andreas Koukorinis
TL;DR
One-sided order execution can leave market makers with inventory, so the paper develops reinforcement-learning agents using a realistic market simulation. The consolidated agent achieves competitive out-of-sample performance across securities and superior risk-adjusted performance.
Problem
Because only one of a market maker’s two orders may execute, quoting on both sides can leave it with non-zero inventory.
Source: §2.2 · p. 2Method
The paper develops reinforcement-learning market-making agents using a realistic simulation and separate value functions for agent, market, and full-state representations.
Source: §4.3 · p. 4, §6 · p. 8Results
The agent produces competitive out-of-sample performance across a basket of securities and superior risk-adjusted performance.
Source: §6 · p. 8Takeaways & Limitations
Separate agent-, market-, and full-state value functions enable faster inventory-value learning without observing every full-state permutation.
Source: §4.3 · p. 4Aggregate limit-order data do not reveal precisely which orders were cancelled when volume at a price level decreases.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Market making is a fundamental trading problem in which an agent provides liquidity by continually offering to buy and sell a security. The problem is challenging due to inventory risk, the risk of accumulating an unfavourable position and ultimately losing money. In this paper, we develop a high-fidelity simulation of limit order book markets, and use it to design a market making agent using temporal-difference reinforcement learning. We use a linear combination of tile codings as a value function approximator, and design a custom reward function that controls inventory risk. We demonstrate the effectiveness of our approach by showing that our agent outperforms both simple benchmark strategies and a recent online learning approach from the literature.
1 INTRODUCTION
The paper applies temporal-difference reinforcement learning to market making and addresses prior challenges involving reward attribution, noise, and partial observability. It reports a consolidated agent with the best risk-adjusted out-of-sample performance against simple benchmarks, a basic RL agent, and a recent online-learning approach.
1.1 Related work.
- Market making has been studied through optimal control, zero-intelligence agents, prediction markets, and reinforcement learning, with prior RL work also examining controlled environments and online learning.Earlier RL work did not capture explicit order placement and cancellation or continuous state variables; the online-learning approach assumes executions at posted prices without slippage.Source: §1.1 · p. 1
1.2 Our contributions.
- The simulation reconstructs a limit order book from data spanning 10 equities across five venues, five levels of depth, and millisecond-scale updates.The data cover a mixture of sectors.Source: §1.2 · p. 2
- Eligibility traces address earlier concerns about temporal-difference learning, which prior work found struggled in noisy, partially observable market-making settings.The paper also evaluates a range of TD-based learning algorithms.Source: §1.1 · p. 1, §1.2 · p. 2
- The natural profit-and-loss reward does not yield the best performance and can destabilize learning, motivating an asymmetrically dampened reward.The proposed reward is reported to improve learning stability and produce higher, more consistent returns.Source: §1.2 · p. 2
- A linear combination of tile codings provides the final state representation, with competitive performance and more stable learning.The paper evaluates three state-space constructions.Source: §1.2 · p. 2
- The consolidated agent achieves the best risk-adjusted out-of-sample performance against simple benchmarks, a basic RL agent, and a recent online-learning approach.The agent combines the strongest variants investigated in the paper.Source: §1.2 · p. 2
2 PRELIMINARIES
A limit order book organizes buy and sell orders by price level, while market makers quote on both sides to capture spreads. One-sided execution can leave a market maker with inventory exposure.
2.1 Limit order books.
- A limit order book aggregates limit orders at fixed price levels, and the gap between adjacent levels is the tick size.A limit order specifies a direction, price, and volume.Source: §2.1 · p. 2
- Market orders execute against the best-priced opposing limit orders and may spill into further price levels when the best level lacks volume.Limit orders join the back of the queue at their quoted price and execute first-come, first-served.Source: §2.1 · p. 2
- The market spread is the difference between the best bid and ask, and the mid-price is their arithmetic mean.The cited example gives a best bid of 100.00, an ask of 100.50, a mid-price of 100.25, and a spread of 0.5.Source: §2.1 · p. 2
2.2 Market making.
- A market maker quotes both sides to provide liquidity and may earn the quoted spread if both orders execute, but a one-sided fill can leave nonzero inventory.An idealized market maker seeks to capture spread while keeping inventory low to limit market exposure.Source: §2.2 · pp. 2–3
3 THE SIMULATOR
The simulator reconstructs a limit order book from historical market data and allows agents to place orders within the top five price levels. Aggregate order data leave uncertainty about where cancellations occur in the queue.
3 THE SIMULATOR
- The simulator reconstructs the limit order book from historical data for 10 securities across four sectors and tracks the top five price levels.The data cover eight months of order-book depth and transaction records.Source: §3 · p. 3
- Because orders are simulated against a historical reconstruction, they cannot affect the market; the authors judge this negligible for their small orders relative to market trading volume.Source: §3 · p. 3
- Aggregate order-book data do not reveal which specific orders were cancelled, so the simulator cannot directly determine whether a cancellation was ahead of or behind its simulated order.It assumes cancellations are uniformly distributed through the queue, making the probability of an ahead cancellation proportional to the volume ahead.Source: §3 · p. 3
4 THE AGENT
The agent makes event-driven quoting decisions, with inventory constraints, and uses dampened PnL rewards and a state representation designed for market making. Its learning setup evaluates several temporal-difference algorithms and variants.
4.1 Trading strategy.
- The agent revises bid and ask quotes relative to the mid-price, with nine actions varying spread and bias and a tenth clearing inventory by market order.It acts on observable order-book events and restricts trading toward neutral inventory when its bounds are reached.Source: §4.1 · p. 3
4.2 Reward functions.
- Basic PnL rewards can destabilize learning and yield unsatisfactory out-of-sample performance, so dampened variants increase spread capture's reward relative to speculation.Symmetric dampening reduces speculative profits and losses; asymmetric dampening reduces speculative profits while leaving losses intact.Source: §4.2 · pp. 3–4
4.3 State representation.
- LCTC combines independently learned agent-state, market-state, and full-state values using fixed influences, with the influences summing to one.Independent value estimates let the agent evaluate inventory exposure without observing every full-state permutation; the market state is partially observable.Source: §4.3 · p. 4
4.4 Learning algorithms.
- The study evaluates Q-learning, SARSA, and R-learning alongside Double Q-learning, Expected SARSA, On-policy R-learning, and Double R-learning.The algorithms use eligibility traces unless otherwise stated, and include discounted and undiscounted methods.Source: §4.4 · pp. 4–5
5 RESULTS
The consolidated agent combines asymmetric reward dampening, an LCTC state space, and SARSA, achieving superior overall risk-adjusted performance to the basic agent and extensions. Results also show that reward design and state representation materially affect learning stability and performance.
5.1 Benchmarks.
- Fixed benchmark strategies with θa,b > 1 were just profitable on average but had high volatility and generally high mean absolute positions; MMMW beat follow-the-leader but underperformed its original-paper results.The paper suggests the latter difference may reflect its more realistic simulation, which models the limit order book in greater detail.Source: §5.1 · pp. 5–6, Table 3 · p. 6
5.2 Basic agent.
- One-step Q-learning and SARSA with the basic agent produced no useful results; eligibility traces yielded occasional out-of-sample profits but remained unsatisfactory.The basic agent uses agent-state variables and non-dampened PnL reward.Source: §5.2 · p. 6, Table 4 · p. 6
5.3 Extensions.
- Off-policy learning variants tended to perform worse than on-policy counterparts, while SARSA was more consistent across equities than other variants.Double Q-learning was less profitable and more volatile on most assets, with CRDI.MI noted as an exception.Source: §5.3 · p. 6
- Asymmetric dampening improved risk-adjusted performance in most cases; at η around 0.1, policies shifted toward small, neutral inventories and lower PnL variance.Learning stability also improved: reward standard deviation diverged at η = 0.0 and decreased for η ≥ 0.1.Source: §5.3 · pp. 6–7, Figure 2 · p. 6, Figure 3 · p. 7
- LCTC outperformed the full-state variant and avoided its divergence issues, combining expressivity and learning efficiency, though results depended on the chosen market variables.The full-state variant regularly degraded or diverged with increased training; LCTC performance varied across stocks.Source: §5.3 · p. 7, Figure 4 · p. 7
5.4 Consolidated agent.
- The consolidated agent achieved superior overall risk-adjusted performance to the basic agent and extended variants, with improved out-of-sample stability despite slightly lower returns than the best individual variants.It combines asymmetric reward dampening, an LCTC state space, and SARSA; its results were also superior to benchmarks overall.Source: §5 · p. 5, §5.4 · pp. 7–8, Table 6 · p. 8
6 CONCLUSIONS
The paper develops a competitive reinforcement-learning market-making agent with superior risk-adjusted performance, then identifies directions for future research.
6 CONCLUSIONS
- The consolidated agent produces competitive out-of-sample performance across a basket of securities and superior risk-adjusted performance.The conclusion identifies this as the outcome of combining the best learning algorithms, reward functions, and state representations.Source: §6 · p. 8
- A highly realistic simulation and eligibility traces address problems raised in past work.Source: §6 · p. 8
- Future work includes advanced learning algorithms, options learning, deep reinforcement learning, and recurrent neural networks.The proposed advanced algorithms include GreedyGQ, Q(σ), and true online variants [15].Source: §6 · p. 8
- Further directions include parameterised action spaces, multiple orders and variable order sizes, and studying rebates, fees, and latency.Source: §6 · p. 8
- The authors also propose sequential Bayesian methods for better order-book reconstruction and order-queue estimation.Source: §6 · p. 8
A APPENDIX
Table 7 reports out-of-sample normalised daily PnL and mean absolute positions for the FTL benchmark strategy derived from [2].
A APPENDIX
- Table 7 reports out-of-sample normalised daily PnL (ND-PnL) and mean absolute positions (MAP) for the FTL benchmark strategy derived from [2].Source: Table 7 · p. 10