Source-linked AI summary
Deep Reinforcement Learning for Trading
Zihao Zhang, Stefan Zohren, Stephen Roberts
TL;DR
The paper addresses limited evidence on converting financial predictive signals into actual trading positions. It applies deep reinforcement learning with volatility-scaled rewards and discrete or continuous actions, finding stronger performance than baseline strategies across futures markets despite heavy transaction costs.
Problem
Prior machine-learning trading research largely predicts returns or market movements, with little discussion of transforming those signals into actual trade positions.
Method
The study trains DQN, policy-gradient, and advantage actor-critic models to output positions directly, using discrete or continuous action spaces and volatility scaling.
Results
RL algorithms outperform baseline models and deliver positive profits across 50 liquid futures contracts from 2011 to 2019 despite heavy transaction costs.
Takeaways & Limitations
The experiments show that RL strategies can follow large market moves without changing positions and scale down or hold during consolidation periods.
Takeaways & Limitations
The paper leaves risk-averse utility functions and portfolio optimization for future work.
Abstract
from arXiv · showhide
We adopt Deep Reinforcement Learning algorithms to design trading strategies for continuous futures contracts. Both discrete and continuous action spaces are considered and volatility scaling is incorporated to create reward functions which scale trade positions based on market volatility. We test our algorithms on the 50 most liquid futures contracts from 2011 to 2019, and investigate how performance varies across different asset classes including commodities, equity indices, fixed income and FX markets. We compare our algorithms against classical time series momentum strategies, and show that our method outperforms such baseline models, delivering positive profits despite heavy transaction costs. The experiments show that the proposed algorithms can follow large market trends without changing positions and can also scale down, or hold, through consolidation periods.
1 Introduction
The paper frames trading as a sequential decision problem and applies reinforcement learning to output positions directly while incorporating market frictions. It evaluates several RL algorithms across liquid futures markets against time-series momentum baselines.
- Financial signals are difficult to design because financial data have low signal-to-noise ratios and markets are dynamic.
- Prior machine-learning research largely predicts returns or market movements, with limited work mapping those signals into actual trade positions.
- Reinforcement learning directly maps market situations to trade positions, bypassing an explicit forecasting step.
- The trading objective is sequential because each action affects final wealth and subsequent returns, fitting reinforcement learning’s cumulative-reward framework.
- The study uses DQN, policy gradients, and advantage actor-critic methods with discrete and continuous action spaces.
- Experiments cover 50 liquid futures contracts from 2011 to 2019 across commodities, equity indexes, fixed income, and FX, comparing RL with classical time-series momentum.
2 Literature Review
The literature includes fundamental, technical, algorithmic, and reinforcement-learning trading approaches. The paper focuses on directly converting momentum and technical information into positions while addressing volatility and action-space challenges.
- Algorithmic trading uses mathematical modelling and automated execution, including trend-following, mean-reversion, statistical arbitrage, and delta-neutral strategies.
- Time-series momentum strategies use the sign of past-year returns and have demonstrated profitability across many liquid instruments.
- Trend-following strategies can lose heavily in sideways markets because signal predictability deteriorates and excess turnover erodes profitability.
- Critic-only RL methods commonly use discrete fully long or short positions, while large action spaces make position scaling difficult.
- Actor-only methods learn policies directly and can generalize to continuous action spaces, but Monte Carlo training may be slow and sample-intensive.
- Actor-critic methods update policy and value models in real time, yet remain less studied in financial applications.
3 Methodology
The methodology formulates trading as a Markov decision process in which reinforcement-learning agents directly select positions from market states and optimize volatility-scaled profit rewards. It evaluates discrete and continuous action spaces using DQN, PG, and A2C, with state features built from prices, returns, and technical indicators.
- RL Algorithms: The study applies DQN, Policy Gradients, and Advantage Actor-Critic to the trading problem.DQN uses fixed targets, Double DQN, and Dueling DQN, while PG directly optimizes the policy and A2C updates it in real time using an actor and critic.
- Markov Decision Process: Trading is modeled as a Markov decision process where states produce actions, rewards, and subsequent states.The agent interacts with the market at discrete time steps.
- State Space: The state representation combines past prices, multi-horizon returns, MACD, and RSI features.Returns are volatility-normalized, while RSI uses a 30-day lookback.
- Action Space: Discrete actions directly encode short, flat, and long positions, while continuous actions range from -1 to 1.Maintaining the same action avoids transaction costs, whereas reversing from fully long to short doubles them.
- Reward Function: The reward function uses profits and volatility scaling to increase positions during low volatility and decrease them during high volatility.A 60-day exponentially weighted volatility estimate is used, helping normalize rewards across contracts.
- RL Algorithms: A2C uses an advantage function and critic network to update policies in real time, supporting continuous action spaces.The implementation uses a synchronous approach with agents operating in parallel environments.
4 Experiments
The experiments compare RL trading algorithms with classical baselines on diversified futures portfolios, using volatility targeting and transaction-cost-aware evaluation. RL methods generally perform better across asset classes, with DQN strongest overall and positive profits under substantial costs.
- Dataset and baselines: The study tests 50 continuous futures contracts across commodities, equity indices, fixed income, and FX, with out-of-sample testing from 2011 to 2019.Models are retrained every five years using data available up to that point, then fixed for the following five years.
- Dataset and baselines: The RL methods are compared with long-only and classical time-series momentum baselines, including Sign(R) and MACD Signal.DQN and PG use discrete actions {-1, 0, 1}, while A2C uses continuous actions in [-1, 1].
- Evaluation: Portfolio performance is evaluated using annualized return, volatility, downside risk, Sharpe, Sortino, maximum drawdown, Calmar, positive-return percentage, and positive-to-negative return ratio.Portfolio returns are calculated across contracts, with equal weights and transaction costs included.
- Evaluation: Portfolio-level volatility scaling places methods at a common volatility target so expected and cumulative trade returns can be compared across models.The results are organized by asset class and also include a portfolio using all contracts.
- Results: RL algorithms outperform most asset-class baselines, except that long-only performs better for equity indices dominated by upward trends.RL performs better in more volatile commodity and FX markets by taking long or short positions at reasonable times.
- Results: DQN performs best overall, followed by A2C, while A2C’s larger turnover produces smaller average returns per turnover.The comparison is shown through cumulative returns and Sharpe ratios across asset classes and portfolios.
- Transaction costs: 25bp transaction costs still allow DQN and A2C to generate positive profits, corresponding to roughly $3.5 per contract.The authors describe this cost level as realistic for a retail trader.
- Robustness: Individual-contract results reinforce that RL performance is generally better and is not driven by a single unusually strong contract.The authors interpret this as evidence of model consistency across contracts.
5 Conclusion
The paper applies reinforcement learning to learn trading strategies for continuous futures and reports outperformance of baseline models with profits under heavy transaction costs. It identifies broader utility functions, including risk-adjusted objectives, as directions for continuation.
- Conclusion: RL algorithms learn trading strategies for continuous futures using discrete and continuous action spaces.The analysis focuses on DQN, Policy Gradients, and Advantage Actor-Critic, with time-series momentum and technical-indicator features forming state representations.
- Conclusion: RL algorithms outperform baseline models and deliver profits even under heavy transaction costs.The methods are tested on 50 liquid futures contracts from 2011 to 2019.
- Conclusion: Future work will investigate different utility functions, including risk-adjusted objectives such as the Sharpe Ratio.The paper also proposes distributional reinforcement learning and portfolio-optimization extensions.
Appendix A
The dataset contains 50 futures contracts spanning commodities, equity indices, fixed income, and forex markets.
- Appendix A: The dataset contains 50 futures contracts.The passage introduces a detailed description of each contract.
- Appendix A: Commodity contracts comprise 25 of the 50 futures contracts.
- Appendix A: The remaining contracts include 11 equity index, 5 fixed income, and 9 forex contracts.
.3 Fixed Incomes
The fixed-income contract list includes European and U.S. government-bond futures, including Bund, Bobl, T-Note, and T-Bond instruments.
- Fixed Incomes: The list includes DT EURO BOND (BUND) and EURO BOBL contracts.
- Fixed Incomes: The list includes FB T-NOTE and a 5-year Composite TY T-NOTE.
- Fixed Incomes: The list includes a 10-year Composite UB EURO BOBL and US T-BONDS, Composite.
.4 Forex
The forex-related list includes Australian, British pound, Canadian, U.S. dollar index, euro, Japanese yen, Mexican peso, Nikkei index, and Swiss franc instruments.
- Forex: The list includes Australian, British pound, and Canadian instruments.
- Forex: The list includes the US DOLLAR INDEX, EURO, and JAPANESE YEN.
- Forex: The list includes the MEXICAN PESO, NIKKEI INDEX, and SWISS FRANC.
Appendix B
Appendix B reports experiment metrics for portfolios without an additional volatility-scaling layer.
- Table 3 presents experiment results for the raw signal without additional portfolio-level volatility scaling.