An Application of Deep Reinforcement Learning to Algorithmic Trading

Thibaut Théate; Damien Ernst

Source-linked AI summary

An Application of Deep Reinforcement Learning to Algorithmic Trading

Thibaut Théate, Damien Ernst

arXiv:2004.06627v3Apr 7, 2020q-fin.TRcs.AIcs.LG

TL;DR

The paper addresses how to determine optimal trading positions in a difficult, partially observable stock-market decision problem. It proposes TDQN, a DRL strategy trained with artificial trajectories and evaluated using a rigorous assessment methodology. The reported results are promising, with TDQN surpassing benchmark strategies on average and adapting to trading costs.

Problem
Algorithmic trading requires sequential position decisions in a stochastic, partially observable, and potentially adversarial environment.
Method
The paper proposes TDQN, a DRL strategy inspired by DQN, trained from artificial trajectories and assessed with a rigorous performance methodology.
Results
TDQN achieves promising results, surpassing benchmark trading strategies on average and reducing trading frequency as trading costs increase.
Takeaways & Limitations
TDQN shows versatility and robustness to diverse trading costs within the stock-market setting studied.
Takeaways & Limitations
TDQN is constrained by poor observability, sensitivity to market-regime shifts, overfitting, and an objective that only relaxes Sharpe-ratio maximisation.

Abstract

from arXiv · show

This scientific research paper presents an innovative approach based on deep reinforcement learning (DRL) to solve the algorithmic trading problem of determining the optimal trading position at any point in time during a trading activity in stock markets. It proposes a novel DRL trading strategy so as to maximise the resulting Sharpe ratio performance indicator on a broad range of stock markets. Denominated the Trading Deep Q-Network algorithm (TDQN), this new trading strategy is inspired from the popular DQN algorithm and significantly adapted to the specific algorithmic trading problem at hand. The training of the resulting reinforcement learning (RL) agent is entirely based on the generation of artificial trajectories from a limited set of stock market historical data. In order to objectively assess the performance of trading strategies, the research paper also proposes a novel, more rigorous performance assessment methodology. Following this new performance assessment approach, promising results are reported for the TDQN strategy.

1. Introduction

The paper positions algorithmic trading as a difficult sequential financial decision problem and proposes a deep reinforcement learning strategy to select trading positions. It also introduces a more rigorous approach for evaluating such strategies.

Algorithmic trading uses computers and mathematical rules to make trading decisions in financial markets.
Its decisions are sequential, stochastic, partially observable, and potentially adversarial.
The paper asks how an AI-based trading policy can compete with widely adopted algorithmic trading strategies.
The proposed DRL solution determines whether to take a long or short trading position at each point in stock-market activity.
The paper formalises the trading problem, links it to reinforcement learning, designs TDQN, and evaluates it through a proposed assessment methodology.

2. Literature review

The literature review highlights limited public evidence and inconsistent evaluation practices in algorithmic trading research. It situates the paper among classical, forecasting-oriented machine learning, and emerging DRL approaches while emphasizing unbiased assessment.

Many algorithmic-trading studies are inaccessible because private FinTech firms are unlikely to publish research results.
Fair strategy comparisons are difficult because no common evaluation framework exists and trading costs are inconsistently defined or omitted.
Classical approaches include trend-following and mean-reversion strategies developed largely outside AI.
Most machine-learning work in algorithmic trading focuses on forecasting, while some studies apply deep Q-learning or DRL to specific markets.
The paper aims to provide an unbiased evaluation because financial research may prioritize positive findings over rigorous criticism.

3. Algorithmic trading problem formalisation

The paper formalises algorithmic trading as a sequential decision-making problem and explicitly connects that formulation to reinforcement learning.

The formalisation presents the sequential decision-making trading problem in detail and establishes its link with the RL framework.

3.1. Algorithmic trading

The paper defines algorithmic trading as machine-based decisions governed by mathematical rules and focuses its formal problem on managing a single-stock portfolio. The agent must decide what, when, how, where, and how much to trade.

Algorithmic trading automatically makes trading decisions using mathematical rules computed by a machine.
The study focuses on stock markets, while applicability to other markets is planned as a future extension.
The portfolio consists of one stock and the agent’s cash, with portfolio value combining cash and share value.
The agent interacts with the market through an order book containing bids and asks, each characterized by price, quantity, and side.
The central task is to decide what, when, how, at which price, and which quantity to trade.

3.2. Timeline discretisation

The continuous trading timeline is discretized into fixed-duration time steps so the trading problem can be modeled sequentially. The interval determines the target trading frequency and limits the maximum achievable frequency.

The continuous trading timeline is discretized into numerous fixed-duration trading time steps t separated by Δt.The notation t+1 and t−1 represents discrete transitions between successive steps.
The interval Δt is linked to the trading frequency targeted by the agent, including intraday, daily, or monthly trading.
1/Δt bounds the maximum achievable trading frequency because technical constraints prevent choosing arbitrarily small time intervals.

3.3. Trading strategy

The trading strategy is a programmed policy that sequentially converts available market information into trading actions. After execution, the process advances to the next time step and repeats.

A trading strategy is a deterministic or stochastic policy π(at|it) that outputs action at from information it available at time t.
The strategy execution is sequential, looping from the next time step back to information updating.
Each trading step updates market information, applies the policy, executes the designated action, and advances to t+1.
The trading decision-making problem is formulated as a reinforcement learning problem because it shares similarities with sequential problems addressed by the RL community.

3.4. Reinforcement learning problem formalisation

The paper formalizes algorithmic trading as an RL problem in which an agent observes limited market information, selects share-based actions, receives rewards, and updates its portfolio sequentially.

3.4. Reinforcement learning problem formalisation: At each step, the RL agent observes an internal environment state through observation ot, selects at using policy π(at|ht), and receives reward rt.The history ht contains prior observations, actions, and rewards.
3.4. Reinforcement learning problem formalisation: RL policies maximize an expected discounted sum of rewards over an infinite horizon, with discount factor γ controlling the importance of future rewards.γ=0 yields myopic behavior, whereas increasing γ makes the agent more long-term oriented.
3.4.1. RL observations: The trading environment is only partially observable, because the agent has limited access to information that may affect market prices.
3.4.1. RL observations: The broader observation design can include time, technical indicators, macroeconomic data, news, and other useful information.Examples include MACD, RSI, interest rates, exchange rates, and sentiment extracted from news.
3.4.1. RL observations: The reduced observation space uses the current position, previous τ+1 daily OHLCV records, and agent state information.The state includes the current position, shares owned, and available cash.
3.4.2. RL actions: The action is the integer number of shares Qt bought or sold at time t, answering whether, how, and how much to trade.
3.4.2. RL actions: Qt>0 buys shares, Qt<0 sells shares, and Qt=0 holds the current position.
3.4.2. RL actions: Cash must remain positive, traded volume is bounded, and short positions are represented by negative share holdings; order execution is delegated to an external system.Portfolio cash and share values are updated after actions, while execution strategies remain outside the paper’s scope.

3.5. Objective

The paper evaluates trading strategies primarily through the Sharpe ratio, which balances profitability and risk across diverse market conditions. Its RL optimization instead uses discounted daily returns, creating a stated gap between training and evaluation objectives.

The Sharpe ratio is the core objective because it considers both generated profit and trading risk, unlike profit maximization alone.
Sharpe ratio components include strategy return Rs, risk-free return Rf, and the standard deviation σr of excess return Rs−Rf.
Daily returns are computed first, their mean-to-standard-deviation ratio is evaluated, and annualization multiplies it by √252.
The strategy is intended to maximize average Sharpe ratio across stock markets with differing trends and volatility.The desired scope includes both bull and bear markets.
The RL algorithm maximizes expected discounted daily returns rather than the Sharpe ratio itself, which the paper describes as a close relaxation.The authors identify narrowing this objective gap as future research.

4. Deep reinforcement learning algorithm design

The paper designs TDQN, a DQN-inspired deep reinforcement learning strategy adapted to algorithmic trading. Its training uses artificial trajectories generated from limited historical stock-market data, alongside modifications targeting trading-specific inputs, stability, exploration, and generalisation.

TDQN adapts the DQN algorithm to determine trading positions in a stock-market decision-making problem.The strategy is designed for the algorithmic trading setting rather than directly applying classical DQN.
TDQN training generates artificial trajectories from a limited set of historical daily OHLCV data.The environment model is unavailable, so training relies on simulated trajectories derived from market history.
The trading agent continuously explores both long and short actions by executing the chosen action and its opposite in copied environments.This exploration trick incurs a small extra computational cost but does not completely resolve the exploration–exploitation trade-off.
TDQN replaces DQN’s image-oriented convolutional architecture with a feedforward DNN suited to time-series observations.The network approximates the action-value function Q(s, a) and uses Leaky ReLU activations.
Double DQN, ADAM, Huber loss, gradient clipping, Xavier initialisation, batch normalisation, and regularisation modify TDQN’s learning process.The cited modifications address overestimation, convergence, training stability, exploding gradients, and overfitting.
Preprocessing and data augmentation filter, normalise, shift, and perturb trading observations to address noise, limited data, and generalisation.The paper also acknowledges that filtering can modify or destroy potentially useful trading patterns.

5. Performance assessment

The paper proposes a broader performance-assessment methodology for TDQN and benchmark strategies, using diversified instruments, periods, indicators, and visualisations. The testbench separates training and testing, while acknowledging limits in temporal and market diversification.

The proposed methodology aims to objectively assess algorithmic trading strategies, including TDQN, through a more reliable performance-assessment procedure.The paper identifies a lack of proper assessment methodology in the algorithmic trading literature.
Performance is evaluated across multiple instruments with diverse patterns to reduce bias from selecting a single apparently profitable market.The methodology is motivated by concerns that single-instrument evaluations may not generalise.
The eight-year horizon uses 2012–2017 for training and 2018–2019 for testing.A validation subset of the training data tunes TDQN hyperparameters, and policy parameters remain fixed throughout testing.
The testbench’s diversification remains limited because adding more stocks and alternative periods would require substantial additional computation.The authors specifically identify broader stock and time-period coverage as improvements.
The benchmark includes passive buy-and-hold and sell-and-hold strategies plus active trend-following and mean-reversion strategies.Trend following follows market movements, whereas mean reversion takes opposing positions against trends.
The assessment combines profitability and risk, prioritising the Sharpe ratio alongside additional quantitative indicators and trading-behaviour visualisations.Plots combine stock price, portfolio value, and trading actions to analyse policy behaviour and strengths or weaknesses.

6. Results and discussion

The TDQN strategy shows promising but variable performance across stocks, outperforming active benchmarks on average while remaining sensitive to volatility, trading costs, observability, and overfitting.

Good results - Apple stock: The Apple case achieved good earnings and risk mitigation, outperforming all considered active and passive benchmark strategies in a typical run.The agent detected major trends but lagged slightly, reflecting a more reactive policy under limited observability.
Good results - Apple stock: Averaged over 50 training runs, Apple’s expected performance was comparable to the typical run, with overfitting apparently handled for this market.Training and test returns differ because their market distributions are not identical.
Mitigated results - Tesla stock: On Tesla, the TDQN achieved a positive Sharpe ratio but generated almost no profit, with an unacceptable risk level and particularly long maximum drawdown duration.Tesla’s pronounced volatility also induced more frequent position changes despite trading costs, increasing riskiness.
Mitigated results - Tesla stock: Tesla’s expected performance exceeded the typical run, exposing substantial policy-selection variance and stronger training than test performance consistent with overfitting.The limited observation space may explain why the algorithm failed to efficiently apprehend this stock.
Global results - Testbench: Across the testbench, TDQN outperformed active strategies on average but only barely surpassed buy and hold in predominantly bullish markets.For multiple stocks, it approached passive-strategy performance by tending toward passive trading when active-trading uncertainty increased.
Trading costs and discount factor: Higher trading costs caused TDQN to reduce trading frequency, and sufficiently high costs led it to stop actively trading and adopt a passive strategy.The experiments also linked the discount factor to trading frequency because trading costs impede position changes.

7. Conclusion

The paper presents TDQN as a deep reinforcement learning solution for stock-market trading, with promising average performance and adaptability to trading costs. It also identifies generalisation and reproducibility as areas for improvement and proposes several research directions.

TDQN is a DRL trading strategy for determining the optimal trading position at each point in stock-market activity.
Following rigorous performance assessment, TDQN achieves promising results and surpasses benchmark trading strategies on average.
The data-driven approach avoids defining explicit rules tailored to each financial market.
TDQN performance could be improved in terms of generalisation and reproducibility, with LSTM, Rainbow, and PPO comparisons suggested as research directions.
Future work also includes extending the observation space, relaxing action constraints, and engineering rewards to narrow the gap with Sharpe-ratio maximisation.

Appendix A. Derivation of action space A

The appendix derives upper and lower bounds for the RL action space from financial feasibility constraints. The upper bound preserves nonnegative cash, while the lower bound ensures sufficient cash to return to a neutral position under trading costs and adverse conditions.

Theorem 1 derives an upper bound for the RL action space A.
The upper bound follows from requiring the agent’s cash value to remain nonnegative throughout the trading horizon.
The upper-bound derivation considers separate cases according to the sign of the traded share quantity Q_t.
Theorem 2 derives a lower bound for A to ensure cash remains sufficient for returning to the neutral position at the next time step.
The lower bound is piecewise: it uses ∆t p_t ϵ(1+C) when ∆t ≥ 0 and ∆t p_t (2C+ϵ(1+C)) when ∆t < 0.
The lower-bound proof enforces the next-step cash constraint after trading, including transaction costs, and separates cases for Q_t.
The quantity ∆t measures the difference between the worst-case cost of returning to neutrality and the agent’s current cash value.
When ∆t is negative, debt repayment poses no problem in the stated conditions; when ∆t is nonnegative, it may pose a problem and determines the tighter lower bound.