Source-linked AI summary
Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy
Hongyang Yang, Xiao-Yang Liu, Shan Zhong, Anwar Walid
TL;DR
Profitable automated trading is difficult in complex, dynamic markets, motivating methods that can learn decisions while accounting for risk and trading constraints. This paper combines PPO, A2C, and DDPG agents in an ensemble that selects among them using the Sharpe ratio. In backtests, the ensemble achieved a Sharpe ratio of 1.30, exceeding the individual agents and the two benchmark strategies.
Problem
Designing profitable automated stock-trading strategies is difficult because analysts must account for many relevant factors in complex and dynamic markets.
Method
The paper trains PPO, A2C, and DDPG actor-critic agents and combines them by automatically selecting the best-performing agent according to the Sharpe ratio.
Results
1.30 Sharpe ratio for the ensemble, compared with 1.12 for A2C, 1.10 for PPO, 0.87 for DDPG, 0.47 for DJIA, and 0.45 for min-variance allocation.
Takeaways & Limitations
The ensemble strategy outperforms the three individual algorithms and two baselines in Sharpe ratio by balancing risk and return under transaction costs.
Abstract
from arXiv · showhide
Stock trading strategies play a critical role in investment. However, it is challenging to design a profitable strategy in a complex and dynamic stock market. In this paper, we propose an ensemble strategy that employs deep reinforcement schemes to learn a stock trading strategy by maximizing investment return. We train a deep reinforcement learning agent and obtain an ensemble trading strategy using three actor-critic based algorithms: Proximal Policy Optimization (PPO), Advantage Actor Critic (A2C), and Deep Deterministic Policy Gradient (DDPG). The ensemble strategy inherits and integrates the best features of the three algorithms, thereby robustly adjusting to different market situations. In order to avoid the large memory consumption in training networks with continuous action space, we employ a load-on-demand technique for processing very large data. We test our algorithms on the 30 Dow Jones stocks that have adequate liquidity. The performance of the trading agent with different reinforcement learning algorithms is evaluated and compared with both the Dow Jones Industrial Average index and the traditional min-variance portfolio allocation strategy. The proposed deep ensemble strategy is shown to outperform the three individual algorithms and two baselines in terms of the risk-adjusted return measured by the Sharpe ratio. This work is fully open-sourced at \href{https://github.com/AI4Finance-Foundation/Deep-Reinforcement-Learning-for-Automated-Stock-Trading-Ensemble-Strategy-ICAIF-2020}{GitHub}.
I. INTRODUCTION
Automated stock trading is difficult in complex, dynamic markets because conventional allocation and dynamic-programming approaches can be costly or difficult to scale. The paper proposes an ensemble of three actor-critic reinforcement-learning agents to select robust trading strategies across market situations.
- Complex and dynamic stock markets make it difficult for analysts to consider all factors needed to maximize returns while managing risk.
- Traditional portfolio allocation requires estimating returns and covariance, then repeatedly revising decisions while incorporating factors such as transaction costs.
- Dynamic programming models stock trading as a Markov Decision Process, but their scalability is limited by the market’s large state spaces.
- Predictive machine-learning approaches generate alpha signals for stock selection but focus on picking high-performance stocks rather than directly learning trading strategies.
- The proposed ensemble combines PPO, A2C, and DDPG actor-critic algorithms, selecting agents with the Sharpe ratio to adapt to different market situations.
- Actor-critic methods update policy and value networks together, with the critic guiding policy improvements through policy gradients.
III. PROBLEM DESCRIPTION
The paper models stock trading as a Markov Decision Process in which states contain prices, holdings, and cash; actions sell, buy, or hold shares; and rewards measure portfolio-value changes.
- The trading state contains stock prices, held shares, and the remaining cash balance.
- Each action is a vector across stocks, with selling, buying, or holding changing share holdings accordingly.
- The reward is the direct payoff from taking an action in a state and arriving at a new state.
- The policy maps each state to a probability distribution over trading actions, while Qπ(s, a) represents expected reward under that policy.
- At each state, an action is taken for each stock, producing transitions among possible portfolio values as prices update.
B. Incorporating Stock Trading Constraints
The trading environment incorporates practical constraints including liquidity, nonnegative cash, transaction costs, and crash risk. A turbulence threshold triggers defensive selling behavior.
- The environment’s assumptions and constraints address transaction costs, market liquidity, and risk aversion in practical trading.
- Market liquidity is modeled by assuming orders execute rapidly at the closing price without affecting the market.
- Actions are constrained so the cash balance remains nonnegative, with stocks partitioned into selling, buying, and holding sets.
- Transaction costs are modeled as 0.1% of each trade’s value for both buying and selling.
- The turbulence index measures extreme asset-price movements using current returns, historical mean returns, and historical-return covariance.
- When turbulence exceeds a threshold, buying halts and the agent sells all shares until the index falls below the threshold.
C. Return Maximization as Trading Goal
The trading objective is to maximize cumulative portfolio-value growth through reinforcement learning, while accounting for holding, selling, buying, and crash-related risk responses.
- The reward function is defined as the change in portfolio value after an action, and the goal is to maximize its positive cumulative change.
- Portfolio returns are decomposed into contributions from holding, selling, and buying shares across successive time steps.
- The strategy should buy and hold stocks expected to rise while selling stocks expected to fall to maximize portfolio-value change.
- The turbulence index is incorporated into the reward to address crash risk, with extreme conditions motivating liquidation of held stocks.
- The Q-value combines immediate reward with discounted future reward, using a discount factor 0 < γ < 1 for convergence.
IV. STOCK MARKET ENVIRONMENT
The trading environment simulates real-world trading so an agent can interact with market information and learn. It models a portfolio containing 30 stocks with continuous actions.
- The environment provides historical prices, current holdings, and technical indicators for agent interaction and learning.It is implemented with OpenAI Gym.
- The portfolio contains 30 stocks, modeled with a continuous action space for multi-stock trading.
1) State Space:
The multi-stock state represents portfolio, market-price, holdings, and technical-indicator information in a 181-dimensional vector. The action space covers trading across 30 stocks and is normalized for relevant algorithms, while load-on-demand reduces memory use.
- State Space: The state space is a 181-dimensional vector containing seven information components: balance, prices, holdings, MACD, RSI, CCI, and ADX.
- State Space: The state includes current balance, adjusted close prices, and shares owned for each stock.
- State Space: MACD, RSI, CCI, and ADX encode momentum, recent price changes, price-versus-average relationships, and trend strength.
- Action Space: The action space has (2k + 1)^30 possible multi-stock actions and is normalized to [−1, 1] for Gaussian policies used by A2C and PPO.
- Load-on-Demand Technique: Load-on-demand generates results only when requested instead of storing all results in memory, reducing memory usage during training.
V. TRADING AGENT BASED ON DEEP REINFORCEMENT LEARNING
The trading agent combines three actor-critic algorithms—A2C, DDPG, and PPO—into an ensemble strategy. These components address policy-gradient variance, continuous actions, and policy-update stability.
- Ensemble Strategy: The ensemble combines A2C, DDPG, and PPO to build a robust trading strategy.
- Advantage Actor Critic (A2C): A2C uses an advantage function to reduce policy-gradient variance and improve model robustness.Its critic estimates the advantage function rather than only the value function.
- Advantage Actor Critic (A2C): A2C averages gradients from independently interacting copies through a coordinator before updating a global network.The global network increases training-data diversity.
- Deep Deterministic Policy Gradient (DDPG): DDPG deterministically maps observations to actions and is designed to handle continuous action spaces in stock trading.It combines Q-learning and policy-gradient frameworks with neural-network function approximators.
- Deep Deterministic Policy Gradient (DDPG): DDPG stores environment transitions in a replay buffer and updates Q-values from sampled batches.
C. Proximal Policy Optimization (PPO)
PPO is included in the ensemble to stabilize policy-gradient training by limiting how far each new policy can move from the previous one. Its clipped objective discourages large policy updates.
- PPO Motivation: PPO controls policy-gradient updates by introducing a clipping term into a simplified trust-region objective.
- PPO Objective: The clipped PPO objective restricts the policy ratio to [1 − ϵ, 1 + ϵ] and takes the minimum of clipped and normal objectives.
- PPO Objective: Restricting policy updates improves policy-network training stability by discouraging large changes at each step.
- PPO Selection: The paper selects PPO for stock trading because it is stable, fast, and simpler to implement and tune.
D. Ensemble Strategy
The ensemble retrains PPO, A2C, and DDPG, selects the highest-Sharpe agent using rolling validation, and deploys it for the next quarter. Backtesting compares the strategy with individual agents and benchmark portfolios.
- The ensemble automatically selects among PPO, A2C, and DDPG using the validation-period Sharpe ratio.The selected agent trades during the following quarter.
- Agents are retrained concurrently with a growing n-month window every three months.
- A 3-month rolling validation window identifies the agent with the highest Sharpe ratio, with turbulence used to adjust risk aversion.
- The strategy is designed to accommodate agents’ differing sensitivity to bullish, bearish, and volatile market trends.
- Backtesting evaluates the three agents and ensemble against the Dow Jones Industrial Average and min-variance portfolio allocation strategy.
B. Performance Comparisons
Performance comparisons use five portfolio metrics, with Sharpe ratio combining return and risk. The agents exhibit complementary behavior across market conditions, while the ensemble and individual agents outperform the two benchmarks in the reported comparisons.
- Five metrics evaluate performance: cumulative return, annualized return, annualized volatility, Sharpe ratio, and maximum drawdown.
- A2C records the lowest annual volatility 10.4% and max drawdown −10.2% among the three agents.
- PPO records the highest annual return 15.0% and cumulative return 83.0% among the three agents.
- PPO is preferred for bullish markets, A2C for bearish markets, and DDPG as a complementary bullish-market strategy.
- The ensemble achieves a higher Sharpe ratio than the three individual agents, Dow Jones Industrial Average, and min-variance portfolio allocation strategy.
3) Performance under Market Crash:
During the first-quarter 2020 crash, the ensemble and individual agents performed well, while the ensemble achieved the strongest risk-adjusted performance across the evaluation period. The strategy uses market turbulence to exit positions during extreme conditions and later resume trading.
- Performance during Market Crash: All three agents and the ensemble strategy performed well during the 2020 stock market crash.
- Crash-Response Mechanism: When the turbulence index reaches a threshold, the agents sell all currently held shares and wait for normal market conditions before resuming trading.The threshold can be lowered to increase risk aversion.
- Benchmark Comparison: The ensemble strategy achieved a Sharpe ratio of 1.30, exceeding DJIA at 0.47 and min-variance allocation at 0.45.Figure 5 compares cumulative returns from 2016/01/04 to 2020/05/08 using an initial portfolio value of $1,000,000.
- Benchmark Comparison: The ensemble strategy also exceeded A2C at 1.12, PPO at 1.10, and DDPG at 0.87 in Sharpe ratio.
- Conclusion: The authors conclude that the ensemble outperforms the individual algorithms and both baselines in Sharpe ratio by balancing risk and return under transaction costs.
- Limitations and Future Work: Future work includes handling larger datasets, such as S&P 500 constituent stocks, and adding richer transaction-cost, liquidity, fundamental, news, and ESG features.The authors also note that directly using Sharpe ratio as the reward would require substantially more historical observations and a rapidly expanding state space.