Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining

1 Introduction

• The direct approach uses DRL to map market observations to trading actions, while the indirect approach uses LSTM price forecasts followed by a trading strategy.Source: §1 · p. 1
• The study constructs a dataset of 20 large-capitalization S&P 500 stocks from 2000–2020, enriched with trend-following technical indicators and log min–max scaling.Source: §1 · p. 1
• Among the compared forecasting losses, MAPE produces the lowest validation error.Source: §1 · p. 1
• Five DRL algorithms are benchmarked against an LSTM strategy on a 2019–2020 out-of-sample backtest, with agents trained once and with forward retraining.The algorithms are A2C, PPO, DDPG, TD3, and SAC.Source: §1 · p. 1

2 Related Work

• The paper’s LSTM forecasting approach draws on prior stock-price prediction work and a time-series tutorial.Source: §2 · p. 1
• The overview contrasts an LSTM price forecast followed by a rule-based strategy with a DRL agent that learns trading actions end to end.Both approaches train on 2000–2018 and backtest on 2019–2020.Source: Figure 1 · p. 2
• Prior equity-trading RL work includes ensemble strategies, financial-market surveys, and interpretable attention-based portfolio agents [7].Source: §2 · p. 2
• The related-work discussion identifies limited real-world data, reward-design trade-offs, and sparse historical patterns as challenges for financial RL.Source: §2 · p. 2

3 Dataset and Features

• The dataset contains 105,680 daily records for 20 large-cap S&P 500 stocks and is split chronologically into 2000–2018 training and 2019–2020 test periods.The universe is restricted to companies with complete histories over 2000–2020.Source: §3 · p. 2

3.1 Technical Indicators

• Seven trend-following indicator families are added, including MACD, RSI, Bollinger bands, CCI, DX, SMA, and EMA, with some using multiple look-back windows.Table 2 shows a sample of the processed dataset.Source: §3.1 · p. 2, Table 2 · p. 3

3.2 Logarithmic Scaling

• Log min–max scaling is applied to price, SMA, and EMA features because exponential market growth makes price-level features nonlinear; the transformation makes patterns easier for a shallow network to capture.Source: §3.2 · p. 3

4 Methods

• The study combines LSTM time-series forecasting with a simple single-stock trading strategy, then uses DRL models to optimize trading directly.Source: §4 · p. 3

4.1 LSTM Time-Series Forecasting

• The forecasting-and-trading pipeline was designed end to end, with price forecasts feeding a simple portfolio strategy.The strategy holds cash below the current-price threshold and buys and holds shares at or above it.Source: §4.1 · p. 3, §4.1.3 · p. 4

4.1.1 Loss Function

• MAPE reduced prediction skew as prices rose, while price-level MSE can weight equal 10% moves 40,000-fold differently across price levels.The comparison motivates evaluating several loss functions for stock prediction.Source: §4.1.1 · p. 3
• Each sample’s MAPE gradient has constant magnitude inversely proportional to price, while squared percentage error scales its gradient with error size.The latter is presented as a variant when smoother gradients near the optimum are desired.Source: §4.1.1 · p. 4

4.1.2 Architecture and Hyperparameters

• A two-layer LSTM followed by one dense layer yielded the most stable predictive model among the explored architectures.The tested alternatives included bidirectional and stacked LSTMs, and LSTMs with multiple dense layers.Source: §4.1.2 · p. 4
• At this dataset scale, too many parameters led to overfitting, while dropout or other regularization increased computational cost without improving metrics.Table 3 compares training and validation metrics for LSTM variants on scaled prices.Source: §4.1.2 · p. 4, Table 3 · p. 4
• Training on all stocks already yielded good predictions, with optional target-stock fine-tuning after pooled training.The paper compares pooled training with single-ticker training and describes a third regime that fine-tunes on the target stock.Source: §4.1.2 · p. 4

4.1.3 Trading Strategy

• Forecasts at or above the current price trigger buying and holding shares; lower forecasts leave the strategy in cash.The position rule supports both an unleveraged variant and leveraged variants.Source: §4.1.3 · p. 4

4.2.1 Model Definition

• Actions represent buying, holding, or selling shares; the general per-stock action space ranges from −k to k, where k is the maximum shares per trade.The basic single-stock actions are −1, 0, and 1.Source: §4.2.1 · pp. 4–5
• The reward is the change in portfolio value between the current and resulting states after an action.Source: §4.2.1 · p. 5
• The agent’s state includes available cash, closing prices, shares held, and 14 technical indicators with specified look-back windows.The indicators include MACD, Bollinger bands, RSI, CCI, DX, and moving averages.Source: §4.2.1 · p. 5

4.2.2 Learning Algorithms

• The benchmark covers A2C, PPO, DDPG, TD3, and SAC, including policy-update clipping, actor-critic learning, continuous-action control, and entropy-regularized exploration.PPO clips its policy objective; A2C combines actor and critic; DDPG is designed for continuous actions; TD3 modifies DDPG to address instability; SAC trades off expected return and entropy.Source: §4.2.2 · p. 5

4.2.3 Forward Retraining

• Forward retraining retrains each agent on all historical data before a successive test slice, which the model never sees while trading.This is a walk-forward scheme with an expanding training window.Source: §4.2.3 · p. 5

5.1 Evaluation Metrics

• Strategies are evaluated on the 2019–2020 test period using return, Sharpe ratio, maximum drawdown, alpha, and beta, with the S&P 500 as benchmark.Sharpe ratio measures return per unit of volatility; drawdown is the largest peak-to-trough decline; alpha and beta measure excess return and market exposure, respectively.Source: §5.1 · p. 6

5.2 Discussion

• The unleveraged LSTM has the lowest annual return at 16.99%, but the smallest maximum drawdown at −9.57% and lowest beta at 0.31.Its Sharpe ratio is 1.19, in the middle of the range.Source: §5.2 · p. 6
• 55.51% annual return, 1.38 Sharpe ratio, and 0.22 alpha make DDPG the leader on those measures, but its 1.24 beta and −33.52% drawdown are also highest or among the deepest.Source: §5.2 · p. 6
• Sharpe ratios of 1.37 and 1.33 and maximum drawdowns of −25.24% and −24.19% give TD3 and SAC a better return-risk balance, respectively.Source: §5.2 · p. 6
• Forward retraining raises annual returns for A2C, PPO, and SAC, leaves TD3 essentially unchanged, and lowers DDPG from 55.51% to 29.82%.The authors say DDPG can be difficult to tune for each retraining window, which may explain the gap.Source: §5.2 · p. 6
• TD3 substantially stabilizes training in these experiments, making results less dependent on hyperparameters and tuning, and performs nearly identically with or without forward retraining.Source: §5.2 · p. 6

• 55.51% annual return: DDPG led returns, while LSTM had the smallest drawdown and market exposure and TD3 paired risk-adjusted performance with stability across training strategies.Source: §6 · p. 7
• The universe of large companies with complete 2000–2020 histories introduces survivorship bias.Source: §6 · p. 7
• The two-year 2019–2020 test period includes a strong bull market and the early-2020 COVID-19 crash, limiting generalization to other market regimes.Source: §6 · p. 7
• DRL sensitivity to random seeds and hyperparameters is not quantified because variability across runs is unreported.Source: §6 · p. 7