Source-linked AI summary
Deep Reinforcement Learning for Optimal Portfolio Allocation: A Comparative Study with Mean-Variance Optimization
Srijan Sood, Kassiani Papasotiriou, Marius Vaiciulis, Tucker Balch
TL;DR
DRL portfolio methods are often evaluated against basic or machine-learning benchmarks rather than practical traditional optimizers such as MVO. This paper compares a risk-adjusted-return DRL framework with MVO through systematic multi-year backtests and reports stronger performance across several metrics.
Problem
Existing DRL portfolio benchmarks often omit practical Mean-Variance Optimization and may use mismatched objectives or insufficiently described MVO implementations.
Method
The paper trains policy-gradient DRL agents with risk-adjusted rewards in a market-replay environment and compares them with MVO through systematic backtests.
Results
DRL outperforms MVO across annual returns, Sharpe ratio, maximum drawdown, and turnover, with a full-period Sharpe ratio of 1.17 versus 0.68 for MVO.
Takeaways & Limitations
The results indicate that the DRL strategy produces more consistent returns and more stable portfolios with decreased turnover across ten US-equities backtests.
Takeaways & Limitations
The study leaves transaction costs and slippage for future work and proposes adding drawdown minimization to the reward.
Abstract
from arXiv · showhide
Portfolio Management is the process of overseeing a group of investments, referred to as a portfolio, with the objective of achieving predetermined investment goals. Portfolio optimization is a key component that involves allocating the portfolio assets so as to maximize returns while minimizing risk taken. It is typically carried out by financial professionals who use a combination of quantitative techniques and investment expertise to make decisions about the portfolio allocation. Recent applications of Deep Reinforcement Learning (DRL) have shown promising results when used to optimize portfolio allocation by training model-free agents on historical market data. Many of these methods compare their results against basic benchmarks or other state-of-the-art DRL agents but often fail to compare their performance against traditional methods used by financial professionals in practical settings. One of the most commonly used methods for this task is Mean-Variance Portfolio Optimization (MVO), which uses historical time series information to estimate expected asset returns and covariances, which are then used to optimize for an investment objective. Our work is a thorough comparison between model-free DRL and MVO for optimal portfolio allocation. We detail the specifics of how to make DRL for portfolio optimization work in practice, also noting the adjustments needed for MVO. Backtest results demonstrate strong performance of the DRL agent across many metrics, including Sharpe ratio, maximum drawdowns, and absolute returns.
1 Introduction
Portfolio optimization allocates assets to balance returns and risk, while recent DRL methods motivate a comparison with traditional financial practice. This paper develops and evaluates a DRL framework against MVO using systematic backtests.
- Motivation: Portfolio management allocates funds across assets to generate returns while minimizing risk and operational costs.Optimization may span asset classes or asset selection within one class.
- Related developments: Recent machine-learning research supports feature selection, forecasting, and estimation of asset means and covariances for portfolio optimization.
- Related developments: Deep reinforcement learning has attracted finance research because sequential decision-making and applied successes support trading and portfolio-allocation applications.
- Paper approach: The paper compares a risk-adjusted-return DRL framework with Mean-Variance Optimization using policy-gradient agents and systematic US-equities backtests.Agents use market-data replay and observation states derived from observed asset prices.
- Headline result: The DRL strategy improves across risk-adjusted returns, maximum drawdown, and portfolio turnover metrics relative to MVO.
2 Related Work
DRL portfolio research commonly compares against DRL, machine-learning, buy-and-hold, or market benchmarks, leaving practical comparison with MVO underdeveloped. The paper addresses methodological inconsistencies to enable a more robust DRL–MVO comparison.
- Existing DRL research: Portfolio-optimization studies explore policy architectures and state information from technical indicators or news data.
- Benchmark gap: Common DRL benchmarks include other DRL or ML methods, buy-and-hold, and market or index performance, which may provide only simplistic relative comparisons.
- Methodological gap: Prior DRL–MVO comparisons often lack detailed methodology, mismatch training rewards with MVO objectives, use exogenous information, and omit MVO implementation details.The paper therefore targets a robust comparison in which both approaches optimize the same goal.
3 Background
Portfolio optimization seeks consistent risk-aware allocation, with MVO providing a return–risk framework and DRL learning policies from sequential interactions. The background establishes the objectives, inputs, and optimization structures underlying both approaches.
- Portfolio objectives: Portfolio optimization continuously diversifies and reallocates funds to maximize realized rewards while restraining risk and seeking consistent performance over time.
- Portfolio objectives: Modern portfolio theory balances risk tolerance and return expectations through diversification, and MVO is presented as a principal technique within that framework.
- Mean-Variance Optimization: MVO allocates portfolio weights using asset returns and covariances, typically estimated from historical data, to maximize return, minimize risk, or maximize return per unit risk.
- Mean-Variance Optimization: MVO can be formulated as a single- or multi-objective optimization problem and commonly solved as a convex optimization problem producing an efficient frontier.
- Mean-Variance Optimization: The Sharpe ratio measures portfolio return per unit risk, and the study selects it as an objective to optimize risk-adjusted returns without fixed return or risk thresholds.
- Reinforcement Learning: Reinforcement learning learns policies by optimizing long-term reward sequences through interaction with an environment formalized as a Markov decision process.
- Deep Reinforcement Learning: Deep reinforcement learning uses neural networks to approximate value functions or learn policy mappings, while portfolio applications increasingly learn deep features and state representations.
4 Problem Setup
The portfolio-allocation problem is framed as a long-only reinforcement-learning task in which agents select portfolio weights from market observations and receive Differential Sharpe rewards. The environment replays historical data, rebalances holdings, and returns updated portfolio values and rewards at each timestep.
- Environment: The environment simulates the market through historical-data replay, converts actions into portfolio allocations, and rebalances using current prices.Whole-share rounding converts residual allocation into cash before the next state is produced.
- Portfolio formulation: The agent selects non-leveraged, long-only portfolio weights that sum to one across n assets and cash.Shorting and leverage are excluded; continuous actions can be constrained with softmax.
- State representation: The observation state is a [(n + 1) × T] matrix containing current allocations, asset log returns, and market-volatility indicators.The volatility features are vol20, vol20/vol60, and VIX; the lookback period is T = 60 days.
- Reward: The reward is the Differential Sharpe Ratio, chosen to optimize risk-adjusted returns at each timestep rather than cumulative profit alone.It is used because the ordinary Sharpe ratio is defined over a period and is unsuitable for online learning.
- Learning algorithm: Policy-gradient methods learn a policy πθ(a|s) that maps each state to a distribution over actions, and the experiments use PPO.The policy parameters are optimized using on-policy experience collected from the latest learned policy.
- Environment: At each timestep, the environment calculates portfolio value from current prices, prior holdings, and cash, then returns the next portfolio value and Differential Sharpe reward.The study assumes no transaction costs and permits immediate rebalancing.
5 Experiments
The experiments use S&P 500 sector and market data with regime-sensitive features and sliding-window training. DRL and MVO are evaluated under aligned conditions through repeated out-of-sample backtests using portfolio-level performance statistics.
- Data and features: Daily adjusted close data for S&P 500 sector indices, the VIX, and the S&P 500 index covers 2006–2021 and is converted into log returns.The sector-index data are illustrated in Figure 1.
- Data and features: The feature set includes 20-day and 60-day rolling volatility measures and their ratio to indicate shifts between lower- and higher-volatility regimes.The DRL observation uses the first and third volatility metrics together with VIX, standardized with an expanding lookback window.
- Training process: The data are divided into 10 one-year-shifted groups, each containing five training years, one validation burn year, and one held-out backtest year.This design addresses scarce, non-stationary financial time series through repeated retraining or fine-tuning.
- Training process: The experiment generates 50 agents from 10 periods and 5 random seeds, with the best validation agent used to seed the next training window.The first window trains on [2006–2011), validates on 2011, and tests on 2012.
- MVO comparison: MVO uses the same 60-day lookback as DRL, estimates means from sample averages, and uses Ledoit-Wolf shrinkage for covariances before Sharpe-ratio optimization.Weights are obtained at every timestep using PyPortfolioOpt.
- Backtesting: Both strategies are evaluated in 10 independent backtests beginning with $100,000, daily trading, long-only weight constraints, and whole-share purchases.Daily portfolio values and returns are used to calculate the reported statistics.
- Backtesting: DRL uses deterministic PPO agents with a gap burn year, whereas MVO recalculates weights daily from the preceding 60-day window without training.The two procedures therefore differ in model training but use comparable historical information windows.
6 Results
Across the 2012–2021 backtests, DRL generally outperformed MVO in risk-adjusted performance, return consistency, drawdown, and portfolio turnover. DRL also produced steadier monthly and annual returns with a narrower monthly-return distribution.
- DRL outperformed MVO with higher Sharpe ratios and lower yearly maximum drawdowns in virtually every backtest year.The comparison covered testing periods from 2012 through 2021.
- DRL annual returns and Sharpe ratio were approximately 1.85× higher than MVO’s across the ten backtests.The DRL strategy achieved a Sharpe ratio of 1.17 versus 0.68 for MVO over the full backtest period.
- DRL generated steadier month-to-month returns and more consistent positive annual returns than MVO.MVO more frequently swung between high- and low-return periods without a steady positive trajectory.
- Table 2 averages performance metrics across ten backtests, except Max Drawdown, which is reported as the maximum observed in any period.
- DRL’s monthly-return distribution had a lower standard deviation and spread than MVO’s, while retaining a positive mean.
- DRL changed portfolio composition less frequently than MVO, implying lower average transaction costs in practice.During the March 2020 downturn, MVO’s average change in portfolio composition was nearly double DRL’s.
7 Conclusion
The paper compares DRL with MVO through repeated US-equity backtests and reports stronger performance for DRL across returns, risk, consistency, and turnover. It also identifies transaction-cost modeling, slippage, drawdown-aware rewards, and regime switching as future extensions.
- Framework: The proposed environment wraps historical market data with market replay and supports multi-asset allocation while allowing transaction costs to be added.This design is intended as a simple, modifiable environment for portfolio-allocation experiments.
- Empirical comparison: Figure 2 presents backtest performance metrics for MVO and DRL portfolio allocation.The figure is used to compare the two approaches over the study’s backtest periods.
- Empirical comparison: Ten backtest experiments show improved DRL performance over MVO in annual returns, Sharpe ratio, and maximum drawdown, with more consistent returns and decreased turnover.The comparison uses different periods in the US Equities Market with S&P 500 Sector indices.
- Future work: The study proposes modeling transaction costs and slippage explicitly during reallocation or through a reward penalty in future work.The current future-work agenda also includes adding drawdown minimization to the reward.
- Future work: A future regime-switching model would allocate funds between low- and high-volatility agents according to market volatility.This would provide an explicit alternative to the current implicit regime parametrization.